Encrypted traffic interception on Hetzner and Linode targeting the largest Russian XMPP (Jabber) messaging service

[He's FAT!]

All evidence suggests that 2 large cloud infrastructure providers, Hetzner and Linode, have been covertly MiTM'ing traffic to jabber.ru/xmpp.ru:

https://notes.valdikss.org.ru/jabber.ru-mitm/

The writeup is by https://github.com/ValdikSS, who has authored GoodbyeDPI (a tool for bypassing state-level deep packet inspection for censorship) among other cool things and is pretty knowledgeable about network interception, you could say.

Note: I do say "evidence suggests" because it is not confirmed, and I wouldn't expect Hetzner or Linode to come out any time soon and say "Yep, we worked with the feds". They are probably under some kind of non-disclosure order.

This situation is also interesting because of the possibility (or implication?) that LetsEncrypt issued a valid certificate for jabber.ru on behalf of the authorities. The weaknesses of the CA model are known and basically exist for convenience because we can't have every site forcing us to check fingerprints like SSH does, can we? But the fact that there were no whistleblowers inside LetsEncrypt or Hetzner is pretty sad if this is this case.

Or maybe ValdikSS/jabber.ru got the analysis wrong and it's the fault of jabber.ru or some other hacker/attacker? We'll see. What are your thougts on this?

 

[Nitro!]

I'm inclined to agree that this was lawful interception with a gag order attached. The article mentions that the network connection was severed and the evidence presented makes a convincing argument that someone pulled the plug from the usual router and plugged the server into a monitored one.

The Let's Encrypt piece of this is more interesting to me. The ACME client that's used with LE can validate in a couple ways, most commonly over HTTP. I think that if they hijacked the jabber.ru site to do a file challenge there would have been a log entry noting this and it would have been in the writeup. There is a type of challenge I'm not familiar with but I think would have been ideal for a MITM type situation where you would not want to alert the sysop, emphasis mine:

TLS-ALPN-01​

This challenge was developed after TLS-SNI-01 became deprecated, and is being developed as a separate standard. Like TLS-SNI-01, it is performed via TLS on port 443. However, it uses a custom ALPN protocol to ensure that only servers that are aware of this challenge type will respond to validation requests. This also allows validation requests for this challenge type to use an SNI field that matches the domain name being validated, making it more secure.

This challenge is not suitable for most people. It is best suited to authors of TLS-terminating reverse proxies that want to perform host-based validation like HTTP-01, but want to do it entirely at the TLS layer in order to separate concerns. Right now that mainly means large hosting providers, but mainstream web servers like Apache and Nginx could someday implement this (and Caddy already does).

(Link | Archive)

I am not qualified to make a judgment, but at surface level, this seems to fit the bill as it could be done in the time that the server was unplugged, or otherwise done in secret. This is of course assuming that Let's Encrypt was not complicit. I am assuming that they are not, if only that they are US-based and it would be harder for a German authority to coerce them into defying their own protocols.

 

[He's FAT!]

this seems to fit the bill as it could be done in the time that the server was unplugged, or otherwise done in secret

The time in the logs for the network losing connection doesn't 100% line up with the start date of when 2 of the certs were issued on that day. Also there were 6 rogue certificates in total, so I don't think its as simple as doing the challenge during that short window.

The one thing you can probably rule out is that it wasn't a DNS hijack or using the DNS challenge, because they never generated a wildcard cert like the original. Wildcard certs can only be generated with DNS-01 as far as I know.

Edit: Come to think of it, Hetzner and Linode could just redirect any traffic coming from Letsencrypt IP ranges to some special server and complete all the challenges. Not that hard.