Live data from Hacker News

Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

devever.net

51–60 of 150 posts

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#51

The summary of the attack from https://notes.valdikss.org.ru/jabber.ru-mitm/ is very interesting: * The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 * The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since…

How would you do certificate pinning if you don't control the clients? My understanding is that certificate pinning is only possible if you control the clients, in which case you can embed which certificates are allowed directly in the client and bypass the whole web PKI. In a situation with general-purpose clients connecting, how would they know which certificates are meant to be allowed? That's what the web PKI is…

TLS client certificates.

Use them.

Stop using domains. Stop.

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#52

The summary of the attack from https://notes.valdikss.org.ru/jabber.ru-mitm/ is very interesting: * The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 * The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since…

> Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses This suggests a compromise of Hetzner and Linode network management.

More likely the carrier upstream of them.

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#53

Earlier quoted context omitted.

extremely difficult to get physical access in a datacenter

All the people working in the datacenter have that level of physical access. Unless they are very closely supervised they can do a lot of damage without anybody being the wiser until they get caught. I've been in (nominally very secure) DCs on behalf of customers and I've seen: - unlocked racks - doors open - temporary network cables and keyboards, monitors and mice attached to running systems - systems logged in lef…

Agree re: everything you said but wanted to add datadentre security staff are some of the most interesting characters I’ve encountered. Not sure I sleep as well at night after seeing what I saw.

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#54

Earlier quoted context omitted.

>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt This implies that the attacker was able to validate domain ownership of jabber.ru and xmpp.ru by either a) being able to respond to the ACME protocol on domain(s) hosted http OR b) being able to write to a DNS TXT file for the domain(s). This implies that the victim already lost control of their process server and/or their nameserver.

No. Most probably it merely implies someone upstream of them (presumably their hosting provider under compulsion from German law enforcement) intercepted and spoofed their unencrypted, unauthenticated ACME HTTP traffic. Same can happen to you.

Okay, but the part I don't understand is why do it this way? If the hosting provider is a threat, they don't need new certs. They can just read your private keys off your disk and MITM with that.

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#55

> What would a perfect attacker do? If you had physical access to the computer, some sort of bus interception to exfiltrate data from the machine.

Thinking laterally for a moment regarding the big picture here, why do we still rely on data centres.

They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points.

And the more we break away from data centres and clouds, the more the internet infrastructure will have to work the way it was designed instead of having to flow through these crazy aggregation points that are both serious points of failure and major security risks.

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#56
post #49

A quick warning on hetzner. I needed a personal bare metal machine so signed up. I was travelling and on an IP in a distant land so their sign up asked for secondary verification via PayPal. All passed and now it’s should get a server? Nope - next day their support emailed telling me they would not approve my account without… no word of a lie here… either 1: a fax of my passport info page or 2: a scan and email conta…

When was this? Few years ago, something similar happend to me but they provided pub gpg key that i could use..

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#57

Earlier quoted context omitted.

No. Most probably it merely implies someone upstream of them (presumably their hosting provider under compulsion from German law enforcement) intercepted and spoofed their unencrypted, unauthenticated ACME HTTP traffic. Same can happen to you.

Okay, but the part I don't understand is why do it this way? If the hosting provider is a threat, they don't need new certs. They can just read your private keys off your disk and MITM with that.

Here's a good example why trust is not binary ("you either trust Hetzner or don't"). Quite possibly wiretapping law doesn't require them to facilitate the server breakin.

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#58

Earlier quoted context omitted.

No. Most probably it merely implies someone upstream of them (presumably their hosting provider under compulsion from German law enforcement) intercepted and spoofed their unencrypted, unauthenticated ACME HTTP traffic. Same can happen to you.

Okay, but the part I don't understand is why do it this way? If the hosting provider is a threat, they don't need new certs. They can just read your private keys off your disk and MITM with that.

Your private keys at rest are / should be encrypted, so it would take a bit more than just reading them.

The next level of mitigating that sort of a thing is to have keys not be on the hosts at all. Enter HSM - Hardware Security Module. A wildly complex topic I cannot hope to cover in an HN comment, but fundamentally, the private keys are not on the same HW as the server software which needs them.

A fundamental property of an HSM is that you, the HSM user, don't actually see the private key. You can ask the HSM to generate one. Derive things from it (in the cryptographic sense of derive). Even prove the provenance of such derived data. But the HSM should not reveal the actual private key.

In the cloud world, the HSM equivalent is known as KMS (Key Management Service), and the Good cloud providers all let you manage your own (with the downside being that you now need to manage your own keys).

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#59

The summary of the attack from https://notes.valdikss.org.ru/jabber.ru-mitm/ is very interesting: * The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 * The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since…

>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt This implies that the attacker was able to validate domain ownership of jabber.ru and xmpp.ru by either a) being able to respond to the ACME protocol on domain(s) hosted http OR b) being able to write to a DNS TXT file for the domain(s). This implies that the victim already lost control of their process server and/or their nameserver.

On April 3rd the IP address for ns1.jabber.ru was changed [1]. Two weeks later the first LetsEncrypt certificate was issued. Could be related?

1. https://dns.coffee/nameservers/ns1.jabber.ru

Re: Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

#60

Who are the end users of xmpp.ru and jabber.ru? Are they hoping to pick up traffic between Russian soldiers? Spies? I hate mass surveillance as much as the next guy but why target Russian domains specifically?

I can't imagine it is war-related, or at least not in any direct sense, like literally trying to intercept soldiers or spies.

I think the more likely scenario is that someone was to catch carders/botnet operators, since Jabber/XMPP is still very popular amongst people in that scene in Russia; you'll see often see screenshots or logs containing @exploit.in, @jabber.ru, and various other servers pretty much in any Krebs on Security article, for example.

That said, I think mass-surveillance is usually pretty gross regardless of intent.

I don't know what the exact ratio of criminal to non-criminal usage of either server is, but just like how Tor simultaneously has a ton of obviously illegal content mixed with whatever percentage of legitimate use [1], they presumably still do have a fair amount of totally normal conversations going on. And so I'm not an exactly a huge fan of invading privacy in order to spy on random cybercriminals likely far out of your jurisdiction in countries unwilling to extradite in the first place.

And while I understand if Hetzner was forced to comply, unless they wanted to become another Lavabit, as someone who recommends them often, it'll still be a little disappointing, IMO, if it turns out they were aware and allowed or assisted in the MitM attack, because otherwise, I think they're one of the best hosting companies.

Their transparency is great. And I get better uptime, reliability, and support than the overwhelming majority of cloud providers without selling my organs to afford it. And their effort put into turning environmentally-friendly design choices into economically smart choices is commendable.

The "reduce" in reduce > reuse > recycle can be seen in their KISS designs—like the raised, angled roof, allowing natural heat escape rather than loads of HVAC [2]; server chassises that are more bare-minimum structural support and air-shrouds than actual chassises; and stripped-down standard motherboard designs with only the most minimal changes, like 90-degree rotated sockets to allow a single fan to cool the VRM, CPU, and memory [3], rather than a dozen screaming, power-slurping Delta fans for every server.

And their knowledge of "reuse" is evident by their server auctions, where they do Dutch-auction-style rentals of legacy hardware rather than scrapping everything, avoiding spending a load on new hardware while also avoiding the environmental cost associated with manufacturing new everything each generation.

I'd love to be able use them more if only they had dedicated servers outside of Europe.

[1]: "legitimate" is hard to define anyway; one country's journalist is another's "foreign agent", and one country's "freedom fighter" is another's "terrorist".

[2]: Der8auer: Over 200,000 Servers in One Place! Visiting Hetzner in Falkenstein - https://youtu.be/5eo8nz_niiM?t=259

[3]: Der8auer: Hetzner shows Special AM5 Board with 90° Rotated Socket - https://youtu.be/V2P8mjWRqpk

Post reply on HN