Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The xz sshd backdoor rabbithole goes quite a bit deeper

twitter.com

Post reply on HN
Print view View original

251–260 of 310 posts

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#251

Post by _8j50 »

cycomanic wrote: post #210
_8j50 wrote: post #164

Earlier quoted context omitted.

This tracks with other nation state sponsored attack patterns. I've had that same reaction before. Most APTs are like this but some Chinese,US and Russian APTs are so well funded, every aspect of their attacks is impressive. Many hackers who work for nation states also have side gigs as crimeware/ransomgang members or actual pentesting jobs. Reminds me of apt3/boyusec: https://www.bleepingcomputer.com/news/security/c…

> It still boggles my mind that americans are against banning companies like huawei and bytedance. The MSS and PLA don't mess around. The problem is that many others would have as much reason to bann US companies. I mean the US has a much more extensive history of using their security apparatus both for intelligence and economic means even against their allies. Now if everyone bans everyone else we will let the world…

China already bans US companies and this is an active malicious threat not a vague possibility of harm. You can jail mark zukerberg but you can't jail xi jiping

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#252

Post by anarazel »

cesarb wrote: post #162
sureglymop wrote: post #96

Does anyone have a good explanation or introduction into the performance testing that was done to find this? And how to get started? Actually measuring performance always seemed to be a very hard task and I'd like to be able to do similar testing as the person which found this backdoor.

From what I understand, it wasn't even the performance testing itself that caught the backdoor. The developer wanted to have the machine as quiescent as possible (so that nothing else running on it would interfere) before starting the performance tests, but sshd was using much more CPU than expected (and this could be observed with simple tools like "top"). My guess is that the usual "backscatter" of password guessin…

Nearly spot on. I indeed was seeing sshd usage via top.

> but the backdoor made each login attempt use a significant amount of CPU time to check whether it was an encrypted request from the attacker

Absurdly enough, the high cpu usage is well before it even tries to figure that out. Due to avoiding "suspicious" strings in both the binary and memory, it has a more expensive string matching routine. Finding all the symbols it from the in-memory symbol tables ends up slow due to that.

That's why even sshd -h (i.e. printing help) is slow when started in the right environment. There's not enough visibility into other things that early during process startup (this happens long before main() is called), so they couldn't check what key is being presented or such. They really "should" have deferred much more of their initialization until after the ed448 check happened.

> (and this particular machine had its sshd open to the Internet because it was a cloud machine being accessed via ssh through the open Internet)

Unfortunately not. It's a machine I have at home, with some port of my public IP redirected to it, as I often need it when not at home. Oddly enough, my threat model did not include getting attacked with something as sophisticated as this, so I thought that was fine (only pubkey, no root).

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#253

Post by anarazel »

ufo wrote: post #206
ufmace wrote: post #14

The weird thing about this one is how it seems super professional in some ways, and rather amateur in others. Professional in the sense of spending a long time building up an identity that seemed trustworthy enough to be made maintainer of an important package, of probably involving multiple people in social manipulation attacks, of not leaking the true identity and source of the attack, and the sophistication and ob…

IIRC, according to Andres Freund the perf regression only happened in machines using the -fno-omit-frame-pointer setting, which was not the default at that point.

The -fno-omit-frame-pointer bit is separate from the slowdown. -fno-omit-frame-pointer lead to valgrind warnings, but no perceptible slowdown in most (including this) cases.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#254

Post by _8j50 »

afiori wrote: post #231
_8j50 wrote: post #164

Earlier quoted context omitted.

This tracks with other nation state sponsored attack patterns. I've had that same reaction before. Most APTs are like this but some Chinese,US and Russian APTs are so well funded, every aspect of their attacks is impressive. Many hackers who work for nation states also have side gigs as crimeware/ransomgang members or actual pentesting jobs. Reminds me of apt3/boyusec: https://www.bleepingcomputer.com/news/security/c…

For bytedance/TikTok the question is not really "should the US ban this company" but rather "should the government be able to ban any companies it wants (and also make it illegal for VPNs to allow US users to access relevant services/websites) without having to provide any substantial evidence? Which is a very different question in practice

US companies? I'm with you, no! Foreign companies, absolutley. Even the suspicion of malicious abuse should be enough to ban a foreign company. Foreign persons and entities have no rights in the US and our government owes them as much explanation as they give us when they ban US companies on a whim.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#255

Post by anarazel »

olejorgenb wrote: post #239
rzwitserloot wrote: post #224

Earlier quoted context omitted.

I find this the most plausible explanation by far: * The highly professional outfit simply did not see teknoraver's commit to remove liblzma as standard dependency of systemd build scripts coming. * The race was on between their compromised code and that commit. They had to win it, with as large a window as possible. * This caused the amateuristic aspects: Haste. * The performance regression is __not__ big. It's luck…

They could also have regrouped and found another way to do the exploit, given the relative ease of updating the payload (though it's probably a limited number of times you could change the test blobs without causing suspicion?). But I agree this explanation is plausible.

If lzma isn't loaded as part of sshd, the path from an lzma backdoor to sshd get a hell of a lot more circuitous and/or easier to catch. You'd pretty much need to modify the sshd binary while compressing a package build, or do something like that to the compiler, to then modify sshd components while compiling.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#256

Post by anarazel »

rzwitserloot wrote: post #224
ufmace wrote: post #14

The weird thing about this one is how it seems super professional in some ways, and rather amateur in others. Professional in the sense of spending a long time building up an identity that seemed trustworthy enough to be made maintainer of an important package, of probably involving multiple people in social manipulation attacks, of not leaking the true identity and source of the attack, and the sophistication and ob…

I find this the most plausible explanation by far: * The highly professional outfit simply did not see teknoraver's commit to remove liblzma as standard dependency of systemd build scripts coming. * The race was on between their compromised code and that commit. They had to win it, with as large a window as possible. * This caused the amateuristic aspects: Haste. * The performance regression is __not__ big. It's luck…

> The performance regression is __not__ big. It's lucky Andres caught it at all. It's also not necessarily all that simple to remove it. It's not simply a bug in a loop or some such. If I was the xz team I'd have enough faith in all the work that was done to give it high odds that they'd get there before discovery. That they'd have time; months, even.

True in the contents of sshd logins it isn't that big, but ~500ms to get from _start() to main() isn't small either, compared to the normal cost of that phase of library startup. Their problem was that the sshd daemon fork+exec's itself to handle a connection, so they had to redo a lot of the work for each connection.

I suspect they started off with much smaller overhead and then it increased gradually, with every feature they added, just like it happens with many software projects. Here's the number of symbols being looked that a reversing effort has documented: https://github.com/smx-smx/xzre/blame/ff3ba18a39bad272ff628b... https://github.com/smx-smx/xzre/blob/ff3ba18a39bad272ff628bb...

Afaict all of this happens before there's any indication of the attacker's keys being presented - that's not visible to the fork+exec'd sshd until a lot later.

They needed to some of the work before main(), to redirect RSA_public_decrypt(). That'd have been some measurable overhead, but not close to 500ms. The rest of the startup could have been deferred until after RSA_public_decrypt() was presented with something looking like their key as part of the ssh certificate.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#257

Post by timschmidt »

usr1106 wrote: post #189
timschmidt wrote: post #113

Earlier quoted context omitted.

Moxie's reasons for disallowing Signal distribution via F-droid always rang a little flat to me ( https://github.com/signalapp/Signal-Android/issues/127 ). Lots of chatter about the supposedly superior security model of Google Play Store, and as a result fewer eyes independently building and testing the Signal code base. Everyone is entitled to their opinions, but independent and reproducible builds seem like a net p…

The link provides interesting reading, but I believe Moxie must have changed his opinion later: I have never had Google Play Store on my phone, but I could install Signal. I am pretty sure I did not install it from any dodgy site. It warned when it got outdated. Not sure how updates work, not using it anymore.

No, he hasn't changed his mind (93 closed issues over 8 years related to F-droid, many asking for F-Droid distribution: https://github.com/signalapp/Signal-Android/issues?q=is%3Ais... ). Signal distributes their own APK from their own site, but still does now allow F-Droid to distribute a version, or for any version built or distributed by anyone other than Signal to connect to the Signal servers. Imagine Jia Tan's build of XZ being the only one allowed, and you get the idea.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#258

Post by lolinder »

fl7305 wrote: post #244
lolinder wrote: post #221

Earlier quoted context omitted.

The biggest thing for me that points to a state actor is the amount of time committed to the social engineering attack versus the expected value of the prize. A for-profit scheme built this way would be irrational, which doesn't preclude it being an irrational actor (or an individual with a motive other than profit) but does point to a state actor as a likely candidate. The total value of the prize, if successful, wo…

> I tend to expect private individuals to seek targets with a higher expected value. If you're a very skilled and dedicated hacker, what other targets do you have that can net you many millions of dollars? > or an individual with a motive other than profit Isn't one of the most striking things about the hacker community the extreme amounts of time and effort that are put into things that are not expected to generate…

> Isn't one of the most striking things about the hacker community the extreme amounts of time and effort that are put into things that are not expected to generate any profit?

You're conflating the hacker-as-in-threat-actor community with the hacker-as-in-Linux-maintainer community.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#259

Post by fl7305 »

lolinder wrote: post #258
fl7305 wrote: post #244

Earlier quoted context omitted.

> I tend to expect private individuals to seek targets with a higher expected value. If you're a very skilled and dedicated hacker, what other targets do you have that can net you many millions of dollars? > or an individual with a motive other than profit Isn't one of the most striking things about the hacker community the extreme amounts of time and effort that are put into things that are not expected to generate…

> Isn't one of the most striking things about the hacker community the extreme amounts of time and effort that are put into things that are not expected to generate any profit? You're conflating the hacker-as-in-threat-actor community with the hacker-as-in-Linux-maintainer community.

Sure, generally speaking, people who try to break into computer systems for profit do not have a lot of overlap with people who spends lots of time writing open source software for fun.

But in this case it is not hard to imagine that the XZ-perpetrator came from the second group, right?

Edit: I mean, this wouldn't be that different from when Ken Thompson demonstrated how to do a hidden backdoor in the C compiler?

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#260

Post by ThePowerOfFuet »

cesarb wrote: post #171
ThePowerOfFuet wrote: post #159

Earlier quoted context omitted.

Re the secret knock in the Apple silicon, a friend of mine once said "that's how you lose the NOBUS on a backdoor", and I think they were absolutely right. The one thing which most leads me to believe this was an intentional backdoor? The S-boxes.

> The one thing which most leads me to believe this was an intentional backdoor? The S-boxes. If you look at that HN discussion, you'll find a link to a Mastodon post from an Asahi Linux developer explaining that these "S-boxes" are actually an ECC calculation, and that the registers are probably cache debug registers, which allow writing directly to the cache bypassing the normal hardware ECC calculation, so you hav…

Correct, but you still have to know the values for the Hamming operation; my point stands.
Post reply on HN