The xz sshd backdoor rabbithole goes quite a bit deeper • Hacker News Board

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#291

Post by richin13 » Tue, Apr 09, 2024, 1:05 PM UTC

richin13 wrote: post #56

ufmace wrote: post #14

The weird thing about this one is how it seems super professional in some ways, and rather amateur in others. Professional in the sense of spending a long time building up an identity that seemed trustworthy enough to be made maintainer of an important package, of probably involving multiple people in social manipulation attacks, of not leaking the true identity and source of the attack, and the sophistication and ob…

I read somewhere that some recent changes in systems would've made the backdoor useless so they had to rush out, which caused them to be reckless and get discovered

> systems

damn it, autocorrect!

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#292

Post by scubbo » Wed, Apr 10, 2024, 9:42 PM UTC

manquer wrote: post #92

k8svet wrote: post #67

Earlier quoted context omitted.

"ever increasing"? Twitter is completely, 110% unusable without an account (and dear god, I dare some of you to make a new account and see what the process and default content is. It's gross ). I say 110% not to be hyperbolic -- It shows you non-latest tweets on profiles, it doesn't let you see tweet threads or replies, even from the original poster when they post a chain of tweets. I literally can't read any of this…

> ever increasing I don’t think it means currently it is reasonable , just that things are continuing to worsen and we have not yet reached a bottom even if a bottom exists.

Indeed. For those of us without an account, the decline has long-ago reached the point where it's unusable. I assume, and hear, that for those with an account, the usability continues to decline.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#293

Post by account42 » Thu, Apr 11, 2024, 1:13 PM UTC

treffer wrote: post #242

ufmace wrote: post #14

The weird thing about this one is how it seems super professional in some ways, and rather amateur in others. Professional in the sense of spending a long time building up an identity that seemed trustworthy enough to be made maintainer of an important package, of probably involving multiple people in social manipulation attacks, of not leaking the true identity and source of the attack, and the sophistication and ob…

Well, there is a pretty logical explanation. Libsystemd was moving to a dlopen architecture for its dependencies. This means that the backdoor would not load as the sshd patch only used libsystemd for notify, which does not need liblzma at all. So they IMHO gave it a last shot. It's OK if it burns as it would be useless in 3 months (or even less). The collateral is the backdoor Binary, but given enough engineering po…

This particular backdoor might have become useless anyway, but control of liblzma would have continued to be very valuable. Not only is it used in so many places (the embedded version even in the kernel), it also is a common part of the toolchain used to build everything else allowing for trusting trust style attacks.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#294

Post by account42 » Thu, Apr 11, 2024, 1:18 PM UTC

bdd8f1df777b wrote: post #161

rav wrote: post #109

Earlier quoted context omitted.

> The backdoor relies on sshd being patched to depend on libsystemd to call sd_notify I remember when we added sd_notify support to our services at work, I was wondering why one would pull in libsystemd as a dependency for this. I mean, there's a pure-Python library [1] that basically boils down to: import os, socket def notify(state=b"READY=1"): sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) addr = os.geten…

> With proper error handling, that's about 50 lines of C code. Writing proper error handling in C is a very tedious and error prone task. So it doesn't surprise me that people would rather call another library instead.

You think error handling for a socket connection + send is outside the capabilities of those developing sshd?

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#295

Post by account42 » Thu, Apr 11, 2024, 1:40 PM UTC

rzwitserloot wrote: post #224

ufmace wrote: post #14

The weird thing about this one is how it seems super professional in some ways, and rather amateur in others. Professional in the sense of spending a long time building up an identity that seemed trustworthy enough to be made maintainer of an important package, of probably involving multiple people in social manipulation attacks, of not leaking the true identity and source of the attack, and the sophistication and ob…

I find this the most plausible explanation by far: * The highly professional outfit simply did not see teknoraver's commit to remove liblzma as standard dependency of systemd build scripts coming. * The race was on between their compromised code and that commit. They had to win it, with as large a window as possible. * This caused the amateuristic aspects: Haste. * The performance regression is __not__ big. It's luck…

> The payload of the 'hack' contains fairly easy ways for the xz hackers to update the payload. They actually used it to remove a real issue where their hackery causes issues with valgrind that might lead to discovering it, and they also used it to release 5.6.1 which rewrites significant chunks;

The valgrind fix in 5.6.1 overwrites the same test files used in 5.6.0 instead of using the injection code's extension hooks. This is done with what should have been a highly suspicious commit: https://github.com/tukaani-project/xz/commit/6e636819e8f0703... - this replaces "random" test files with other "random" test files. The state reson is questionable to begin with but not including the seed used when the the purpoted reason was to be able to re-create the files in the future is highly suspicous. This should have raised red flags bug no one was watching. I'd say this is another part of the operation that was much more sloppy than it needed to be.

> almost entirely eliminates the value of all those 2 years of hard work.

Except control over xz-utils/liblzma would have still been very valuable even without the sshd exploit path as it's central use in the toolchain used to build Linux distributions would have allowed for many other attacks.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#296

Post by account42 » Thu, Apr 11, 2024, 1:58 PM UTC

anarazel wrote: post #256

rzwitserloot wrote: post #224

Earlier quoted context omitted.

I find this the most plausible explanation by far: * The highly professional outfit simply did not see teknoraver's commit to remove liblzma as standard dependency of systemd build scripts coming. * The race was on between their compromised code and that commit. They had to win it, with as large a window as possible. * This caused the amateuristic aspects: Haste. * The performance regression is __not__ big. It's luck…

> The performance regression is __not__ big. It's lucky Andres caught it at all. It's also not necessarily all that simple to remove it. It's not simply a bug in a loop or some such. If I was the xz team I'd have enough faith in all the work that was done to give it high odds that they'd get there before discovery. That they'd have time; months, even. True in the contents of sshd logins it isn't that big, but ~500ms…

If I understand things correctly the hooking of RSA_public_decrypt is done with an audit hook called for every symbol of newly loaded libraries. With this approach it doesn't matter how much is hooked since all functions are always processed. It's also harder to hook functiosn later because the GOT/PLT will have been marked read only. The exploit code also doesn't directly contain any of the strings (presumably for obfuscation reasons) and instead has a trie to map given strings to internal IDs which also requires an approach like this where you look at all symbols and then decide what to do with each symbol.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#297

Post by account42 » Thu, Apr 11, 2024, 2:01 PM UTC

anarazel wrote: post #255

olejorgenb wrote: post #239

Earlier quoted context omitted.

They could also have regrouped and found another way to do the exploit, given the relative ease of updating the payload (though it's probably a limited number of times you could change the test blobs without causing suspicion?). But I agree this explanation is plausible.

If lzma isn't loaded as part of sshd, the path from an lzma backdoor to sshd get a hell of a lot more circuitous and/or easier to catch. You'd pretty much need to modify the sshd binary while compressing a package build, or do something like that to the compiler, to then modify sshd components while compiling.

Perhaps but sshd is also not the only potential exploit. E.g. the landlock commit is a hint that they were also planning an exploit via the xz-utils commands directly. Seems rash to burn over two years of gaining trust for a very central library and set of tools just because the initially chosen exploit path disappeared.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#298

Post by account42 » Thu, Apr 11, 2024, 2:19 PM UTC

c6400sc wrote: post #110

k8svet wrote: post #106

Earlier quoted context omitted.

(really tinfoil hatty - ) I almost wonder if it's misdirection?

Or a whitehat who couldn't get attention another way?

This hat is way too dirty to be white. Even if well intentioned (which is into fairy tale theory level of likelyness), the implementation is well beyond ethical.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#299

Post by account42 » Thu, Apr 11, 2024, 2:21 PM UTC

atomicnumber3 wrote: post #4

The sophistication here is really interesting. And it all got caught because of a fairly obvious perf regression. It reminds of a quote I heard in one of those "real crime" shows: "There's a million ways to get caught for murder, and if you can think of half of them, you're a genius."

Yet plenty of murders go unsolved forever. Sometimes the secret ingredient is a big pile of luck I guess.

Re: The xz sshd backdoor rabbithole goes quite a bit deeper

#300

Post by account42 » Thu, Apr 11, 2024, 2:41 PM UTC

eli wrote: post #12

dboreham wrote: post #6

Earlier quoted context omitted.

> And it all got caught because of a fairly obvious perf regression Always possible that was "parallel construction" evidence. Someone at a TLA discovered the attack by some other means, had a quiet Signal chat with a former colleague who works at MS...

There doesn’t seem to be any evidence to support this whatsoever yet it’s nearly impossible to disprove. Classic conspiracy theory.

There also isn't really a reason for some contrived parallel construction here - whoever found the issue could just point to it without explaining how it was detected. They could even do that anonymously.

> Classic conspiracy theory.

I would not be too quick to shit on "conspiracy theories" however as there are plenty of proven cases of people conspiring against the interests of the public.