Backdoor in upstream xz/liblzma leading to SSH server compromise • Hacker News Board

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#941

Post by yrro » Sun, Mar 31, 2024, 12:59 PM UTC

delroth wrote: post #25

agwa wrote: post #8

> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded…

> The systemd notification protocol could have been as simple as just writing a newline to a pipe It basically is. libsystemd links to liblzma for other features not related to notifications. (The protocol is that systemd passes the path to a unix socket in the `NOTIFY_SOCKET` env variable, and the daemon writes "READY=1" into it.)

The funny thing is that libsystemd _used_ to be split into several different libraries. I certainly remember libsystemd-journal (which is presumably the part of libsystemd that pulls in liblzma) being separate to libsystemd-daemon (which is the part that implements sd_notify, as used by OpenSSH [after patching by distros]).

If that split had never happened, then liblzma wouldn't have ended up being linked into sshd...

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#942

Post by pmarreck » Sun, Mar 31, 2024, 1:12 PM UTC

speleding wrote: post #835

jonathanspw wrote: post #154

Earlier quoted context omitted.

Yesterday sure was fun wasn't it :p Thanks for all your help/working with me on getting this cleaned up in Fedora.

PSA: I just noticed homebrew installed the compromised version on my Mac as a dependency of some other package. You may want to check this to see what version you get: xz --version Homebrew has already taken action, a `brew upgrade` will downgrade back to the last known good version.

sorry, what exact version(s) is the one(s) affected again?

(or SHAs, etc.)

(EDIT: 5.6.0 and 5.6.1 ?)

(EDIT 2: Ooof, looks like the nix unstable channel uses xz 5.6.1 at this time)

I use Nix to manage this stuff on Mac, not Homebrew...

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#943

Post by phendrenad2 » Sun, Mar 31, 2024, 4:15 PM UTC

agwa wrote: post #18

asveikau wrote: post #10

That's completely crazy, the backdoor is introduced through a very cryptic addition to the configure script. Just looking at the diff, it doesn't look malicious at all, it looks like build script gibberish.

Thanks to autoconf, we're now used to build scripts looking like gibberish. A perfect place to hide a backdoor.

Not just autoconf. Build systems in general are a bad abstraction, which leads to lots and lots of code to try to make them do what you want. It's a sad reality of the mismatch between a prodecural task (compile files X, Y, and Z into binary A) and what we want (compile some random subset of files X, Y, and Z, doing an arbitrary number of other tasks first, into binary B).

For fun, you can read the responses to my musing that maybe build systems aren't needed: https://news.ycombinator.com/item?id=35474996 (People can't imagine programming without a build system - it's sad)

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#944

Post by esbranson » Sun, Mar 31, 2024, 4:17 PM UTC

agwa wrote: post #8

> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded…

Not a programmer, but couldn't the distribution's sshd patches for systemd (and all other distro patches for privileged daemons) use static includes? Wouldn't that have only pulled in the simple client-side communication API? Would that have defeated this vector? Would it be doable?

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#945

Post by Rhea_Karty » Sun, Mar 31, 2024, 6:24 PM UTC

bed99 wrote: post #922

Rhea_Karty wrote: post #672

TLDR: Some people have been throwing around “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone. These are my notes on time stamps/zones. There are a few interesting bits that I haven't fully fleshed out. The following analysis was conducted on JiaT75’s ( https://github.com/JiaT75?…

I asked ChatGPT 4 based on Jia's Github avatar image: The timezones that ChatGPT thinks the avatar comes from aligns with +2 and +3, see what how it ranked and at the end the description of Jia's avatar: --- Rank, Score, Country, City, Timezone, Criteria 1, 10, Saudi Arabia, Mecca, AST (UTC+3), Heartland of Islam, deeply rooted calligraphic traditions. 2, 9.5, Iran, Tehran, IRST (UTC+3:30), Integral Persian calligrap…

Interesting :). However, I think that EET is the only time zone that works. (This is mostly because is seems that the area follows DST, which most non western countries in the worlf do not).

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#946

Post by k8svet » Mon, Apr 01, 2024, 3:25 AM UTC

Epa095 wrote: post #710

I hope Lasse Collin is doing OK! Here is a older message from him [1] "I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see. It's also good to keep in mind that this is an unpaid hobby project. " Gi…

I would like to see more attention given to this. I'm capable of compartmentalization and not over-guilting myself, but holy hell, I really hope he's doing alright. This would kind of destroy me.

I was actually telling my dad about this. I have a project, 500+ users, not quite root access, but enough to cause serious damage. I can think of at least one covert way to backdoor the binary artifacts from it.

About two years ago, someone showed up, started making good commits. In this case, they have some other community rep that goes back a bit further but... man it's an unsettling feeling.

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#947

Post by dikei » Mon, Apr 01, 2024, 3:36 AM UTC

_cenw wrote: post #689

0xthr0w4 wrote: post #339

Out of curiosity I looked at the list of followers of the account who committed the backdoor. Randomly picked https://github.com/Neustradamus and looked at all their contributions. Interestingly enough, they got Microsoft to upgrade ([0],[1]) `vcpkg` to liblzma 5.6.0 3 weeks ago. [0] https://github.com/microsoft/vcpkg/issues/37197 [1] https://github.com/microsoft/vcpkg/pull/37199

Hey, I remember this guy! Buddy of someone who tried to get a bunch of low quality stuff into ifupdown-ng, including copying code with an incompatible license and removing the notice. He's in every PR, complaining the "project is dead". He even pushes for the account to be made "team member". https://github.com/ifupdown-ng/ifupdown-ng/pulls/easynetdev He follows 54k accounts though, so it may indeed just be coinciden…

I wouldn't be surprised if that is just a bot.

He even follows me, though I have never published any open-source project on my own.

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#948

Post by 01apr2029 » Mon, Apr 01, 2024, 8:16 AM UTC

k8svet wrote: post #946

Epa095 wrote: post #710

I hope Lasse Collin is doing OK! Here is a older message from him [1] "I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see. It's also good to keep in mind that this is an unpaid hobby project. " Gi…

I would like to see more attention given to this. I'm capable of compartmentalization and not over-guilting myself, but holy hell, I really hope he's doing alright. This would kind of destroy me. I was actually telling my dad about this. I have a project, 500+ users, not quite root access, but enough to cause serious damage. I can think of at least one covert way to backdoor the binary artifacts from it. About two ye…

> I'm capable of compartmentalization

teach me how. help me learn how, please. any resources with practical utility you can share? or any class of therapists that are good at teaching this with right frameworks offered? thank you

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#949

Post by prissy04 » Mon, Apr 01, 2024, 12:33 PM UTC

mborch wrote: post #724

Rhea_Karty wrote: post #672

TLDR: Some people have been throwing around “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone. These are my notes on time stamps/zones. There are a few interesting bits that I haven't fully fleshed out. The following analysis was conducted on JiaT75’s ( https://github.com/JiaT75?…

This 2011 addition to the XZ Utils Wikipedia page is interesting because a) why is this relevant, b) who is Mike Kezner since he's not mentioned on the Tukaani project page ( https://tukaani.org/about.html ) under "Historical acknowledgments". https://en.wikipedia.org/w/index.php?title=XZ_Utils&diff=pre... Arch Linux played an important role in making this compression software trusted and depended upon. Perhaps not a…

If you check the history of that IP address, it added Mike Kezner to other pages. No clue why.

Re: Backdoor in upstream xz/liblzma leading to SSH server compromise

#950

Post by xign » Mon, Apr 01, 2024, 12:40 PM UTC

pmarreck wrote: post #820

jnxx wrote: post #477

Earlier quoted context omitted.

And, Joey Hess has counted at least 750 commits to xz from that handle. https://hachyderm.io/@joeyh/112180715824680521 This does not look trust-inspiring. If the code is complex, there could be many more exploits hiding.

I have some questions. 1) Are there no legit code reviews from contributors like this? How did this get accepted into main repos while flying under the radar? When I do a code review, I try to understand the actual code I'm reviewing. Call me crazy I guess! 2) Is there no legal recourse to this? We're talking about someone who managed to root any linux server that stays up-to-date.

"Jia Tan" was not a contributor, but a maintainer of the project. The key point here is that this is a multi-year project of infiltrating the xz project and gaining commit access.

In a large tech company (including ones I have worked at), sometimes you have policy where every change has to be code reviewed by another person. This kind of stuff isn't possible when the whole project only has 1-2 maintainers. Who's going to review your change other than yourself? This is the whole problem of OSS right now that a lot of people are bringing up.

I maintain a widely used open source project myself. I would love it if I could get high quality code review for my commits similar to my last workplace lol, but very very few people are willing to roll up their sleeves like that and work for free. Most people would just go to the Releases page and download your software instead.