Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Story of XZ Backdoor [video]

youtube.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Story of XZ Backdoor [video]

#31

Post by chatmasta »

lrasinen wrote: post #18
nerevarthelame wrote: post #5

Earlier quoted context omitted.

Still no smoking gun, but possibly Russia. From the video https://youtu.be/aoag03mSuXQ?t=2883 : > A lot of the aliases, like Jia Tan, they sound like Asian names, and the published changes are all timestamped in UTC+8, Beijing time. So the signs point to China. And that's why it's probably not China. I mean, why would they make it that obvious? Every other part of the operation has been so meticulous, so cautious. >…

UTC+2 isn't very convincing as an argument for Russia. Only the Kaliningrad exclave uses that timezone, and if I were in a state-backed group, I'd live in one of the big cities. Also quick search suggested UTC+3 was seen during the summer, and Russia doesn't do DST either. Edit: some of the UTC+2/3 times are attributable to being differences in git committer and author dates (e.g. email patches)

I’ve always found this an amusing method of attribution considering top tier hackers are unlikely to be writing code only during office hours.

Re: Story of XZ Backdoor [video]

#32

Post by rwmj »

mayama wrote: post #29
amiga386 wrote: post #17

Lovely video, going into almost everything... ...and yet, zero mention of systemd's recommendation for programs to link in the libsystemd kitchen sink just to call sd_notify() (which should really be its own library) ...and no mention of why systemd felt the need to preemptively load compression libraries, which it only needs to read/write compressed log files, even if you don't read/write log files at all? Again, it…

From my vague memory of xz backdoor, I don't even recall systemd being involved. Now, I get what people are talking about when they said systemd is taking over everything and why there was so much pushback to systemd when it was being added to distros. For me as a end user/dev, it mattered little whether services were started by systemd, openrc etc.

libsystemd was the indirect dependency that caused liblzma to be pulled into sshd.

Re: Story of XZ Backdoor [video]

#33

Post by amiga386 »

mayama wrote: post #29
amiga386 wrote: post #17

Lovely video, going into almost everything... ...and yet, zero mention of systemd's recommendation for programs to link in the libsystemd kitchen sink just to call sd_notify() (which should really be its own library) ...and no mention of why systemd felt the need to preemptively load compression libraries, which it only needs to read/write compressed log files, even if you don't read/write log files at all? Again, it…

From my vague memory of xz backdoor, I don't even recall systemd being involved. Now, I get what people are talking about when they said systemd is taking over everything and why there was so much pushback to systemd when it was being added to distros. For me as a end user/dev, it mattered little whether services were started by systemd, openrc etc.

systemd was the key to the whole backdoor.

OpenSSH is maintained by the OpenBSD developers. OpenSSH does not use liblzma (xz) at all.

Linux distros which chose to switch to systemd also chose to patch OpenSSH to call systemd's sd_notify() function, to inform systemd when sshd is fully started.

This sd_notify() function is in the huge, sprawling kitchen sink of a library called libsystemd. sd_notify() is only a few lines of code, but it's convenient (to Linux distro packagers) to make systemd a dependency of OpenSSH, link in the whole library and call that one function. It makes their patches of the upstream software smaller and easier to review for correctness.

In the sprawling libsystemd is an entire subsystem for reading/writing systemd's famous binary log files, and the user can choose compression (xz, zstd or lz4). It depended on and loaded all three of these compression libraries, whether you read/write compressed logs or not. In the video you hear about the imminent request to load these libraries dynamically on demand -- https://github.com/systemd/systemd/pull/31550 -- but this arrives many years adding these functions to the libsystem kitchen sink, and generally speaking most programs shouldn't use the libsystemd functions for reading/writing log files, they only need to send log messages to journald via syslog() or sd_journal_print()

So you can see this unwarranted dependency chain was introduced by Linux distros adding systemd to everything, and nation-state level hackers saw and tried to exploit it, seeking out the xz maintainer for social engineering.

Re: Story of XZ Backdoor [video]

#34

Post by dralley »

ApolloFortyNine wrote: post #26
tokyobreakfast wrote: post #25

Earlier quoted context omitted.

> and it was caught luckily at the last minute This isn't correct at all. The changes were merged into xz and made it into testing branches of major Linux distros. It was caught at T plus a few minutes only because a neurotic Microsoft employee performing debugging noticed an obscure performance issue. You can literally say Microsoft saved Linux that day. Imagine thinking this 25 years ago. It's the difference betwee…

Redhat actually stumbled on the bug separately with valgrind errors triggering, so it's days were likely numbered regardless. Probably saved them a lot of debugging but the writing was on the wall.

Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue, so it's not really clear that the original version would have still gotten as deep of an investigation as would have been needed to find the issue.

It's possible though. The noise around it did at least put Freund on alert and we should be very glad both that "Jia Tan" made the mistakes they made originally and that Freund followed up on their gut feeling

Re: Story of XZ Backdoor [video]

#35

Post by gopalv »

II2II wrote: post #10

Even though the video is somewhat sensationalized at some points, it is well worth a watch for people who are interested in computers but don't have a background in it. There is a nice mixture of everything from history (e.g. the founding of the FSF) to a clear explanation of a compression algorithm (clear enough that one should be able to implement it). It also makes claims that should make some people stop and thin…

> a clear explanation of a compression algorithm

The huffman tree, LZ77 and LZMA explanation is truly excellent for how concise the explanation is.

The earlier Veritasium video on Markov Chains in itself is linked if you don't know what a markov chain is.

I expected Veritasium to tank when it got sold to private equity & Derek went to Australia, but been surprised to see the quality of the long form stuff churned out by Casper, Petr, Henry & Greg.

Re: Story of XZ Backdoor [video]

#36

Post by bigbadfeline »

amiga386 wrote: post #17

Lovely video, going into almost everything... ...and yet, zero mention of systemd's recommendation for programs to link in the libsystemd kitchen sink just to call sd_notify() (which should really be its own library) ...and no mention of why systemd felt the need to preemptively load compression libraries, which it only needs to read/write compressed log files, even if you don't read/write log files at all? Again, it…

> Is the real master hacker Lennart Poettering, for making sure his architectural choices didn't appear in this video?

systemd is doing what it was designed to do... Cute videos are doing what they were designed to do too - hiding that!

> OpenBSD don't include any lines of code in OpenSSH to support systemd. Come to think of it, "BSD" is another thing they don't mention in the script

And this!

Re: Story of XZ Backdoor [video]

#37

Post by amiga386 »

dralley wrote: post #34
ApolloFortyNine wrote: post #26

Earlier quoted context omitted.

Redhat actually stumbled on the bug separately with valgrind errors triggering, so it's days were likely numbered regardless. Probably saved them a lot of debugging but the writing was on the wall.

Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue, so it's not really clear that the original version would have still gotten as deep of an investigation as would have been needed to find the issue. It's possible though. The noise around it did at least put Freund on alert and we should be very glad both that "Jia Tan" made the…

> Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue

Video of Jia Tan fixing the valgrind bugs: https://www.youtube.com/watch?v=A16YuzuKN58&t=138s

Re: Story of XZ Backdoor [video]

#38

Post by lrasinen »

lrasinen wrote: post #18
nerevarthelame wrote: post #5

Earlier quoted context omitted.

Still no smoking gun, but possibly Russia. From the video https://youtu.be/aoag03mSuXQ?t=2883 : > A lot of the aliases, like Jia Tan, they sound like Asian names, and the published changes are all timestamped in UTC+8, Beijing time. So the signs point to China. And that's why it's probably not China. I mean, why would they make it that obvious? Every other part of the operation has been so meticulous, so cautious. >…

UTC+2 isn't very convincing as an argument for Russia. Only the Kaliningrad exclave uses that timezone, and if I were in a state-backed group, I'd live in one of the big cities. Also quick search suggested UTC+3 was seen during the summer, and Russia doesn't do DST either. Edit: some of the UTC+2/3 times are attributable to being differences in git committer and author dates (e.g. email patches)

I couldn't let this be, so I went through the commits and as far as I can tell, that's the case. The committer/author names and timestamps are consistent with using --author on a commit (... or in a few cases, --amend --author).

Except one: commit 3d1fdddf9 has Jia Tan as both author and committer but the author timestamp is in +0300 while the commit timestamp is +0800.

Re: Story of XZ Backdoor [video]

#39

Post by wasting_time »

mbauman wrote: post #6

I'm still floored that Andres both found this and didn't ignore it. It's such a testament to an incredible engineer. (But also, my conspiratorially-inclined mind is quite entertained by the thought of some sort of parallel construction or tip from a TLA.)

With the enormous budgets we allocate in the name of "national security", this is exactly the kind of work I expect TLAs to do.

Instead we have come to expect them to cowardly sit on exploits, or actively introduce them, rather than working to secure the general public from adversaries.

What a mess.

Re: Story of XZ Backdoor [video]

#40

Post by jraph »

II2II wrote: post #10

Even though the video is somewhat sensationalized at some points, it is well worth a watch for people who are interested in computers but don't have a background in it. There is a nice mixture of everything from history (e.g. the founding of the FSF) to a clear explanation of a compression algorithm (clear enough that one should be able to implement it). It also makes claims that should make some people stop and thin…

And they interviewed some people involved in this event, which is quite nice.
Post reply on HN