What we know about the xz Utils backdoor that almost infected the world • Hacker News Board

Re: What we know about the xz Utils backdoor that almost infected the world

#251

Post by EdiX » Mon, Apr 01, 2024, 5:01 PM UTC

Karellen wrote: post #155

EdiX wrote: post #19

My personal takeaways from this: 1. Source distribution tarballs that contain code different from what's in the source repository are bad, we should move away from them. The other big supply chan attack (event-stream) also took advantage of something similar. 1a. As a consequence of (1) autogenerated artifacts should always be committed. 2. Autogenerated artifacts that everyone pagedowns over during code reviews is a…

> 1a. As a consequence of (1) autogenerated artifacts should always be committed. Why don't object files and binaries count as autogenerated artifacts? Should we commit those to the repo too? Where is the line between an artifact that should be committed, and one that shouldn't be? > 4. Libsystemd is a problem for the ecosystem. libc will dynamically load libnss-* on a lot of platforms, some of which can link to a bu…

> Why don't object files and binaries count as autogenerated artifacts? Should we commit those to the repo too? Where is the line between an artifact that should be committed, and one that shouldn't be?

I'd say anything that is input to the compiler should be.

> libc will dynamically load libnss-* on a lot of platforms, some of which can link to a bunch of other helper libraries. What if the attack had come via one of those 2-or-3-dependencies-removed libraries? libc is big and complicated and most programs only use a tiny fraction of it. Is libc a problem for the ecosystem?

Yes, the libnss stuff is also a problem.

Re: What we know about the xz Utils backdoor that almost infected the world

#252

Post by unethical_ban » Mon, Apr 01, 2024, 5:06 PM UTC

TacticalCoder wrote: post #236

xyst wrote: post #136

What can be learned here and improve? - systemd and libsystemd is an absolute mess. What are the alternatives at this point? - what processes can be improved for Linux distributions to catch malicious actors trying to push back doors into the ecosystem? - is there a dependency graph that would show how many programs use a specific utility library? Might aid in finding other possible attempts to backdoor

> What can be learned here and improve? - which firewalling setups would still prevent that exploit from working (like whitelisting IP addresses or blocks allowed to SSH in or, drumroll..., port-knocking). I don't use port-knocking but now I really wonder what's going to be the argument against port-knocking. Is it really security theatre in the fact of the rube-goldberg too-big-to-ever-be-secure systemd, loading tot…

Port knocking isn't security theater, in that it is a layer of obfuscation and (weak) authentication. I consider it another weak "noise decreased" like moving SSH to a nonstandard port. Your sshd logs will be quieter making analysis easier.

Re: What we know about the xz Utils backdoor that almost infected the world

#253

Post by mardifoufs » Mon, Apr 01, 2024, 5:06 PM UTC

markus_zhang wrote: post #170

AshamedCaptain wrote: post #69

Why do they say "almost" infected the world? At least 3 quite popular Linux distributions (arch, gentoo, and opensuse tumbleweed) ended up shipping the backdoor _for weeks_ , and it was most definitely working in at least tumbleweed. For weeks! A backdoored ssh! Hardly "almost".

I hope Open Source maintainers and the big companies get the message -- they need to change the financial outlook of open source maintaining.

The original xz maintainer stated that he delegated maintenance due to (then) current mental illness issues he was facing, not because of finances.

Re: What we know about the xz Utils backdoor that almost infected the world

#254

Post by mardifoufs » Mon, Apr 01, 2024, 5:08 PM UTC

bogwog wrote: post #71

hk__2 wrote: post #38

Earlier quoted context omitted.

> Is that common to only communicate only through email/github? Yes. I’ve joined half a dozen open-source projects of various sizes (from 100 to 30k stars on GitHub) without ever calling anyone; written communication is the standard.

Sure, but handing over maintainership is a different situation from accepting a few PRs

I don't think he ever fully gave up his "top maintainer" status or gave away the repo. He just let Jian have defacto maintainership because no one else was really contributing

Re: What we know about the xz Utils backdoor that almost infected the world

#255

Post by acdha » Mon, Apr 01, 2024, 5:08 PM UTC

EdiX wrote: post #19

My personal takeaways from this: 1. Source distribution tarballs that contain code different from what's in the source repository are bad, we should move away from them. The other big supply chan attack (event-stream) also took advantage of something similar. 1a. As a consequence of (1) autogenerated artifacts should always be committed. 2. Autogenerated artifacts that everyone pagedowns over during code reviews is a…

I think your eight point is regrettable but mostly true - I’d soften it to professional relationships, which kind of sucks for anyone trying to get started in the field who doesn’t get a job with someone established, and adds an interesting wrinkle to the RTO discussion since you might “work” with someone for years without necessarily knowing anything about them.

It also seems like we need some careful cultural management around trust: enshrine trust-but-verify pervasively to avoid focusing only on, say, Chinese H1-Bs or recent immigrants (whoops, spent all of your time on them and it turns out you missed the Mossad and Bulgarian hackers) and really doubling down on tamper-evidence, which also has the pleasant property of reducing the degree to which targeting OSS developers makes sense.

Combining your 7th point with that one, I’ve been wondering whether you could expand what happened with OpenSSL to have some kind of general OSS infrastructure program where everyone would pay to support a team which prioritizes supporting non-marquee projects and especially stuff like modernizing tool chains, auditing, sandboxing, etc. so basically any maintainer of something in the top n dependencies would have a trusted group to ask for help and be able to know that everyone on that team has gone through background checks, etc.

Re: What we know about the xz Utils backdoor that almost infected the world

#256

Post by xorcist » Mon, Apr 01, 2024, 5:10 PM UTC

kuschku wrote: post #197

xorcist wrote: post #139

Earlier quoted context omitted.

No they did not. The dependency is still there, it's just being lazy loaded. This would be prevented this particular exploit, which would have needed to take another approach, but at the price of making dependencies invisible and hard to debug. You could no longer have found vulnerable systems by way of ldd. The only solution for the attack surface of systemd is to make the individual components more loosely coupled.…

Even if it hadn't been loaded by libsystemd, liblzma is also loaded by SELinux, which would have allowed the same vulnerability via a different vector. Personally I think projects like fedora silverblue/kinoite and other container-based OSes are going in the right direction. We need a base OS that's as small as possible so it can be audited, and everything else then needs to live in a container so it doesn't have to…

Well, not the same vulnerability, as libselinux isn't loaded by sshd. Any such request would probably have a low probability of acceptance among the openssh maintainers.

If anything, I think this shows that real world security is hard and must happen at every level. This library is likely to be included in any base OS no matter how small, and rebuilding the container world just to patch is inefficient.

This attack may have been found by luck alone, even if that luck involved having talented developers on our side, but it really showed how well the open source community responds to such attacks. Within a day of it being public, we had well mapped out what the problem was and how to best respond to it. A day that was also a holiday in large parts of the world.

Re: What we know about the xz Utils backdoor that almost infected the world

#257

Post by pxc » Mon, Apr 01, 2024, 5:10 PM UTC

bongodongobob wrote: post #179

markus_zhang wrote: post #170

Earlier quoted context omitted.

I hope Open Source maintainers and the big companies get the message -- they need to change the financial outlook of open source maintaining.

I think the message would more likely be "don't use open source and pay for closed source" than "give money to open source and cross your fingers that it does something".

It might be riskier (because you'd have to identify yourself with government documents) to plant a backdoor in a similar way at a large, proprietary software vendor like Microsoft. But I don't know that it would be harder. And in the case of proprietary software there would not be nearly as much public scrutiny, and the scrutinizing public would have fewer resources for inspecting it.

Re: What we know about the xz Utils backdoor that almost infected the world

#258

Post by paulmd » Mon, Apr 01, 2024, 5:25 PM UTC

EdiX wrote: post #19

My personal takeaways from this: 1. Source distribution tarballs that contain code different from what's in the source repository are bad, we should move away from them. The other big supply chan attack (event-stream) also took advantage of something similar. 1a. As a consequence of (1) autogenerated artifacts should always be committed. 2. Autogenerated artifacts that everyone pagedowns over during code reviews is a…

The other problem is that C’s engineering culture is termites all the way down.

A test resource getting linked into a final build is, itself, a problem - the tooling should absolutely make this difficult, and transparent/obvious when it happens.

But that’s difficult because C never shed the “pile of bash scripts” approach to build engineering… and fundamentally it’s an uphill battle to engineering a reliable system out of a pile of bash scripts.

The oft-discussed problems with undefined behavior, obscure memory/aliasing rules, etc are just the obvious smoke. C is termites all the way down and really shouldn’t be used anymore, it’s just also Too Big To Fail. Like if the world’s most critical infrastructure had been built in PHP.

Re: What we know about the xz Utils backdoor that almost infected the world

#259

Post by rmbyrro » Mon, Apr 01, 2024, 5:28 PM UTC

markus_zhang wrote: post #181

bongodongobob wrote: post #179

Earlier quoted context omitted.

I think the message would more likely be "don't use open source and pay for closed source" than "give money to open source and cross your fingers that it does something".

Either way, it gives more jobs and $$$ to software developers in general. I'm fine with both :) Just imagine how many more jobs will be created if every large company decides to roll their own stuffs. A lot are actually doing this, but not enough.

You should have more discretion with those ideas or disclose irony clearly, which I hope is the case here.

Re: What we know about the xz Utils backdoor that almost infected the world

#260

Post by rmbyrro » Mon, Apr 01, 2024, 5:30 PM UTC

Denvercoder9 wrote: post #81

EdiX wrote: post #19

My personal takeaways from this: 1. Source distribution tarballs that contain code different from what's in the source repository are bad, we should move away from them. The other big supply chan attack (event-stream) also took advantage of something similar. 1a. As a consequence of (1) autogenerated artifacts should always be committed. 2. Autogenerated artifacts that everyone pagedowns over during code reviews is a…

> 8. This sucks to say, but code reviews and handing off maintainership, at the moment, need to take into account geopolitical considerations. That won't help. There's no evidence that Jia Tan is a real name, or even a real person for that matter. If projects stop accepting contributions from asian-sounding names, the next attack will just use Richard Jones as a name.

This is so obvious that needing to say it shows how prejudice can blind people.