Live data from Hacker News

Timeline of the xz open source attack

research.swtch.com

451–460 of 482 posts

Re: Timeline of the xz open source attack

#451

One big take away for me is that we should stop tolerating inscrutable code in our systems. M4 has got to go! Inscrutable shell script have got to go! Its time to stop accepting that the way we've done this in the past is the way we will continue doing it ad infinitum.

I do think we have enough eyeballs at this point that we should stop entertaining the Dancing Bear in low level libraries and start insisting on crisp, self-explaining code. There are a lot of optimizations pushed into compilers these days, and there are a lot of architectural changes that can make things fast without making them inscrutable. We should be moving from No Obvious Bugs to Obviously No Bugs (Tony Hoare).

>start insisting on crisp, self-explaining code

That’s the beauty of FOSS: if you don’t like what’s written, you can write your own replacement.

Re: Timeline of the xz open source attack

#452

Earlier quoted context omitted.

reproducible builds never made sense to me. if you trust the person giving you the hash, just get the binary from them. you don't need to reproduce the build at all. if you trust that they're giving you the correct hash, but not the correct binary, then you're not thinking clearly.

But you can build the code yourself, and verify that your hash is the same as theirs, meaning the binary is the same.

Just get their binary, then. Why do you need to build it yourself?

If you trust the people giving you the hash of the binary, just get the binary from them, too.

Re: Timeline of the xz open source attack

#453

> merges hidden backdoor binary code well hidden inside some binary test input files. [...] Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves. So much for the folks advocating for binary (driver) blobs in OSS t support otherwise unsupported hardware. It's either in source form and reproducable or it's not there.

> It's either in source form and reproducable or it's not there.

Forbidding binary data in source repositories is neither practical nor solving the issue.

Binary code can be hidden steganograhically in e. g. image files or text files.

Re: Timeline of the xz open source attack

#454

Earlier quoted context omitted.

reproducible builds never made sense to me. if you trust the person giving you the hash, just get the binary from them. you don't need to reproduce the build at all. if you trust that they're giving you the correct hash, but not the correct binary, then you're not thinking clearly.

Actually, this xz-utils was a sort of reproducible build issue. But the twist it is wasn't the binary not built reproducibly. It was the tar ball. The natural assumption is it just reflected the public git repository. It didn't. Debian's response is looking to be mandating the source must come from the git repository, not a tar ball. And it will be done using a script. And the script must produce the same output ever…

So if something is hidden in the source, a reproducible build will give you the confidence to believe that the source is fully vetted and clean.

I see what you’re saying, but I don’t buy that reproducible builds actually solve anything, especially long term. As this whole xz thing has shown us, lots of things fly under the radar if the circumstances are right. This kind of thing will absolutely happen again. In the future it may not even require someone usurping an existing library, it could be a useful library created entirely by the hacker for the express purpose of infiltration a decade later.

Reproducible builds are a placebo. You must still assume there are no bad actors anywhere in the supply chain, or that none are capable of hiding anything that can be reproducibly built, and we can no longer afford to make that assumption.

A reproducible vulnerability is still a vulnerability.

Re: Timeline of the xz open source attack

#455

Earlier quoted context omitted.

But you can build the code yourself, and verify that your hash is the same as theirs, meaning the binary is the same.

Just get their binary, then. Why do you need to build it yourself? If you trust the people giving you the hash of the binary, just get the binary from them, too.

It's easier to hide nastiness in a binary than it is in source. And indeed this XZ hack too relied on multiple obfuscated binary blobs. The hash helps because it makes it a little harder to hide things like this. It's not a silver bullet, but it would have in this specific instance made it harder to hide that malicious m4/build-to-host.m4 in the tarball - after all, had the attacker done that despite publishing a hash they would have needed to use the hash including the modified build script, but then anybody building from the git repo would have had a different hash, and that's a risk for detection.

Reproducible builds and hashes thereof aid in transparency and detecting when that transparency breaks down. Of course, it doesn't mean hackers can't hide malicious code in plain sight, but at least it makes it slightly harder to hide between the cracks as happened here.

Re: Timeline of the xz open source attack

#456

Earlier quoted context omitted.

I do think we have enough eyeballs at this point that we should stop entertaining the Dancing Bear in low level libraries and start insisting on crisp, self-explaining code. There are a lot of optimizations pushed into compilers these days, and there are a lot of architectural changes that can make things fast without making them inscrutable. We should be moving from No Obvious Bugs to Obviously No Bugs (Tony Hoare).

>start insisting on crisp, self-explaining code That’s the beauty of FOSS: if you don’t like what’s written, you can write your own replacement.

And now we have 14 standards.

Re: Timeline of the xz open source attack

#457
post #140
post #132

Earlier quoted context omitted.

and that's how dynamic linking works -- really ignorant comment Read the lobste.rs thread for some quotes on process separation and the Unix philosophy. There are mechanisms other than dynamic linking -- this is precisely the question. Also, Unix supported remote logins BEFORE dynamic linking existed. --- What about sshd didn't work prior to 2015? Was the dependency worth it? not particularly surprising given the kit…

I don't think it's acceptable to create a subprocess for what's effectively a library function call because it comes from a dependency. The problem is the design of rtld and the dynamic linking model, where one shared library can detect and hijack the function calls of another by using the auditing features of rtld. Hardened environments already forbid LD_PRELOAD for injection attacks like this, but miss audit hooks.…

I don't agree -- process isolation is a simple, effective, and traditional mechanism. It has downsides (e.g. parsing and serializing), but they don't apply here.

Many mitigations work at the process level, like ASLR, cgroups, and more.

There's a reason that Chrome doesn't allow parsing and rendering in the same process:

https://www.chromium.org/developers/design-documents/multi-p...

Something like the rule of 2 should apply to the sshd -> xzutils dependency -- it's inherently dangerous.

https://chromium.googlesource.com/chromium/src/+/master/docs...

Note that Firefox spent years of engineering effort to catch up with Chrome, retrofitting a multiprocess architecture.

Multiprocess architecture is demonstrably more secure, e.g. if you look at all the actual exploits from https://en.wikipedia.org/wiki/Pwn2Own

Re: Timeline of the xz open source attack

#458
post #8

What stands out to me is this particular justification: > 2024-02-23: Jia Tan merges hidden backdoor binary code well hidden inside some binary test input files. The associated README claims “This directory contains bunch of files to test handling of .xz, .lzma (LZMA_Alone), and .lz (lzip) files in decoder implementations. Many of the files have been created by hand with a hex editor, thus there is no better "source…

He put this comment because he knows that FOSS enthusiasts and especially Debian always prefer source over binary. This is not only true for program code, but also includes docs, images etc.

The correct way to do that would be a source that generates a test file and then a script which reproducibly produces the desired corruption.

Re: Timeline of the xz open source attack

#459
post #8

What stands out to me is this particular justification: > 2024-02-23: Jia Tan merges hidden backdoor binary code well hidden inside some binary test input files. The associated README claims “This directory contains bunch of files to test handling of .xz, .lzma (LZMA_Alone), and .lz (lzip) files in decoder implementations. Many of the files have been created by hand with a hex editor, thus there is no better "source…

Well, test files shouldn't be affecting the actual production binary. But in practice that's not something that can be enforced for arbitrary projects without those projects having set something up specifically. For example, the project could track the effect on binary size of the production binary after every PR. But then it still requires a human (or I guess an AI bot?) to notice that the increase would be unexpect…

Debian often removes these kind of binaries by patching the upstream tarball. When they are not used, that should be quite easy anyway. That's why the attacker put the statement in the first place. It increases the chance that distributions will accept these.

Re: Timeline of the xz open source attack

#460
post #455

Earlier quoted context omitted.

Just get their binary, then. Why do you need to build it yourself? If you trust the people giving you the hash of the binary, just get the binary from them, too.

It's easier to hide nastiness in a binary than it is in source. And indeed this XZ hack too relied on multiple obfuscated binary blobs. The hash helps because it makes it a little harder to hide things like this. It's not a silver bullet, but it would have in this specific instance made it harder to hide that malicious m4/build-to-host.m4 in the tarball - after all, had the attacker done that despite publishing a has…

"slightly harder" isn't enough. That's what I'm saying that people are not accepting.

The days of 3rd party libraries simply being trusted because they're open source are slowly coming to an end. the problem is not unreproducible builds, the problem is that we implicitly trust source code found online.

Projects relying on 100s of 3rd party libraries are a major problem, and no one seems to care. They like it when their Rust build lists all the libraries that they didn't have to write as it compiles them. They like it when they can include an _extremely_ small Node library which itself relies on 4 other _extremely_ small Node libraries, which each rely on another 4 _extremely_ small Node libraries, until you have a node_modules directory with 30,000 packages in it.

We don't know how to write software any more. We know how to sew bits of things together into a loincloth, and we get mad when the loincloth makes us itch. Of course it does, you didn't LOOK at anything you are relying on. You didn't stop to think, even for a moment, that maybe writing the thing instead of trusting someone else with it was even an option.

As we continue doing this, relying on 3rd parties to write the heavy lifting code, we lose the skill to write such code ourselves. We are transferring the skills that we need into the hands of people that we can't trust, when viewed from a high level.

We need to get back to small applications, which we know because we wrote them in their entirety, and that someone else can trust because the code for the thing is 2500 lines long and has extremely few, if any, dependencies.

We need to get away from software which imports 30,000 (or even 100) third party libraries with implicit trust because it's open source.

"All bugs are shallow with enough eyes" requires that people use their eyes.

Post reply on HN