I believe the XZ compromise partly stemmed from including binary files in what should have remained a source-only project. From what I remember, well-run projects such as those of the GNU project have always required that all binaries—whether executables or embedded data such as test files—be built directly from source, compiling a purpose-built DSL if necessary. This ensures transparency and reproducibility, both of which might have helped catch the issue earlier.
Could the XZ backdoor been detected with better Git/Deb packaging practices?
31–40 of 116 posts
Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?
#32Earlier quoted context omitted.
Ahh yes, if only the commit message was better. That would have stopped the xz attack.
It's not just the commit message, but the fact that it's a single commit with >100k lines changed (though, if that's just a merge commit, it might be not super unusual for that kind of workflow. Though big merge commits are a good place to hide things in git, given that they can introduce their own changes)
> It's not just the commit message, but the fact that it's a single commit with >100k lines changed
When you diff the `bash-5.3` tag against the `bash-5.3-rc2` tag, the set of changes is reduced by a ton. It's the same story with previous release commits (at least for as far as I care to go back)... there's a "next version" branch that gets tagged with alpha, beta, and rc releases, and then there's a release commit that's made on master with the changes from the "next version" branch, plus some additional changes.
Why do they do things this way? I have no idea, but it clearly seems to work for them.
Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?
#33Earlier quoted context omitted.
mostly no, from my reading - it was a multi-stage chain of relatively normal looking things that added up to an exploit. helped by the tests involved using compressed data that wasn't human-readable. you can of course come up with ways it could have been caught, but the code doesn't stand out as abnormal in context. that's all that really matters, unless your build system is already rigid enough to prevent it, and ha…
The article references a technical write-up: https://research.swtch.com/xz-script
ah, yes, this is one I remember seeing early on! thank you! I couldn't find much past the blogspam this time :/
Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?
#34I believe the XZ compromise partly stemmed from including binary files in what should have remained a source-only project. From what I remember, well-run projects such as those of the GNU project have always required that all binaries—whether executables or embedded data such as test files—be built directly from source, compiling a purpose-built DSL if necessary. This ensures transparency and reproducibility, both of…
thats not the issue, there will always be prebuilt binaries (hell, deb/rpm are prebuilt binaries).
The issue for xz was that the build system was not hermetic (and sufficiently audited).
Hermitic build environments that can’t fetch random assets are a pain to maintain in this era, but are pretty crucial in stopping an attack of this kind. The other way is reproducible binaries, which is also very difficult.
EDIT: Well either I responded to the wrong comment or this comment was entirely changed. I was replying to a comment that said. “The issue was that people used pre-built binaries” which is materially different to what the parent now says, though they rhyme.
Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?
#35I believe the XZ compromise partly stemmed from including binary files in what should have remained a source-only project. From what I remember, well-run projects such as those of the GNU project have always required that all binaries—whether executables or embedded data such as test files—be built directly from source, compiling a purpose-built DSL if necessary. This ensures transparency and reproducibility, both of…
thats not the issue, there will always be prebuilt binaries (hell, deb/rpm are prebuilt binaries). The issue for xz was that the build system was not hermetic (and sufficiently audited). Hermitic build environments that can’t fetch random assets are a pain to maintain in this era, but are pretty crucial in stopping an attack of this kind. The other way is reproducible binaries, which is also very difficult. EDIT: Wel…
The XZ project’s build system is and was hermetic. The exploit was right there in the source tarball. It was just hidden away inside a checked-in binary file that masqueraded as a test for handling of invalid compressed files.
(The ostensibly autotools-built files in the tarball did not correspond to the source repository, admittedly, but that’s another question, and I’m of two minds about that one. I know that’s not a popular take, but I believe Autotools has a point with its approach to source distributions.)
Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?
#36I believe the XZ compromise partly stemmed from including binary files in what should have remained a source-only project. From what I remember, well-run projects such as those of the GNU project have always required that all binaries—whether executables or embedded data such as test files—be built directly from source, compiling a purpose-built DSL if necessary. This ensures transparency and reproducibility, both of…
This was a devops exploit because they used the same env for building the app as they did for the test code. Many miss this entirely and think it is because a binary was shipped.
Ideally a test env and a build env should be entirely isolated should the test code some how modify the source. Which in this case it did.
Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?
#37Earlier quoted context omitted.
Compression algorithms are deterministic over fixed data though (possibly with some effort). There's no good reason to have opaque, non generated data in the repository and it should certainly be a red flag going forwards.
committed files with carefully crafted bad data is extremely common for testing how your code handles invalid data, especially with regression tests. and lzma absolutely needs to test itself against bad, possibly-malicious data.
Yes, but the carefully crafted bad data should be explainable.
Instead of committing blobs, why not commit documented code which generates those blobs? For example, have a script compress a bunch of bytes of well-known data, then have it manually corrupt the bytes belonging to file size in the archive header.
Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?
#38>Can we trust open source software? Yes — and I would argue that we can only trust open source software. But should we trust it? No!! That's why we're here! I'm not satisfied with the author's double-standard-conclusion. Trust, but verify does not have some kind of hall pass for OSS "because open-source is clearly better." Trust, but verify is independent of the license the coders choose.
Yes, I would say that being able to view the source code and build it yourself is a necessary but not sufficient condition of properly trusting the software. (which is not quite the same thing as it being open source, but it's relatively rare outside of being a very big customer that you can do this for non-open-source code).
When you get the source code as a big costumer, that is open source. It might even be free software.
Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?
#39Earlier quoted context omitted.
committed files with carefully crafted bad data is extremely common for testing how your code handles invalid data, especially with regression tests. and lzma absolutely needs to test itself against bad, possibly-malicious data.
I agree, but perhaps OP is suggesting that the hand-crafted data can be generated in a more transparent way. For example, via a script/tool that itself can be reviewed.
could have, absolutely.
should not have been in any commit, which is basically necessary to prevent this case, almost definitely not. it's normal, and requiring all data to be generated just means extremely complicated generators for precise trigger conditions... where you can still hide malicious data. you just have to obfuscate it further. which does raise the difficulty, which is a good thing, but does not make it impossible.
I completely agree that it's a good/best practice, but hard-requiring everywhere it has significant costs for all the (overwhelmingly more common) legitimate cases.
Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?
#40Earlier quoted context omitted.
thats not the issue, there will always be prebuilt binaries (hell, deb/rpm are prebuilt binaries). The issue for xz was that the build system was not hermetic (and sufficiently audited). Hermitic build environments that can’t fetch random assets are a pain to maintain in this era, but are pretty crucial in stopping an attack of this kind. The other way is reproducible binaries, which is also very difficult. EDIT: Wel…
The XZ project’s build system is and was hermetic. The exploit was right there in the source tarball. It was just hidden away inside a checked-in binary file that masqueraded as a test for handling of invalid compressed files. (The ostensibly autotools-built files in the tarball did not correspond to the source repository, admittedly, but that’s another question, and I’m of two minds about that one. I know that’s not…
I thought that the exploit was not injected into the Git repository on GitHub at all, but only in the release tarballs. And that due to how Autoconf & co. work, it is common for tarballs of Autoconf projects to include extra files not in the Git repository (like the configure script). I thought the attacker exploited the fact that differences between the release tarball and the repository were not considered particularly suspicious by downstream redistributors in order to make the attack less discoverable.