Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Could the XZ backdoor been detected with better Git/Deb packaging practices?

optimizedbyotto.com

Post reply on HN
Print view View original

61–70 of 116 posts

Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?

#61

Post by imoverclocked »

Sophira wrote: post #60
jacquesm wrote: post #51

Earlier quoted context omitted.

This is not going to be popular: I think the whole idea that a build system just fetches resources from outside of the build environment is fundamentally broken. It invites all kinds of trouble and makes it next to impossible to really achieve stability and to ensure that all code that is in the build has been verified. Because after you've done it four times the fifth time you won't be looking closely. But if you do…

I completely agree with you - I think that automatic downloading of dependencies when building is a bad idea. However, for the sake of devil's advocacy, I do also want to point out that the first thing a lot of people used to do after downloading and extracting a source tarball was to run "./configure" without even looking at what it is they were executing - even people who (rightly) hate the "curl | bash" combo. You…

Adding sudo in front makes it secure though, right? /sarcasm

--

Automatic downloading of dependencies can be done in a sane way but not sane without significant effort. eg: building Debian packages can install other pre-packaged dependencies. In theory other packages are built the same way.

Where this becomes an issue specifically is where language-specific mechanisms reach-out and just install dependencies. To be fair, this has happened for a long time (I'm looking at you, CPAN) and does provide a lot of potential utility to any given developer.

What might be better than advocating for "not doing this at all" is "fixing the pattern." There are probably better ideas than this but I'll start here:

1) make repositories include different levels of software by default. core vs community at an absolute minimum. Maybe add some levels like, "vetted versions" or "gist" to convey information about quality or security.

2) make it easy to have locally vetted pools to choose from. eg: Artifactory makes it easy to locally cache upstream software repos which is a good start. Making a vetting process out of that would be ... useful but cumbersome.

At the end of the day, we are always running someone else's code.

Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?

#62

Post by lmm »

mananaysiempre wrote: post #47
dijit wrote: post #40

Earlier quoted context omitted.

I thought that the exploit was not injected into the Git repository on GitHub at all, but only in the release tarballs. And that due to how Autoconf & co. work, it is common for tarballs of Autoconf projects to include extra files not in the Git repository (like the configure script). I thought the attacker exploited the fact that differences between the release tarball and the repository were not considered particul…

First of all, even if that were true, that wouldn’t have much to do with hermetic builds as I understand the term. You could take the release tarball and build it on an air-gapped machine, and (assuming the backdoor liked the build environment on the machine) you would get a backdoored artifact. Fetching assets from the Internet (as is fashionable in the JavaScript, Go, Rust, and to some extent Python ecosystems) doe…

> Unconditionally autoreconfing things would be better, but risk breakage because the backwards compatibility story in Autotools has historically not been good, because they’re not supposed to be used that way.

Yes and no. "make -f Makefile.cvs" has been a supported workflow for decades. It's not what the "build from source" instructions will tell you to do, but those instructions are aimed primarily at end users building from source who may not have M4 etc. installed; developers are expected to use the Makefile.cvs workflow and I don't think it would be too unreasonable to expect dedicated distro packagers/build systems (as distinct from individual end users building for their own systems) to do the same.

Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?

#63

Post by tedunangst »

kragen wrote: post #57
jiggawatts wrote: post #50

Something that the XZ back door made me realise is that the fundamental difference between proprietary and open source software is not the price or source availability for most of its users — no not developers! — it is the reputation and protected brand of the former and the anonymity of the latter. We have no clue who “Jia Tan” is, a name certain to be a pseudonym. Nobody has seen his face. He never provided ID to a…

I think the difference is that the undoubtedly numerous times that this has happened with Microsoft and other proprietary-software vendors, the users weren't in a position to find out.

Why not? This wasn't found by source review. The computer was slow, somebody looked into why. The bug was discovered via analysis of binary artifacts, and only then traced back to the source. Bruce Dawson does this all the time on Windows.

https://randomascii.wordpress.com/category/uiforetw-2/

Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?

#64

Post by lmm »

1718627440 wrote: post #46
johnny22 wrote: post #45

Earlier quoted context omitted.

many folks make a distinction between source available and open source.

The latter meaning: accepts patches, or what?

The latter meaning the four freedoms or something equivalent (e.g. complying with the OSD and/or the the DFSG). They don't have to accept patches upstream but they do have to permit sharing your patches with other users one way or another.

Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?

#65

Post by lmm »

charcircuit wrote: post #6

It shouldn't have happened in the first place. OpenSSH should control their exact dependencies and Debian shouldn't be meddling with them and swapping them out, loading random code into OpenSSH's process. >we can only trust open source software. There is no way to audit closed source software The ability to audit software is not sufficient, nor neccessary for it to be trustworthy. >systems of a closed source vendor w…

> You can't audit open source vendors either.

You can audit a lot of Debian's infrastructure - their build systems are a lot more transparent than the overwhelming majority of software vendors (which is not to say there isn't still room for improvement). You can also skip their prebuilt packages and build everything on your own systems, which of course you then have the ability to audit.

Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?

#66

Post by dataflow »

jacquesm wrote: post #51
dijit wrote: post #34

Earlier quoted context omitted.

thats not the issue, there will always be prebuilt binaries (hell, deb/rpm are prebuilt binaries). The issue for xz was that the build system was not hermetic (and sufficiently audited). Hermitic build environments that can’t fetch random assets are a pain to maintain in this era, but are pretty crucial in stopping an attack of this kind. The other way is reproducible binaries, which is also very difficult. EDIT: Wel…

This is not going to be popular: I think the whole idea that a build system just fetches resources from outside of the build environment is fundamentally broken. It invites all kinds of trouble and makes it next to impossible to really achieve stability and to ensure that all code that is in the build has been verified. Because after you've done it four times the fifth time you won't be looking closely. But if you do…

> I think the whole idea that a build system just fetches resources from outside of the build environment is fundamentally broken

I think your phrasing is a bit overbroad. There's nothing fundamentally broken with the build system fetching resources; what's broken is not verifying what it's fetching. Audit the package beforehand and have your build system verify its integrity after downloading, and you're fine.

Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?

#67

Post by imoverclocked »

crote wrote: post #30
goodpoint wrote: post #21

Earlier quoted context omitted.

The author is incorrect. Keeping the packaging files under git is done out of convenience but it does not help for security and reproducibility. The packages uploaded in Debian are what matters and they are versioned.

And how are you supposed to verify that the right packages have been uploaded? The easiest way to verify that is by using a reproducible automated pipeline, as that moves the problem to "were the packaging files tampered with". How do you verify the packaging files? By making them auditable by putting them in a git repository, and for example having the packager sign each commit. If a suspicious commit slips in, it'l…

> The easiest way to verify that is by using a reproducible automated pipeline

Conversely, this is also an attack surface. It can be easy to just hit "accept" on automated pipeline updates.

New source for bash? Seems legit ... and the source built ... "yeah, ok."

Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?

#68

Post by dhx »

ottoke wrote: post #2

How did the changes in the binary test files tests/files/bad-3-corrupt_lzma2.xz and tests/files/good-large_compressed.lzma, and the makefile change in m4/build-to-host.m4) manifest to the Debian maintainer? Was there a chance of noticing something odd?

There's a few obvious gaps, seemingly still unsolved today:

1. Build environments may not be adequately sandboxed. Some distributions are better than others (Gentoo being an example of a better approach). The idea is that the package specification specifies the full list of files to be downloaded initially into a sandboxed build environment, and scripts in that build environment when executed are not able to then access any network interfaces, filesystem locations outside the build environment, etc. Even within a build of a particular software package, more advanced sandboxing may segregate test suite resources from code that is built so that a compromise of the test suite can't impact built executables, or compromised documentation resources can't be accessed during build or eventual execution of the software.

2. The open source community as a whole (but ultimately in the hands of distribution package maintainers) are not being alerted to and apply caution for unverified high entropy in source repositories. Similar in concept to nothing-up-my-sleeve numbers.[1] Typical examples of unverified high entropy where a supply chain attack can hide payload: images, videos, archives, PDF documents etc in test suites or bundled with software as documentation and/or general resources (such as splash screens in software). It may also include IVs/example keys in code or code comments, s-boxes or similar matrices or arrays of high entropy data which may not be obvious to human reviewers how the entropy is low (such as a well known AES s-box) rather than high and potentially undifferentiated from attacker shellcode. Ideally when a package maintainer goes to commit a new package or package update, are they alerted to unexplained high entropy information that ends up in the build environment sandbox and required to justify why this is OK?

[1] https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number

Re: Could the XZ backdoor been detected with better Git/Deb packaging practices?

#70

Post by schuyler2d »

jiggawatts wrote: post #58
kragen wrote: post #57

Earlier quoted context omitted.

I think the difference is that the undoubtedly numerous times that this has happened with Microsoft and other proprietary-software vendors, the users weren't in a position to find out.

Okay, how , could someone like Jia Tan sneak code into a codebase where commits can only be made by authenticated users with staff accounts on a private network? Versus… a random email offers to help, someone says “sure!”, and… that’s it. That’s the entire hurdle. Google did discover a Chinese hacker working for them on the payroll. That kind of thing does occur, but it’s rare. It’s massively harder and more risky.

Well, xz is a rare event too.

There's no knowing how many backdoors were added by small network companies or contractors. But there's rarely accountability when it happens because the company would rather cover it up, or just not ask too many questions about that weird bug

Post reply on HN