Reflections on Distrusting xz • Hacker News Board

Re: Reflections on Distrusting xz

#291

Post by kevans91 » Wed, Apr 03, 2024, 5:29 PM UTC

Gormo wrote: post #266

constantcrying wrote: post #62

Earlier quoted context omitted.

>Which raises the concerning question of how much more sleeper maintainers there are. Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. For an intelligence operation it is also extremely cheap, you just need a few knowledgeable developers spending some time each week on the project. The upside being a backdoor into a significant portion of infrastructure, the downside being wast…

> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. This particular case seems to be an example of the exact opposite. It took "Jia Tan" two years to conduct all of the social engineering necessary to get into a position to introduce the backdoor, then upon doing so, was caught almost immediately, with the initial discovery not even coming from a dedicated security researcher, b…

fwiw, characterizing Andres as a sysadmin isn't really the whole picture; he's a postgres developer that conducts benchmarking operations with some frequency (and he's quite good at what he does)... he's perhaps naturally a bit more sensitive to things like the cumulative effect of 500ms or so over a number of sshd invocations.

Re: Reflections on Distrusting xz

#292

Post by sherburt3 » Wed, Apr 03, 2024, 5:45 PM UTC

constantcrying wrote: post #62

irdc wrote: post #8

One thing that comes to mind is that “Jia Tan” might be more accurately seen as a “sleeper” of some sort: a foot soldier who infiltrates a juicy open source project and waits for further instructions; backdooring sshd might not have been part of the original plan. Which raises the concerning question of how much more sleeper maintainers there are.

>Which raises the concerning question of how much more sleeper maintainers there are. Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. For an intelligence operation it is also extremely cheap, you just need a few knowledgeable developers spending some time each week on the project. The upside being a backdoor into a significant portion of infrastructure, the downside being wast…

[dead]

Re: Reflections on Distrusting xz

#293

Post by logro » Wed, Apr 03, 2024, 6:09 PM UTC

loftsy wrote: post #86

I took a look at the diff linked in the article with code that "we are all running". The top of the diff certainly looks interesting. They remove the bounds check in dict_put() and add a safe version dict_put_safe(). This kind of change is difficult to make without mistakes because it silently changes the assumptions made when code calling dict_put() was originally written. ALL call sites would need to be audited to…

This is 100% malicious or novice coder. And we surely know it's not the latter.

If you need an unsafe call, you add a dict_put_unsafe(). That again should of course be rejected in a code review.

Re: Reflections on Distrusting xz

#294

Post by HankB99 » Wed, Apr 03, 2024, 6:20 PM UTC

dullcrisp wrote: post #279

david_draco wrote: post #202

Earlier quoted context omitted.

> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. I read it exactly the other way around: the infiltration took years and detecting it was the default with a fuzzer, which had to be disabled for the exploit to succeed. It speaks to the hardening and that hardening should be required more. And of course the precarious roles of maintainers, which have been discussed elsewhere.

The exploit could be detected with a fuzzer perhaps. The infiltration seems like it was something that would be very easy for a funded intelligence agency to do, and would be nigh undetectable if they had been subtly introducing bugs rather than shipping a sophisticated backdoor to every Linux distro. You have to assume if this was an intelligence agency they didn’t burn their only agent like this.

> they didn’t burn their only agent like this.

I think I'd characterize that as "their only identity." What's the chance that some number (>1) of actual agents were sharing this identity? If that's the case, I'd extrapolate that to multiple identities. In fact the social engineering to gain the trust of the original maintainer likely involved several identities.

I suspect that detectability of intentionally injected bugs would be very low.

Re: Reflections on Distrusting xz

#295

Post by acdha » Wed, Apr 03, 2024, 6:32 PM UTC

Gormo wrote: post #287

acdha wrote: post #285

Earlier quoted context omitted.

The first sentence was a category error. It is a national security issue as long as systems which national security depend on are running the code in question. I totally agree on the FOSS side, but consider that the proposal could be as simple as having, say, NIST give the Linux foundation money annually to pay a supporting team of developers rather than the government taking over maintenance on anything.

> It is a national security issue as long as systems which national security depend on are running the code in question. This is itself a category error. By this standard anything and everything upstream of certain government agencies implicates a "national security", and you might as well say that the supply chain for staplers is a national security issue because government officials staple documents together, or th…

> you might as well say that the supply chain for staplers is a national security issue because government officials staple documents together, or that the manufacture of socks is a national security issue because government agents wear socks.

You’re leaving out the key part: this only works if there’s a way for a flaw in those staples or socks to impact national security functions. Once you correctly make the analogy you can easily see why that’s true for a server operating system but not either of your examples.

Re: Reflections on Distrusting xz

#296

Post by berkes » Wed, Apr 03, 2024, 7:26 PM UTC

david_allison wrote: post #122

berkes wrote: post #36

Earlier quoted context omitted.

I once got a (probably scam) offer for adding a cryptominer to a library that I maintained at that time. And a more serious offer to add trackers to a popular >1M installs app. Both cases I obviously ignored it. But it made me aware of a nasty attack vector: someone who's thanklessly building a wordpress-plugin, pip, npm, or whatever software, thanklessly dealing with issues, PRs, support, maintainence, often for no…

> suddenly gets offered three figure sums to add a few lines of "affiliate stuff" or such Back of the envelope calculation: you're looking at 2 orders of magnitude more money from "affiliate stuff" than you would be from generous user donations

Well, yes. But it's also something you can do once. When (not if) it comes out, all credibility is lost.

Whereas donations, regardless of how puny, are recurring and potentially forever.

Re: Reflections on Distrusting xz

#297

Post by berkes » Wed, Apr 03, 2024, 7:54 PM UTC

constantcrying wrote: post #46

berkes wrote: post #2

Full on tinfoil hat here. But warranted and practical. I'm wondering what fallout we'll see from this backdoor in the coming weeks, months or years. Was the backdoor used on obscure build servers or obscure pieces of build infrastructure somewhere? Lying dormant for a moment in future to start injecting code into built packages maybe? Are distro's going to go full-on tinfoil-hat and lock down their distribution, halt…

>Are software developers (finally?) going to remove dependencies(now proven to be liabilities!), causing months of refactoring and rewriting, without any other progress? How is that even a possibility? xz was very useful software, which can only exist if people with significant knowledge put effort into it. Not every OSS project has the ability or resources to duplicate that. The same goes for many, many other depend…

Not all dependecies are "xz" complexity.

Minimizing dependencies, probably means keeping a few libs things like crypto or compression and such.

But do you need a library to color console output? Even if colored console output is a business critical feature, you don't need a (tree of) dependencies for that. I see so many, rather trivial software that comes with hundreds or thousands of dependencies, it's mind boggling really. Why have 124million people downloaded a rubygem that loads and parses a .env file, something I do in a bash oneliner? Why do 21k public npm packages depend on a library that does "rm -f"?

The answer, I'm afraid is mostly that people don't realize this isn't just some value added to their project but rather a liability.

Some liabilities are certainly worth it. XZ is probably one of them. But a Library that does "rm -f" certainly isn't.

Re: Reflections on Distrusting xz

#298

Post by cb321 » Wed, Apr 03, 2024, 7:59 PM UTC

ptx wrote: post #262

nottorp wrote: post #144

Earlier quoted context omitted.

> If a system has been infiltrated, you can't trust any part of it. So it's better to discard any part that has been reached or possibly affected. systemd! let's discard systemd!

They just added an example to the documentation[0] of how to implement the sd_notify protocol without linking to libsystemd, so a little bit of discarding systemd (or at least parts of it) does seem to be part of the solution. [0] https://github.com/systemd/systemd/pull/32030/files

People have been bandying about "10 lines of C", but I'm curious if you know why the protocol is not "2 characters" of shell, namely ":>PATH" (ok, ok, PATH is probably something like /run/serviceName/I-B-ready). At the user (i.e. service daemon) -level this seems much simpler. (EDIT: and systemd would unlink the file as soon as it "gets the message", of course.)

There's just a 40 year culture of using some "official" lib to implement socket protocols - even if the docs suggest you roll your own. I feel like file creation escapes that "reach-for-the-official-lib TCP/UDP/datagram" culture.

It's probably not harder for systemd either if they just use/require the Linux inotify and incorporate that into its select or poll or whatever. I mean, if they wanted to be portable to non-inotify kernels some timeouts/stat-loop would be an ok fallback that would probably be rarely-to-never needed.

It sounds like it's not even hard to add this simpler channel in after the fact just as an alternative option for `whateverd` and then deprecate the datagram one for 10 years (if they even care to).

Re: Reflections on Distrusting xz

#299

Post by brabel » Wed, Apr 03, 2024, 8:30 PM UTC

reisse wrote: post #61

brabel wrote: post #50

Earlier quoted context omitted.

Everyone working on important open source code should have a real identity associated with them. The fact that "Jia Tan" was able to become a maintainer without anyone ever trying to figure out their real identity shows a huge weakness in our trust model in OSS (everyone real would have something like a Linked In page, Facebook, Twitter, Instagram, or better, their own website with stuff that could be used to ensure…

No-one working on open source code on their time owes anything to anyone using the code. If you want an important open source project to be maintaned by non-anonymous person, surprise-surprise, hire that person and pay them. Besides, some of the best open source contributors I know are almost-anonymous people behind nicknames and anime girls avatars.

It is not about owing someone... it's about having provenance of code.

If you're just an anonymous guy doing stuff for free and want to remain anonymous, that's fine, but then your software shouldn't be used by anyone who cares about toolchain attacks as there's just no way to trust you, and no way to verify every single commit you make on new releases.

For software that gets used by many, which is a goal of OSS (otherwise just don't even bother to publish stuff, what's the point?), there needs to be a face behind it.

I do agree with others that identity is a hard problem, but people here are pretending there's no solution to that (or misinterpreting what I wrote to mean people should have a Facebook or Twitter account, which is absolutely not what I was trying to say - I just mentioned the most popular websites real people are likely to be found on, as that could be used to prove their identity... for example, I have a Keybase account where my proof of identity, which is tied to my public keys, can be found on my GitHub profile - but they let you choose Facebook or Twitter for that purpose as well) when obviously there is. I should know, I work on this space.

Re: Reflections on Distrusting xz

#300

Post by brabel » Wed, Apr 03, 2024, 8:40 PM UTC

acdha wrote: post #163

brabel wrote: post #50

Earlier quoted context omitted.

Everyone working on important open source code should have a real identity associated with them. The fact that "Jia Tan" was able to become a maintainer without anyone ever trying to figure out their real identity shows a huge weakness in our trust model in OSS (everyone real would have something like a Linked In page, Facebook, Twitter, Instagram, or better, their own website with stuff that could be used to ensure…

> everyone real would have something like a Linked In page, Facebook, Twitter, Instagram, or better, their own website with stuff that could be used to ensure they're a real person Have you seen the campaigns people have run building fake LinkedIn profiles and slowly adding “connections”? There was one a few years ago which roped in a lot infosec people who should have known better and it’s gotten much worse with AI…

I believe the solution to identity on the Internet needs to be tied to governments, that's unfortunate but in real life, that's always been the case and I can see no alternative here. Blockchain is a pipedream and no serious work is going to associate a person's identity to a key which cannot be revoked, cannot be recovered in case of "loss", can be tracked on a public ledger etc. etc...

But there's actual good work going on in the identity industry, like Verifiable Credentials, so this will become a reality soon: you will be able to verify someone's identity as long as you trust the issuer of their "credential" (which in the case here would mean basically a username and a public key or reference to a JWKS which can be used to verify the signature of the person, very much like the digital version of an identity card which can be used to check the signature on some piece of paper, but actually cyptographically safe)... so you would need to add a few governments to your list of "approved issuers", or something more indirect like universities (which themselves would rely on the government-issued identity) or traffic authorities (if you rely on driving licenses). Sure, Governments can lie, and people go to great lengths to steal others' identities in real life, but in the current world, we're still able to get bank accounts, passports etc. based on this model... just because the system is not perfect doesnt' mean it's not good enough, specially when there's no better alternative at all.