Live data from Hacker News

Timeline of the xz open source attack

research.swtch.com

311–320 of 482 posts

Re: Timeline of the xz open source attack

#311

Earlier quoted context omitted.

I think you've misunderstood my suggestion. I said the job has two responsibilities. No more. You added a bunch of other responsibilities, I'm saying those wouldn't be allowed. It would be in the employment agreement that is purely payment for doing the maintainership tasks they were already doing. It would be a culture expectation that the company not apply pressure on maintainers. > but if they just employ 'open so…

Leaving aside issues of how it could work, Lasse Collin wasn't the one who saw this attack and stopped it so how would paying him have helped against this attack? > " Lasse would suggest Tan as a suitable replacement. Big Company's hiring pipeline would approach Tan and start the hiring process (in person interviews, tax docs, etc etc). At some point they would realize Tan isn't a real person and not hire him " What…

> Lasse Collin wasn't the one who saw this attack and stopped it so how would paying him have helped against this attack?

Well, there's a few issues I'm trying to target. I'm trying to work backwards from "how do we stop bad actor Tan from getting maintainer access to the project?" Creating an identify-verified relationship (employment) is a good fit for that, I think. And it nicely solves some other related issues with the current volunteer maintainership model. Lasse may not have felt the strong pressure/health issues if he was being paid to do the work. Or, if he was feeling burnt out, he may have felt more comfortable passing the torch earlier if there was a clear framework to do so, backed by an entity that can do some of the heavy lifting of naming/validating a successor.

> What if they find that Tan is a real person but he doesn't want to work for Amazon or legally can't

I think this would be a fairly rare occurrence, but it's one I called out as a potential problem in my original post, yeah ("smaller pool of possible maintainers"). If there isn't a clear successor, I think the maintainer could inform the Big Company that they'd like to move on in the next year or two, and Big Company could maybe find an internal engineer who wants to take over the role. Or maybe this more formal sponsored-maintainership arrangement would create incentives for outside contributors to aim for those positions, so there's more often someone waiting to take over (and then be verified by Big Company's hiring process).

> is there an obligation [for the maintainer] to review all of Tan's historical commits? Would the new maintainer be obliged to review all historic commits or audit the codebase or anything? Would Amazon [fire] Lasse once it became clear that he had let a bad actor commit changes which opened a serious security hole during his tenure as maintainer?

(I tweaked your questions a tiny bit to rephrase them as I interpreted them. I think the spirit of your questions was kept, I apologize if not.) If these tasks fall under the "don't make everyone mad" job responsibility, then yes. If not, then no, to all of these. There are no obligations other than the two I mentioned: don't piss off the community and help name a successor. It's up to the project's community to decide if the maintainer is not meeting their obligations, not the sponsoring Big Company.

> What he was doing already was apparently ignoring issues for months and making people like Jigar Kumar annoyed.

I'm not sure. It seems like Kumar was a bad actor. Was there actually a real maintenance issue? If so, maybe it could have been avoided in the first place by the sponsorship arrangement, like I mentioned at the top of this reply. Or, the community could raise the problem to Big Company, who can do the work of verifying that there is a problem and working with the maintainer to resolve it. Instead what happened here, which was for one burned out guy deciding to hand the keys over to some email address.

Re: Timeline of the xz open source attack

#312

Joe Cooper's take on pressuring project maintainers: https://forum.virtualmin.com/t/dont-panic-ssh-exploit-in-ble... somewhat ironic, but I'd say effective.

I don't even think it's necessarily moderation so much as maintaining these high-traffic projects is now akin to a full-time job minus the pay. At a certain point, companies need to step up and provide funding or engineering work or they should just keep expecting to get owned.

If there is a larger shift to companies paying for the software they rely on that is currently free, it's not going to be companies paying open-source maintainers, it's going to be companies paying other companies that they can sign contracts with and pass legal responsibility to.

Re: Timeline of the xz open source attack

#313

One big take away for me is that we should stop tolerating inscrutable code in our systems. M4 has got to go! Inscrutable shell script have got to go! Its time to stop accepting that the way we've done this in the past is the way we will continue doing it ad infinitum.

That and we need to pay open source maintainers and find new ways to support them.

And all code that gets linked into security critical applications/libraries needs to be covered by under some sort of security focused code review.

So no patching the compression code that openSSL links to with random junk distribution maintainers.

Re: Timeline of the xz open source attack

#314

Two points I'd be interested in discussing. 1. It seems a lot of people are assuming Jia Tian was compromised all along. I haven't seen a reason to believe this was a long play, but rather that they were compromised along the way (and selected for their access). Why play the long game of let's find a lone package, w/ a tired maintainer, and then try to get in good with them vs. Let's survey all packages w/ a tired ma…

I took a look at Jia Tan's early behavior, and I found it to be consistent with being "compromised" from the beginning. They had months of contributions on private repos before forking a test library and making superficial changes to it, and then diving headlong into archival libraries. It all looks set up and I see no evidence of an actual person at any point.

I also think it is more difficult to get away with bribing/beating an existing contributing than you suggest; esp since failure means likely exposure.

Re: Timeline of the xz open source attack

#315
post #3

I think this analysis is more interesting if you consider these two events in particular: 2024-02-29: On GitHub, @teknoraver sends pull request to stop linking liblzma into libsystemd.[1] (not in the article) 2024-03-20: The attacker is now a co-contributor for a patchset proposed to the Linux kernel, with the patchset adding the attacker as a maintainer and mirroring the attacker's activity with gaining the trust ov…

> target embedded systems (such as military transport vehicles

You are giving a lot of credit to that, I have seen military ones with ancient software, even the “new” updated ones are still on ubuntu 18.04 because of some drivers/sdks compatibility, but it isn’t a major issue since most of the times they are not connected to the public internet.

Re: Timeline of the xz open source attack

#316
post #245

Earlier quoted context omitted.

That would be an improvement for sure... but this is not fundamentally a technical problem. Reading the timeline, the root cause is pure social engineering. The technical details could be swapped out with anything. Sure there are aspects unique to xz that were exploited such as the test directory full of binaries, but that's just because they happened to be the easiest target to implement their changes in an obfuscat…

> Is the real problem here handing over substantial trust to an anonymous contributor? Unless there's some practical way of comprehensively solving The Real Problem Here, it makes a lot of sense to consider all reasonable mitigations, technical, social or otherwise. > If a person has more to lose than maintainership then would this have happened? I guess that's one possible mitigation, but what exactly would that loo…

Create an organization of paid professionals who are responsible for maintaining these libraries (and providing support to library users).

It's heartblead all over again (though those were honest mistakes, not an intentional attack

Re: Timeline of the xz open source attack

#317

I am reminded of the infamous Sendmail worm from 1989(?). If this compromised OpenSSHd had become the default across millions of systems all over the world, could a worm-like thing have brought a major chunk of the Internet down? Imagine millions of servers suddenly stuck in a boot-loop, refusing to boot. And all because one owner of a library had some mental health issues. We should not have such SPOFs.

> And all because one owner of a library had some mental health issues

Wrong takeaway.

Re: Timeline of the xz open source attack

#318
post #286

Earlier quoted context omitted.

Most of these could still be covered with IO limited to the project files though. There is a sizable movement in the Rust ecosystem to move all build-time scripts and procedural macros to (be combined to) WASM. This allows you to write turing-complete performant code to cover all use-cases people can reasonably think of, while also allowing trivially easy sandboxing. It's not perfect, for example some build scripts d…

But the xz backdoor didn’t involve a build script that tried to compromise the machine the build was running on. It involved a build script that compromised the code being built. Sandboxing the build script wouldn’t have helped much if at all. Depending on the implementation, it might have prevented it from overwriting .o files that were already compiled, maybe. But there would still be all sorts of shenanigans it co…

Maybe not untenable.

If everything is done carefully enough with reproducible builds, I think using a binary whose hash can be checked shouldn't be a great extension of trust.

You could have multiple independent autobuilders verifying that particular source does indeed generate a binary with the claimed hash.

Re: Timeline of the xz open source attack

#319

I am reminded of the infamous Sendmail worm from 1989(?). If this compromised OpenSSHd had become the default across millions of systems all over the world, could a worm-like thing have brought a major chunk of the Internet down? Imagine millions of servers suddenly stuck in a boot-loop, refusing to boot. And all because one owner of a library had some mental health issues. We should not have such SPOFs.

Impossible here because the exploit was carefully engineered to be unreplayable and NOBUS (nobody but us) so it couldn’t go viral. Even if you intercepted a complete tcp byte trace of the attack there was nothing you could do with that to attack other systems.

Re: Timeline of the xz open source attack

#320

> It's also good to keep in mind that this is an unpaid hobby project. That's the root cause. At some point a corp/gov consortium needs to adopt key projects and hire out the maintainers and give them real power and flexibility, perhaps similar to the way a nation might nationalize key infrastructure. But this is against the core ethos at some level. Freedom and safety can be antagonistic.

You are implying that governments are more technically competent and more trustworthy than open source communities..

I said nothing of the sort.

I'm implying they are richer.

Post reply on HN