Live data from Hacker News

Timeline of the xz open source attack

research.swtch.com

381–390 of 482 posts

Re: Timeline of the xz open source attack

#381

Earlier quoted context omitted.

I feel for Lasse. It's time for more of the big vendors who use these projects in their offerings to step up and give people running these small projects more resources and structure. $20k to have maintainers for each project actually meet twice a year at a conference is chump change for the biggest vendors, especially when compared against the cost of the audits they'll now be doing on everything Jia Tan and Co. tou…

As an OSS maintainer, $20k wouldn't help me enough unless I was retired. The issue is not money (or not just money), but time. If a maintainer has a full-time job, they may not have time, and developers/maintainers tend to have full-time jobs, so... Now maybe one could build a career out of OSS maintainerships, with work/time funded by lots of donations much smaller than a salary but amounting to a salary.

I was thinking more of a fix to the issue of "who the hell's maintaining this package our distro/service/whatever is based on" than a way to make money. The bigger projects (like the kernel) and vendors (MS, IBM/Red Hat, Canonical, Google, etc.) all have a vested interest in knowing the actual identity and basic personalities of people who maintain the important packages. If maintainers avail themselves for a weekend at a conference twice a year (or maybe even a lighter commitment like a few short meetings with a manager) they get some resources for their efforts. The flip side of this, of course, is that these organizations will prefer to include packages from maintainers who agree to this arrangement over those who don't.

Furthermore, these organizations are in a place to put experienced, trustworthy contributors on projects that need maintainers if need be. If Lasse had been able to go to, idk, the Linux Foundation and say, "Listen, I'm getting burnt out, got anyone?" and they said "Sure, we've got this contributor with an established record who would love to help maintain your project", none of this is happening right now.

Re: Timeline of the xz open source attack

#382

Earlier quoted context omitted.

Also in security and 100% tired of the code slingers who get annoyed by security reviews Here on HN it’s been derided as “company just checking a box for compliance but adds no functionality. It slows us down when we want to disrupt!” - developer of yet another todo list or photo editor app… Buffer overflows and the like are one thing. Notions from this blog that certain normal files won’t be well reviewed is a bad s…

A security review or snake oil AI black box didn't stop this. It was stopped by a 'code slinger' who noticed a performance regression in ssh. Deloitte coming in a checklist would have NEVER stopped this one.

[dead]

Re: Timeline of the xz open source attack

#383

This attack doesn’t exploit a technical issue or bug, it exploits the open source philosophy, and unless the community will come up with a systematic process to counter it, expect more sophisticated attacks similar to it in the future. This time we got lucky that some smart nerd -I am a nerd too, this is a praise not to be taken in a bad way- noticed and notified the community in less than 20 days of the second backd…

Could've happened just as easily if not more easily with proprietary software.

Re: Timeline of the xz open source attack

#384

Earlier quoted context omitted.

I'm not familiar with how distros get the source code for upstream dependencies. I'm trying to understand what Andres meant when he said this: > One portion of the backdoor is solely in the distributed tarballs Is it that the tarball created and signed by Jia had the backdoor, but this backdoor wasn't present in the repo on github? And the Debian (or any distro) maintainers use the source code from tarball without co…

Exactly. The threat actor had signed and uploaded the compromised source tarball to GitHub as a release artifact. They then applied for an NMU (non-maintainer upload) with Debian, which got accepted, and that's how the tarball ended up on Debian's infrastructure.

Thanks for the extra explanation. I guess this is harder to protect against than I thought and it's more that some distro's got somewhat lucky than debian and fedora doing something that is out of the ordinary.

Re: Timeline of the xz open source attack

#385
post #101
post #78

"It's also good to keep in mind that this is an unpaid hobby project." ~ Lasse Collin, 2022-06-08. As someone working in security, the fact that _foundational_ pieces of the computing/networking rely on motivated individuals and essentially goodwill is mind blowing. There are great aspects to the FOSS movement, but the risks – particularly the social engineering aspects as demonstrated here – and potential blast radi…

That's just the nature of free software- you have to either trust the maintainer or do it yourself. There is not really a way around it. - Corporate maintainers are great until they enshittify things in the pursuit of profit (see Oracle) - Nonprofits are probably the best but can go the same route as corps (see Mozilla) - Hobbyists are great until they burnout (see xz)

While it's interesting to philosophize about alternatives like this and it's seemingly obviously true that there's no trivial solution to maintainership that solves all problems perfectly, I'm a little wary about presenting these flawed approaches as somehow equivalent; I highly doubt they're even remotely equally bad - nor that they have equally big upsides.

Re: Timeline of the xz open source attack

#386
post #195

Earlier quoted context omitted.

It'd be naive to assume that nation state actors are not trying to penetrate the supply chain at all levels, as it just takes a single weak link in the chain. That weak link could be behind corporate doors or in the open. The main issue is that this attack shows how a relatively unknown component, as part of a much larger and more critical infrastructure, is susceptible to pressure as a result of "this is a hobby pro…

> this attack shows how a relatively unknown component why just this one? do we collectively have the memory of a gold fish? just recently, log4j had a similar blast radius. is it because one was seemingly malicious that the other doesn't count?

While blast radius of both is large, there are major differences between them. Log4J was a largely app-level vulnerability affecting Java-based systems.

This vulnerability, had all gone to the attackers plan, would have been present on the major distros next major releases through a key infrastructure component which would have been installed far more widely, IMO.

Another major difference is that Log4J is already part of the Apache Foundation, which means it should have greater oversight/security maintenance anyway, while this is an attack against a solo developer.

It's definitely not to downplay the severity of the Log4J incident, by any means. But they are decidedly different.

Re: Timeline of the xz open source attack

#387

One big take away for me is that we should stop tolerating inscrutable code in our systems. M4 has got to go! Inscrutable shell script have got to go! Its time to stop accepting that the way we've done this in the past is the way we will continue doing it ad infinitum.

Functionality like IFUNC is completely inexcusable outside of dev/debug builds imo. It's rot.

Re: Timeline of the xz open source attack

#388
post #189

Earlier quoted context omitted.

Check the GitHub profile of anybody that commits. Is there a photo of the person? Can you see a commit history and repos that help validate who they seem to be. In this instance, noticing the people emailing to pressure you have fake looking names that start with adjacent letters and the same domain name. Be more paranoid.

> Is there a photo of the person? Does that even matter these days? Especially if we're talking nation state level stuff convincing histories are not hard to create to deflect casual observers. >Be more paranoid. Most people in OSS just want to write some code to do something, not defend the world against evil.

“Does it even matter?”

Yes, it would have prevented this attack. It isn’t totally sufficient but it’s quick and easy and would have prevented this attack.

“Most people don’t want …”

I get it. I think the issue is that pushing junk code from malicious contributors into your project causes more hassle in the long run. If you just want to code and make stuff work, you should probably be careful who you pull from. It’s not just for the benefit of others, it’s first and foremost to protect the code base and the time and sanity of other contributors.

Re: Timeline of the xz open source attack

#389

Earlier quoted context omitted.

Ultimately you're going to have to be adept at stuff like the Underhanded C Contest to spot this kind of thing in any Turing-complete language, so the idea of auditing the source is unreliable at worst. So I'd take another page from the Java/Maven ecosystem and require hashed+signed binaries, with the possible addition of requiring builds to be performed on a trusted remote host so that at least we can verify the bin…

reproducible builds never made sense to me. if you trust the person giving you the hash, just get the binary from them. you don't need to reproduce the build at all. if you trust that they're giving you the correct hash, but not the correct binary, then you're not thinking clearly.

One thing a culture of reproducible builds (and thus stable hashes) however does provide is a lack of excuse as to why the build isn't reproducible. Almost nobody will build from source - but when there's bugs to be squashed and weird behavior, some people, sometimes, will. If hashes are the norm, then it's a little harder for attackers to pull of this kind of thing not because you trust their hash rather than their blob, but rather because they need to publish both at once - thus broadening the discovery window for shenanigans.

To put it another way: if you don't have reproducible builds and you're trying to replicate an upstream artifact then it's very hard to tell what the cause is. It might just be some ephemeral state, a race, or some machine aspect that caused a few fairly meaningless differences. But as soon as you have a reproducible build, then failure to reproduce instantly marks upstream as being suspect.

It's also useful when you're trying to do things like tweak a build - you can ensure you're importing upstream correctly by checking the hash with what you're making, and then even if your downstream version isn't a binary dependent on upstream (e.g. optimized differently, or with some extra plugins somewhere), you can be sure that changes between what you're building and upstream are intentional and not spurious.

It's clearly not a silver bullet, sure. But it's not entirely useless either, and it probably could help as part of a larger ecosystem shift; especially if conventional tooling created and published these by default such that bad actors trying to hide build processes actually stick out more glaringly.

Re: Timeline of the xz open source attack

#390

Earlier quoted context omitted.

Leaving aside issues of how it could work, Lasse Collin wasn't the one who saw this attack and stopped it so how would paying him have helped against this attack? > " Lasse would suggest Tan as a suitable replacement. Big Company's hiring pipeline would approach Tan and start the hiring process (in person interviews, tax docs, etc etc). At some point they would realize Tan isn't a real person and not hire him " What…

> Lasse Collin wasn't the one who saw this attack and stopped it so how would paying him have helped against this attack? Well, there's a few issues I'm trying to target. I'm trying to work backwards from "how do we stop bad actor Tan from getting maintainer access to the project?" Creating an identify-verified relationship (employment) is a good fit for that, I think. And it nicely solves some other related issues w…

> "I'm trying to work backwards from "how do we stop bad actor Tan from getting maintainer access to the project?" Creating an identify-verified relationship (employment) is a good fit for that, I think."

It would stop a sock puppet, but Jai Tan might be a real person, a real developer paid or blackmailed by a hostile group; Amazon might just have hired him and handed over maintainer access to him thinking it was above board, if a problem hadn't been found yet. I don't know where Jai Tan claimed to be from, but it's quite possible they would say "I don't have a passport", "I can't leave my family to travel to America for an in-person interview", "I'm not in good health to travel", "I don't speak English well enough for an in-person interview", "I live in a poor country without a functioning government and have no tax documents", or etc. etc. excuses which are quite plausible

> "Or, if he was feeling burnt out, he may have felt more comfortable passing the torch earlier if there was a clear framework to do so, backed by an entity that can do some of the heavy lifting of naming/validating a successor."

Your suggested $200k is equivalent to £160k GBP in the UK; look at this UK average salary list: https://uk.jobted.com/ no job comes close; not Managing Director, IT director, Finance Director, Aerospace engineer, DevOps engineer, neurosurgeon, nothing on the list is above £110k. Sure there are many people earning that much as a senior devops AI cloud security specialist in a fast paced London based fintech trading house, but the idea that someone would comfortably pass up a salary around the 98th percentile of incomes in the country for like 2 days a month of work because they're "feeling burnt out" is unthinkable. Anyone sensible would hold onto that until they pried it out of one's cold dead hands; American tech salaries are almost literally unbelievable. Even moreso if we consider a maintainer in a poorer country.

> "I tweaked your questions a tiny bit to rephrase them as I interpreted them. I think the spirit of your questions was kept, I apologize if not"

I started writing Tan, but then changed it. A lot of your reply is assuming that we know there were malicious patches and suspect Jigar Kumar was a bad actor and that the big company would be somewhat US friendly. But we can't plan to know all that for all situations like this. Some people will be speculating that the previous paid maintainer was complicit and all their work and merges are now suspect. The billion dollar company who hired Collins in this hypothetical maintainer could be Baidu or Yandex or Saudi Aramco, and then people would be suspicious. It's one thing to have your task be "don't make people mad" but doesn't that change if people getting mad can give you unbounded retrospective work and responsibility?

> "If these tasks fall under the "don't make everyone mad" job responsibility, then yes. [...] Was there actually a real maintenance issue? [...] Or, the community could raise the problem to Big Company, who can do the work of verifying that there is a problem and working with the maintainer to resolve it."

As soon as the internet becomes aware that they can get anything merged ASAP by threatening to get mad, everyone will be mad about everything all the time. Whom at the BigCo will do the work of verifying whether there is a problem? I mean, let's put Lasse Collins on a specific team along with other employees who are expected to work 40-80 hour weeks while he isn't. The pressure on the team manager to ditch the maintainer and distribute his salary among the other team members would be constant. If those other team members see him doing less work for similar or more money it would be a morale killer and they would want to leave. If they also have to know his project well enough to follow all the drama and things people are complaining about and tease out what is and isn't a real problem and coerce him to do his job, sorry 'work with him', well, they won't be very motivated to do that.

Post reply on HN