Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the xz open source attack

research.swtch.com

Post reply on HN
Print view View original

191–200 of 482 posts

Re: Timeline of the xz open source attack

#191

Post by peter_d_sherman »

My takeaways:

First from the article itself:

>"At this point Lasse seems to have started working even more closely with Jia Tan. Evan Boehs observes that Jigar Kumar and Dennis Ens both had nameNNN@mailhost email addresses that never appeared elsewhere on the internet, nor again in xz-devel."

That is an important observation!

Takeaway: Most non-agenda driven actual people on the Internet, leave a trail -- an actual trail of social media and other posts (and connected friends) that could be independently verified by system/social media/website administrators via voice or video calls (as opposed to CAPTCHA or other computer-based "Is it a human?" tests, which can be gamed) for stronger confidences in the identity / trustworthiness of the remote individual at the other end of a given account...

Next takeaway from the linked article: (https://boehs.org/node/everything-i-know-about-the-xz-backdo...):

>"AndresFreundTec @AndresFreundTec@mastodon.social writes:

Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd,

showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol.

Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates."

Takeaway: We could always use more observability of where exactly CPU time is spent in given programs...

Binaries without symbol tables (which is the majority of programs on computers today) make this task challenging, if not downright impossible, or at least very impractical -- too complex for the average user...

Future OS designers should consider including symbol tables for all binaries they ship -- as this could open up the capability for flame graphs / detailed CPU usage profiling (and the subsequent ability to set system policies/logging around these) -- for mere mortal average users...

Re: Timeline of the xz open source attack

#192

Post by tredre3 »

jhoechtl wrote: post #16

> merges hidden backdoor binary code well hidden inside some binary test input files. [...] Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves. So much for the folks advocating for binary (driver) blobs in OSS t support otherwise unsupported hardware. It's either in source form and reproducable or it's not there.

Are you running linux-libre?

Re: Timeline of the xz open source attack

#193

Post by psanford »

One big take away for me is that we should stop tolerating inscrutable code in our systems. M4 has got to go! Inscrutable shell script have got to go!

Its time to stop accepting that the way we've done this in the past is the way we will continue doing it ad infinitum.

Re: Timeline of the xz open source attack

#194

Post by Denvercoder9 »

edg5000 wrote: post #70

I wonder, once the attacker gained commit permissions, were they able to rewrite and force push existing commits? In that case rolling back to older commits may not be a solution. If my speculation is correct then the the exact date on which access was granted must then first be known, after that a trusted backup of the repo from before that date is needed. Ideally Lasse Collin would have a daily backup of the repo.…

There are trusted copies of historic releases from third-party sources (at least Linux distributions, but there's probably other sources as well), it's pretty easy to check whether the tags in the git repository match those. (This can be done as the tarballs are a superset of the files in the git repository, the other way around doesn't work).

Re: Timeline of the xz open source attack

#195

Post by nrawe »

jethro_tell wrote: post #172
dylan604 wrote: post #161

Earlier quoted context omitted.

This is where the but FOSS is reviewable so it is trusted falls down. This situation is a prime example of how that fallacy is misconstrued. By being FOSS didn't make it trustworthy, it just meant that people had a fighting chance to find out why when something does happen. That's closing the barn door after the horses already left. I'm not knocking FOSS at all. I just think some people have the concept twisted. Just…

You don't think every nation state has people inside private software shops? Especially big tech? Look at stuff getting signed with MS keys, hardware vendors with possible backdoors. Social engineering is social engineering and it can happen anywhere no matter the profit motivation or lack there of. Money interest in software won't save you any more than Foss.

It'd be naive to assume that nation state actors are not trying to penetrate the supply chain at all levels, as it just takes a single weak link in the chain. That weak link could be behind corporate doors or in the open.

The main issue is that this attack shows how a relatively unknown component, as part of a much larger and more critical infrastructure, is susceptible to pressure as a result of "this is a hobby project, lend a hand".

At what point do these components become seen as a utility and in some way adopted into a more mainline, secure, well-funded approach to maintenance? That maintenance can, and probably should, happen in the open, but with the requisite level of scrutiny and oversight worthy of a critical component.

We got very lucky, _this time_.

Re: Timeline of the xz open source attack

#196

Post by coldpie »

jodrellblank wrote: post #157
coldpie wrote: post #76

Earlier quoted context omitted.

It's not a perfect solution, but I think Big Companies could have a role to play here. Some kind of tenure/patronage system, where for every $1B a Big Company makes in profit, they employ one critical open source maintainer with a decent salary (like $200k or something). The job would only have two requirements: 1) don't make everyone so mad that everyone on the Internet asks for you to be fired, and 2) name a suitab…

> " It's not a perfect solution " Is it a solution at all? Say Oracle offer Lasse Collin $200k for maintaining xz but he doesn't want to work for Oracle so he refuses, then what? Amazon offer Lasse $200k but they require fulltime work on other open source packages which he isn't experienced with or interested in, so he refuses, then what? Google employ someone else for $200k but they can't force Lasse Collin to hand…

I think you've misunderstood my suggestion. I said the job has two responsibilities. No more. You added a bunch of other responsibilities, I'm saying those wouldn't be allowed. It would be in the employment agreement that is purely payment for doing the maintainership tasks they were already doing. It would be a culture expectation that the company not apply pressure on maintainers.

> but if they just employ 'open source maintainers' scattering their attention over all kinds of random projects they have no prior experience with

They would pay the people who are doing the work now. Under this hypothetical, one of the Big Companies would have hired Lasse as the xz maintainer, for example. His job responsibilities are to maintain xz as he had been doing, and identify a successor when he's ready to move on. Nothing else.

> So it's a sinecure position, doing nothing is the safest thing?

No. Not doing the maintenance tasks would make everyone mad, violating one of the two job responsibilities.

> How could Lasse Collin have named a more suitable replacement than someone who seemed technically capable, interested in the project, and motivated to work on [it]

Lasse would suggest Tan as a suitable replacement. Big Company's hiring pipeline would approach Tan and start the hiring process (in person interviews, tax docs, etc etc). At some point they would realize Tan isn't a real person and not hire him. Or, the adversary would have to put up "a body" behind the profile to keep up the act, which is a much higher bar to clear than what actually happened.

Re: Timeline of the xz open source attack

#197

Post by kibwen »

psanford wrote: post #193

One big take away for me is that we should stop tolerating inscrutable code in our systems. M4 has got to go! Inscrutable shell script have got to go! Its time to stop accepting that the way we've done this in the past is the way we will continue doing it ad infinitum.

That's a great first step, but ready your pitchforks for this next take, because the next step is to completely eliminate Turing-complete languages and arbitrary I/O access from standard build systems. 99.9% of all projects have the capability to be built with trivial declarative rulesets.

Re: Timeline of the xz open source attack

#198

Post by deathanatos »

> Evan Boehs observes that Jigar Kumar and Dennis Ens both had nameNNN@mailhost email addresses

This is the second time I've read this "observation", but this observation is just wrong? Jigar's email is "${name}${number}@${host}", yes, but Dennis's is just "${name}@${host}" — there's not a suffixed number. (There's a 3, but it's just a "leetcode" substitution for the E, i.e., it's semantically a letter.)

(They could still both be sockpuppets, of course. This doesn't prove or disprove anything. But … let's check our facts?)

Re: Timeline of the xz open source attack

#199

Post by Denvercoder9 »

sloowm wrote: post #138
chubot wrote: post #103

Something to add to the timeline: when did this avenue of attack become available? It only happened in the last 10 years apparently. Why do sshd and xz-utils share an address space? When was the sshd -> systemd dependency introduced? When was the systemd -> xz-utils dependency introduced? --- To me this ARCHITECTURE issue is actually bigger than the social engineering, the details of the shell script, and the details…

Another point relevant on the timeline is when downstream starts using binaries instead of source. I think people are flying past that important piece of the hack. Without that this would not have been possible. If there is a trusted source in the middle building the binaries instead of the single maintainer and the hacker this attack becomes extremely hard to slip by people.

> Another point relevant on the timeline is when downstream starts using binaries instead of source.

No downstream was using binaries instead of source. Debian and Fedora rebuild everything from source, they don't use the binaries supplied by the maintainer. The backdoor was inserted into the build system.

Re: Timeline of the xz open source attack

#200

Post by unethical_ban »

tommiegannert wrote: post #29
lukaslalinsky wrote: post #24

The social side of this is really haunting me over the last days. It's surprisingly easy to pressure people to giving up control. I've been there myself. I can't even imagine how devastating this must be to the original author of XZ, especially if he is dealing with other personal issues as well. I hope at least this will serve a strong example to other open source people, to never allow others to pressure them into…

The Jigar Kumar nudges are so incredibly rude. I would have banned the account, but perhaps they contributed something positive as well that isn't mentioned. I wonder if it would be possible to crowdsource FOSS mailing list moderation.

It seems from the reading of this article that jigar is in on the scam. That said, I agree.
Post reply on HN