Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A supply chain attack on PyTorch

johnstawinski.com

Post reply on HN
Print view View original

71–80 of 109 posts

Re: A supply chain attack on PyTorch

#71

Post by schemescape »

simonw wrote: post #13

The key to this attack is: "The result of these settings is that, by default, any repository contributor can execute code on the self-hosted runner by submitting a malicious PR." Problem: you need to be a "contributor" to the repo for your PR to trigger workflows without someone approving them first. So: "We needed to be a contributor to the PyTorch repository to execute workflows, but we didn’t feel like spending ti…

> I really don't like this aspect of GitHub that people who have submitted a typo fix gain additional privileges on the repo by default.

What privileges do they gain? Anything other than the one you mentioned?

This was definitely not obvious to me, so thanks for bringing it up!

Re: A supply chain attack on PyTorch

#72

Post by bloopernova »

sroussey wrote: post #67
SethMLarson wrote: post #29

Great write-up! There's a few things you can do as either a producer or consumer to thwart this sort of attack: Producers: * Self-hosted infrastructure should not be running anonymous code. PRs should be reviewed before code executes on your infrastructure. Potentially should be a GitHub default when using self-hosted runners? * Permissions for workflows and tokens should be minimal and fine-grained. "permissions: re…

I just wish python would isolate all the pip install stuff and put in the project folder like has been done with nodejs for years.

Python's virtualenv does something similar by keeping all files under one directory.

Re: A supply chain attack on PyTorch

#73

Post by YetAnotherNick »

SethMLarson wrote: post #29

Great write-up! There's a few things you can do as either a producer or consumer to thwart this sort of attack: Producers: * Self-hosted infrastructure should not be running anonymous code. PRs should be reviewed before code executes on your infrastructure. Potentially should be a GitHub default when using self-hosted runners? * Permissions for workflows and tokens should be minimal and fine-grained. "permissions: re…

> PRs should be reviewed before code executes on your infrastructure

Very often local tests results can't be trusted specially for projects with architecture level codes like pytorch. Before merging the test results needs to be checked. And it doesn't require just PR review to be safe, it requires review of all the commits as the contributor is making the changes to fix the testcase. Even if we assume that the maintainer will review each of the commit within a day, it could take weeks or months for the contributor to fix the failing testcase with this and maintainer to be looking at the same PR everyday.

Re: A supply chain attack on PyTorch

#74

Post by mschuster91 »

adnanthekhan wrote: post #64
korhojoa wrote: post #59

Earlier quoted context omitted.

If they're pulling from master instead of from a known version, it could be changed to be malicious, and the next time it is fetched, the malicious version would be used instead. It's a vulnerability.

Oh, you'll like this one then. Until 3 months ago GitHub's Runner images was pulling a package directly from Aliyun's CDN. This was executed during image testing (version check). So anyone with the ability to modify Aliyun's CDN in China could have carried out a pretty nasty attack. https://github.com/actions/runner-images/commit/6a9890362738... Now it's just anyone with write access to Aliyun's repository. :) (p.s.…

Whoa, there's a lot of stuff in there [1] that gets installed straight from vendors, without pinning content checksums to a value known-good to Github.

I get it, they want to have the latest versions instead of depending on how long Ubuntu (or, worse, Debian) package maintainers take to package stuff into their mainline repositories... but creating this attack surface is nuts. Imagine being able to compromise just one of the various small tools they embed, and pivoting from there to all GitHub runners everywhere (e.g. by overwriting /bin/bash or any other popular entrypoint, or even libc itself, with a malware payload).

[1] https://github.com/actions/runner-images/tree/main/images/ub...

Re: A supply chain attack on PyTorch

#75

Post by adnanthekhan »

mschuster91 wrote: post #74
adnanthekhan wrote: post #64

Earlier quoted context omitted.

Oh, you'll like this one then. Until 3 months ago GitHub's Runner images was pulling a package directly from Aliyun's CDN. This was executed during image testing (version check). So anyone with the ability to modify Aliyun's CDN in China could have carried out a pretty nasty attack. https://github.com/actions/runner-images/commit/6a9890362738... Now it's just anyone with write access to Aliyun's repository. :) (p.s.…

Whoa, there's a lot of stuff in there [1] that gets installed straight from vendors, without pinning content checksums to a value known-good to Github . I get it, they want to have the latest versions instead of depending on how long Ubuntu (or, worse, Debian) package maintainers take to package stuff into their mainline repositories... but creating this attack surface is nuts. Imagine being able to compromise just o…

Yeah, the security posture of that repository is kind of a mess (which is why something like https://adnanthekhan.com/2023/12/20/one-supply-chain-attack-... was even possible in the first place).

The balance there is overwhelmingly in favor of usability and having new tools (hence the 1 week deployment cadence). Maybe there is some process they have to go over everything before it makes it into the production pool, but that’s quite an undertaking to perform properly every week.

Re: A supply chain attack on PyTorch

#76

Post by adnanthekhan »

schemescape wrote: post #71
simonw wrote: post #13

The key to this attack is: "The result of these settings is that, by default, any repository contributor can execute code on the self-hosted runner by submitting a malicious PR." Problem: you need to be a "contributor" to the repo for your PR to trigger workflows without someone approving them first. So: "We needed to be a contributor to the PyTorch repository to execute workflows, but we didn’t feel like spending ti…

> I really don't like this aspect of GitHub that people who have submitted a typo fix gain additional privileges on the repo by default. What privileges do they gain? Anything other than the one you mentioned? This was definitely not obvious to me, so thanks for bringing it up!

This is more subtle, but there is an “author_association”field within Actions event contexts that can be one of:

NONE, CONTRIBUTOR, COLLABORATOR, MEMBER, OWNER

There are some cases where people use checks for that as part of gating for workflows that run on pull_request_target/issue_comment, but might confuse contributor and collaborator (which requires explicitly adding someone to the repository). Ultimately this is a misconfiguration on part of the maintainer but another example where fixing a typo can play a part in an attack.

Re: A supply chain attack on PyTorch

#78

Post by schemescape »

adnanthekhan wrote: post #76
schemescape wrote: post #71

Earlier quoted context omitted.

> I really don't like this aspect of GitHub that people who have submitted a typo fix gain additional privileges on the repo by default. What privileges do they gain? Anything other than the one you mentioned? This was definitely not obvious to me, so thanks for bringing it up!

This is more subtle, but there is an “author_association”field within Actions event contexts that can be one of: NONE, CONTRIBUTOR, COLLABORATOR, MEMBER, OWNER There are some cases where people use checks for that as part of gating for workflows that run on pull_request_target/issue_comment, but might confuse contributor and collaborator (which requires explicitly adding someone to the repository). Ultimately this is…

Thanks, that makes more sense than automatically granting privileges. It definitely seems easy to mix up those two terms, however!

Re: A supply chain attack on PyTorch

#79

Post by Groxx »

KptMarchewa wrote: post #56
Groxx wrote: post #45

Earlier quoted context omitted.

> The vast majority of repos should be able to run CI on pull requests with no privileges at all When there are no side effects and no in-container secrets and the hosting is free or reasonably limited to prevent abusers, ideally yes. Outside that, heck no, that'd be crazy. You're allowing randos to run arbitrary code on your budget. Locking it down until it's reviewed is like step 1, they can validate locally until…

GH actions are free on public repos.

Until someone tries to mine Bitcoin in yours, yeah
Post reply on HN