Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A supply chain attack on PyTorch

johnstawinski.com

Post reply on HN
Print view View original

101–109 of 109 posts

Re: A supply chain attack on PyTorch

#101

Post by lmeyerov »

n2d4 wrote: post #55
trevyn wrote: post #51

Earlier quoted context omitted.

It’s called GitHub secrets. Builds off of main get the secrets, pull requests from randos don’t. And public repos don’t pay for CI on GitHub. Not rocket science, people.

Pytorch did use GH secrets for the valuables and you can see that this wasn't enough, right there in the OP, because the self-hosted runners are still shared

I think 'environments' was meant, where diff GHA environments get diff secrets, and policies dictate who gets to run what actions with what envs.

But that is real work to setup, audit, and maintain. It'd be better if, like phone app capabilities, the default would be no privs, any privs are explicitly granted, and if they aren't being used, the system detects that and asks if you want to remove specific ones.

Re: A supply chain attack on PyTorch

#102

Post by richardwhiuk »

mike_hearn wrote: post #94
Twirrim wrote: post #70

Earlier quoted context omitted.

I know Marcus, the guy they mention that first caught the problem. He had no end of trouble getting Meta to acknowledge the severity of what he'd found, and they just constantly went radio silence on him, in between not really understanding the problem. I ended up having to reach out to someone senior I knew in the security org there to get them to swoop in and pick up the report, before it got any actual traction (I…

One may suspect that they do know, but if you widen the scope of bug bounty programmes to encompass open source project supply chain then your programme immediately turns into a dollar piñata. For a long time Apple didn't have a bug bounty programme at all. This wasn't because they didn't care about security. It's because their own internal audits were generating enough reports to saturate the available capacity for…

That sort of suggests you are under-funding your fixing capability.

Re: A supply chain attack on PyTorch

#103

Post by pvg »

planede wrote: post #97
pvg wrote: post #20

Earlier quoted context omitted.

This question comes of up frequently with these and it's premised on the hypothetical value of the bug on 'the black market'. The vast majority of such reported vulnerabilities have a 'black market' value of roughly zero, though, including this one. This doesn't say anything about the quality of the research, just that it's pretty hard to get monetary or other value out of most vulnerabilities.

While I'm not familiar with the black market, I refuse to believe that an arbitrarily modified pytorch binary release has zero value.

That’s fine but it isn’t much of an argument.

Re: A supply chain attack on PyTorch

#104

Post by sroussey »

bloopernova wrote: post #72
sroussey wrote: post #67

Earlier quoted context omitted.

I just wish python would isolate all the pip install stuff and put in the project folder like has been done with nodejs for years.

Python's virtualenv does something similar by keeping all files under one directory.

Yes, but it should be the default

Re: A supply chain attack on PyTorch

#105

Post by gabriel-samfia »

n2d4 wrote: post #55
trevyn wrote: post #51

Earlier quoted context omitted.

It’s called GitHub secrets. Builds off of main get the secrets, pull requests from randos don’t. And public repos don’t pay for CI on GitHub. Not rocket science, people.

Pytorch did use GH secrets for the valuables and you can see that this wasn't enough, right there in the OP, because the self-hosted runners are still shared

Ouch! This is why ephemeral runners should be used. Preferably virtual machines. On an infrastructure that can define security group rules to prevent lateral movement.

Re: A supply chain attack on PyTorch

#106

Post by Bognar »

LtWorf wrote: post #84
o11c wrote: post #35

Earlier quoted context omitted.

The problem is that there are fundamentally 2 different kinds of builds, but the current tooling is weak: * pre-merge builds on PRs. These should not have privileges, but making the distinction between the two cases requires a lot of care. * official builds of "master" or a feature branch from the main repo. These "need" privileges to upload the resulting artifacts somewhere. Of course, if all it did was wake up some…

> These "need" privileges to upload the resulting artifacts somewhere. Only because pypi removed support for gpg and their only "signature" now basically is "was uploaded from a gh action". Otherwise a maintainer could do it offline without needing to give microsoft total access to uploading to pypi.

If you're using GHA to publish then you still need a trusted branch to provide secrets to. If you're not publishing using CI, then you can still upload to PyPI manually with an API Token.

> without needing to give microsoft total access to uploading to pypi

I assume you're referring to Trusted Publishers here? It's a per-project configuration using the industry standard of OIDC that you don't have to opt in to, so "total access" is a silly characterization. Also if you're insinuating that MS is going to generate fraudulent OIDC tokens to compromise a PyPI package, then you might want to start weaning yourself off the kool-aid.

Re: A supply chain attack on PyTorch

#107

Post by withinboredom »

ethbr1 wrote: post #52
withinboredom wrote: post #22

Earlier quoted context omitted.

Self-hosted runners is the way to go, IMHO. Especially if you have bare metal resources. I love how fast my builds are with 16 cores, and gobs of ram.

What's the GitHub Actions tooling like for emphemeral self-hosted runners? Afaict, a huge portion of this attack came from persistence on the self-hosted runner. Absent that, they would have needed a container jailbreak as well, which substantially ups the difficulty. And if a repo is running <100 builds a day, spin up + kill container seems a small per-build price to pay for the additional security isolation.

The default kubernetes implementation owned by github[1] assumes ephemeral runners by default. You can also specify what policies they should have using regular network policies provided by kubernetes. So, if you have a kubernetes cluster, that's the way to go.

[1]: https://github.com/actions/actions-runner-controller

Re: A supply chain attack on PyTorch

#108

Post by LtWorf »

Bognar wrote: post #106
LtWorf wrote: post #84

Earlier quoted context omitted.

> These "need" privileges to upload the resulting artifacts somewhere. Only because pypi removed support for gpg and their only "signature" now basically is "was uploaded from a gh action". Otherwise a maintainer could do it offline without needing to give microsoft total access to uploading to pypi.

If you're using GHA to publish then you still need a trusted branch to provide secrets to. If you're not publishing using CI, then you can still upload to PyPI manually with an API Token. > without needing to give microsoft total access to uploading to pypi I assume you're referring to Trusted Publishers here? It's a per-project configuration using the industry standard of OIDC that you don't have to opt in to, so "t…

> If you're using GHA to publish then you still need a trusted branch to provide secrets to

The vulnerability was exactly that the secrets of the trusted branch can get leaked :)

> you can still upload to PyPI manually with an API Token

I can, BUT if you want to be a Trusted Publisher™ the only way is to do it via github.

https://docs.pypi.org/trusted-publishers/

Most likely the plan is to make it compulsory for all projects eventually, just like they made 2fa compulsory.

So less secure is considered as MORE secure by pypi :) Which is consistent with the idea that no PGP signature is more secure than signed uploads. Or the idea that a global token in a clear text file is somehow safer than a password that gets typed every time.

Re: A supply chain attack on PyTorch

#109

Post by growse »

guappa wrote: post #88
Groxx wrote: post #45

Earlier quoted context omitted.

> The vast majority of repos should be able to run CI on pull requests with no privileges at all When there are no side effects and no in-container secrets and the hosting is free or reasonably limited to prevent abusers, ideally yes. Outside that, heck no, that'd be crazy. You're allowing randos to run arbitrary code on your budget. Locking it down until it's reviewed is like step 1, they can validate locally until…

They are so difficult. I wanted to stop random people to run code on my repository… I don't have any secrets or write access or anything to exploit. Just to avoid burning quota. The issue is that now the pull requests don't get tested at all. I have to manually, locally, get all the commits, make a branch on the main repository with them, and then the actions run.

One approach might be to review manually, then label the pr and trigger the pr based off the label being added.

Challenge there is that if the pr changes, it's a bit clunky to retrigger the CI (you have to remove, then re-add).

I guess you could also do this with comments - can you trigger a workflow based on a specific comment being added from a specific user?

Post reply on HN