Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A supply chain attack on PyTorch

johnstawinski.com

Post reply on HN
Print view View original

61–70 of 109 posts

Re: A supply chain attack on PyTorch

#61

Post by gz5 »

jackwilsdon wrote: post #53
ethbr1 wrote: post #52

Earlier quoted context omitted.

What's the GitHub Actions tooling like for emphemeral self-hosted runners? Afaict, a huge portion of this attack came from persistence on the self-hosted runner. Absent that, they would have needed a container jailbreak as well, which substantially ups the difficulty. And if a repo is running <100 builds a day, spin up + kill container seems a small per-build price to pay for the additional security isolation.

GitHub themselves don't seem to provide any mechanism to make runners ephemeral. It looks like all they allow you to do is flag a runner as ephemeral, meaning it will be de-registered once a job is completed - you need to write your own tooling to wipe it yourself (either via starting a whole new runner in a new environment and registering that or wiping the existing runner and re-registering it). https://docs.github…

there are 3rd party foss options (1):

1. ephemeral + zero implicit trust (2) https://blog.openziti.io/my-intern-assignment-call-a-dark-we...

2. zero implicit trust: https://github.com/openziti/ziti-webhook-action

(1) disclosure, maintainer (2) zero implicit trust in this case = no open inbound ports on underlay; need to access via app-specific overlay which requires strong identity, authN, authZ

Re: A supply chain attack on PyTorch

#63

Post by adnanthekhan »

n2d4 wrote: post #55
trevyn wrote: post #51

Earlier quoted context omitted.

It’s called GitHub secrets. Builds off of main get the secrets, pull requests from randos don’t. And public repos don’t pay for CI on GitHub. Not rocket science, people.

Pytorch did use GH secrets for the valuables and you can see that this wasn't enough, right there in the OP, because the self-hosted runners are still shared

Yup! This is what makes this kind of attack scary and very unique to GitHub Actions. The baseline GITHUB_TOKEN just blows the door open on lateral movement via workflow_dispatch and and repository_dispatch events.

In several of our other operations, not just PyTorch, we leveraged workflow_dispatch to steal a PAT from another workflows. Developers tend to over-provision PATs so often. More often than not we'd end up with a PAT that has all scopes checked and org admin permissions. With that one could clean out all of the secrets from an organization in minutes using automated tools such as https://github.com/praetorian-inc/gato.

Re: A supply chain attack on PyTorch

#64

Post by adnanthekhan »

korhojoa wrote: post #59
EE84M3i wrote: post #54

Earlier quoted context omitted.

I don't understand this PR. How is it an "attack"? It seems to just be pinning a package version, was the package compromised, or was this more a "vulnerability"?

If they're pulling from master instead of from a known version, it could be changed to be malicious, and the next time it is fetched, the malicious version would be used instead. It's a vulnerability.

Oh, you'll like this one then. Until 3 months ago GitHub's Runner images was pulling a package directly from Aliyun's CDN. This was executed during image testing (version check). So anyone with the ability to modify Aliyun's CDN in China could have carried out a pretty nasty attack. https://github.com/actions/runner-images/commit/6a9890362738...

Now it's just anyone with write access to Aliyun's repository. :) (p.s. GitHub doesn't consider this a security issue).

Re: A supply chain attack on PyTorch

#65

Post by kevin_nisbet »

mlazos wrote: post #60
trevyn wrote: post #51

Earlier quoted context omitted.

It’s called GitHub secrets. Builds off of main get the secrets, pull requests from randos don’t. And public repos don’t pay for CI on GitHub. Not rocket science, people.

If they use the same runners, couldn’t the attacker just wait? The runners would need to be sequestered too

Absolutely. The real difficulty is tests on PR are by definition remote code execution by an untrusted source, so a full risk analysis and hardening needs to be done.

Here's a similar mistake on an OSS repo a company I worked for made: https://goteleport.com/blog/hack-via-pull-request/

Re: A supply chain attack on PyTorch

#66

Post by trevyn »

adnanthekhan wrote: post #63
n2d4 wrote: post #55

Earlier quoted context omitted.

Pytorch did use GH secrets for the valuables and you can see that this wasn't enough, right there in the OP, because the self-hosted runners are still shared

Yup! This is what makes this kind of attack scary and very unique to GitHub Actions. The baseline GITHUB_TOKEN just blows the door open on lateral movement via workflow_dispatch and and repository_dispatch events. In several of our other operations, not just PyTorch, we leveraged workflow_dispatch to steal a PAT from another workflows. Developers tend to over-provision PATs so often. More often than not we'd end up w…

To clarify, this secret stealing is not an issue with GitHub-hosted runners, correct?

Re: A supply chain attack on PyTorch

#67

Post by sroussey »

SethMLarson wrote: post #29

Great write-up! There's a few things you can do as either a producer or consumer to thwart this sort of attack: Producers: * Self-hosted infrastructure should not be running anonymous code. PRs should be reviewed before code executes on your infrastructure. Potentially should be a GitHub default when using self-hosted runners? * Permissions for workflows and tokens should be minimal and fine-grained. "permissions: re…

I just wish python would isolate all the pip install stuff and put in the project folder like has been done with nodejs for years.

Re: A supply chain attack on PyTorch

#68

Post by adnanthekhan »

trevyn wrote: post #66
adnanthekhan wrote: post #63

Earlier quoted context omitted.

Yup! This is what makes this kind of attack scary and very unique to GitHub Actions. The baseline GITHUB_TOKEN just blows the door open on lateral movement via workflow_dispatch and and repository_dispatch events. In several of our other operations, not just PyTorch, we leveraged workflow_dispatch to steal a PAT from another workflows. Developers tend to over-provision PATs so often. More often than not we'd end up w…

To clarify, this secret stealing is not an issue with GitHub-hosted runners, correct?

Correct. For fork PR workflows on the pull_request trigger the GITHUB_TOKEN has read only permissions, so you can’t do anything with it.

The key thing with a non-ephemeral runner is that (after obtaining persistence) you can grab the GITHUB_TOKEN from a subsequent non-fork PR build or a build on another trigger, which will have write permissions unless restricted by the repository maintainers.

Re: A supply chain attack on PyTorch

#69

Post by Twirrim »

zX41ZdbW wrote: post #37
zX41ZdbW wrote: post #34

Recently, there were similar attempts (two) of supply chain attacks on the ClickHouse repository, but: - it didn't do anything because CI does not run without approval; - the user's account magically disappeared from GitHub with all pull requests within a day. Also worth reading a similar example: https://blog.cloudflare.com/cloudflares-handling-of-an-rce-v... Also, let me recommend our bug bounty program: https://gi…

But I continue to find garbage in some of our CI scripts. Here is an example: https://github.com/ClickHouse/ClickHouse/pull/58794/files The right way is to: - always pin versions of all packages; - this includes OS package repositories, Docker repositories, as well as pip, npm, cargo, and others; - never download anything from the master/main or other branches; specify commit sha; - ideally, copy all Docker images to…

Also: Vendor your dependencies. There's zero reason to be wget'ing like that. It's two short perl scripts.

Re: A supply chain attack on PyTorch

#70

Post by Twirrim »

richardwhiuk wrote: post #49
gray_-_wolf wrote: post #8

Hm, from the reading, it seem he was pretty careful to not do any harm, but still, is this type of practical research actually legal?

Essentially, generally, no. Once you've discovered a security hole, exploiting it to see how much access you can get is generally frowned upon.

I know Marcus, the guy they mention that first caught the problem. He had no end of trouble getting Meta to acknowledge the severity of what he'd found, and they just constantly went radio silence on him, in between not really understanding the problem.

I ended up having to reach out to someone senior I knew in the security org there to get them to swoop in and pick up the report, before it got any actual traction (I'd worked with that senior security engineer in a previous job).

Post reply on HN