Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem of Shai-Hulud attack on November 24th, 2025

posthog.com

Post reply on HN
Print view View original

61–70 of 77 posts

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#61

Post by black3r »

amluto wrote: post #52
hardwaresofton wrote: post #47

Dear everyone. Never use pull_request_target. This is not the first time it’s bitten people. It’s not safe, and honestly GitHub should have better controls around it or remove and rework it — it is a giant footgun. > One of our engineers figured out this was because it triggered on: pull_request which means external contributions (which come from forks, rather than branches in the repo like internal contributions) wo…

Dear GitHub Actions: what the heck? There are so many things about GitHub Actions that make no sense. Why are actions configured per branch? Let me configure Actions somewhere on the repository that is not modifiable by some yml files that can exist in literally any branch. Let me have actual security policy for configuring Actions that is separate from permission to modify a given branch. Why do workflows have such…

While I agree with the general sentiment that lots of things about GH actions don't make sense, when you actually look at what the vulnerability was, you'll find that for lots of your questions it wasn't GitHub Actions' fault.

This is the vulnerable workflow in question: https://github.com/PostHog/posthog/blob/c60544bc1c07deecf336...

> Why are actions configured per branch?

This workflow uses `pull_request_target` targeting where the actions are configured by the branch you're merging PR into, which should be safe - attacker can't modify the YML actions are running.

> Why do workflows have such strong permissions?

What permissions are workflow run with is irrelevant here, because the workflow runs the JS script with a custom access token instead of the permissions associated with the GH actions runner by default.

> Why is security almost impossible to achieve instead of being the default?

The default for `pull_request_target` is to checkout the branch you're trying to merge into (which again should be safe as it doesn't contain attacker's files), but this workflow explicitly checks out the attacker's branch on line 22.

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#62

Post by [deleted] »

hardwaresofton wrote: post #47

Dear everyone. Never use pull_request_target. This is not the first time it’s bitten people. It’s not safe, and honestly GitHub should have better controls around it or remove and rework it — it is a giant footgun. > One of our engineers figured out this was because it triggered on: pull_request which means external contributions (which come from forks, rather than branches in the repo like internal contributions) wo…

[deleted]

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#63

Post by mulmboy »

LtWorf wrote: post #24
jameskilton wrote: post #4

Earlier quoted context omitted.

> The PR was opened, the workflow run, and the PR closed within the space of 1 minute (screenshots include timestamps in UTC+2, the author's timezone): It's an unfortunately common problem with GitHub Actions, it's easy to set things up to where any PR that's opened against your repo runs the workflows as defined in the branch. So you fork, make a malicious change to an existing workflow, and open a PR, and your code…

Problem is that you might want to have the tests run before even looking at it. I think the mistake was to put secrets in there and allow publishing directly from github's CI. Hilariously the people at pypi advise to use trusted publishers (publishing on pypi from github rather than local upload) as a way to avoid this issue. https://blog.pypi.org/posts/2025-11-26-pypi-and-shai-hulud/

It does largely avoid the issue if you configure to allow only specific environments AND you require reviews before pushing/merging to branches in that environment.

https://docs.pypi.org/trusted-publishers/adding-a-publisher/

For a malicious version to be published would then require full merge which is a fairly high bar.

AWS allows similar

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#64

Post by fragmede »

dreamcompiler wrote: post #55
themly wrote: post #19

Long story short: they messed up the assign-reviewers.yml workflow, allowing external contributors to merge PRs without proper reviews. From this point on, you're fully open to all kinds of bad stuff.

Here's an AI product I would actually use: Write my damn GH actions yml for me. Oh, and describe for me exactly how it works and why. And be right about it.

Why does it need to be a distinct product and not Cursor/ChatGPT/Claude code/any of the other existing tools?

(If you're so anti-AI that you're still writing boilerplate like that by hand, I mean, not gonna tell you what you do, but the rest of us stopped doing that crap as soon as it was evident we didn't have to any more.)

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#65

Post by LtWorf »

mulmboy wrote: post #63
LtWorf wrote: post #24

Earlier quoted context omitted.

Problem is that you might want to have the tests run before even looking at it. I think the mistake was to put secrets in there and allow publishing directly from github's CI. Hilariously the people at pypi advise to use trusted publishers (publishing on pypi from github rather than local upload) as a way to avoid this issue. https://blog.pypi.org/posts/2025-11-26-pypi-and-shai-hulud/

It does largely avoid the issue if you configure to allow only specific environments AND you require reviews before pushing/merging to branches in that environment. https://docs.pypi.org/trusted-publishers/adding-a-publisher/ For a malicious version to be published would then require full merge which is a fairly high bar. AWS allows similar

As we're seeing, properly configuring github actions is rather hard. By default force pushes are allowed on any branch.

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#66

Post by woodruffw »

LtWorf wrote: post #24
jameskilton wrote: post #4

Earlier quoted context omitted.

> The PR was opened, the workflow run, and the PR closed within the space of 1 minute (screenshots include timestamps in UTC+2, the author's timezone): It's an unfortunately common problem with GitHub Actions, it's easy to set things up to where any PR that's opened against your repo runs the workflows as defined in the branch. So you fork, make a malicious change to an existing workflow, and open a PR, and your code…

Problem is that you might want to have the tests run before even looking at it. I think the mistake was to put secrets in there and allow publishing directly from github's CI. Hilariously the people at pypi advise to use trusted publishers (publishing on pypi from github rather than local upload) as a way to avoid this issue. https://blog.pypi.org/posts/2025-11-26-pypi-and-shai-hulud/

> Problem is that you might want to have the tests run before even looking at it.

Why is this a problem? The default `pull_request` trigger isn't dangerous in GitHub Actions; the issue here is specifically with `pull_request_target`. If all you want to do is have PRs run tests, you can do that with `pull_request` without any sort of credential or identity risk.

> Hilariously the people at pypi advise to use trusted publishers (publishing on pypi from github rather than local upload) as a way to avoid this issue.

There are two separate things here:

1. When we designed Trusted Publishing, one of the key observations was that people do use CI to publish, and will continue to do so because it conveys tangible benefits (mostly notably, it doesn't tie release processes to an opaque phase on a developer's machine). Given that people do use CI to publish, giving them a scheme that provides self-expiring, self-scoping credentials instead of long-lived ones is the sensible thing to do.

2. Separately, publishing from CI is probably a good thing for the median developer: developer machines are significantly more privileged than the average CI runner (in terms of access to secrets/state that a release process simply doesn't need). One of the goals behind Trusted Publishing was to ensure that people could publish from an otherwise minimal CI environment, without even needing to configure a long-lived credential for authentication.

Like with every scheme, Trusted Publishing isn't a magic bullet. But I think the proscription to use it here is essentially correct: Shai-Hulud propagates through stored credentials, and a compromised credential from a TP flow is only useful for a short period of time. In other words, Trusted Publishing would make it harder for the parties behind Shai-Hulud to group and orchestrate the kinds of compromise waves we're seeing.

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#67

Post by slashdave »

dreamcompiler wrote: post #55
themly wrote: post #19

Long story short: they messed up the assign-reviewers.yml workflow, allowing external contributors to merge PRs without proper reviews. From this point on, you're fully open to all kinds of bad stuff.

Here's an AI product I would actually use: Write my damn GH actions yml for me. Oh, and describe for me exactly how it works and why. And be right about it.

Except the model would have been trained on the available corpus of known runners and will achieve the same average level of quality...

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#68

Post by mulmboy »

LtWorf wrote: post #65
mulmboy wrote: post #63

Earlier quoted context omitted.

It does largely avoid the issue if you configure to allow only specific environments AND you require reviews before pushing/merging to branches in that environment. https://docs.pypi.org/trusted-publishers/adding-a-publisher/ For a malicious version to be published would then require full merge which is a fairly high bar. AWS allows similar

As we're seeing, properly configuring github actions is rather hard. By default force pushes are allowed on any branch.

Yes and anyone who knows anything about software dev knows that the first thing you should do with an important repo is set up branch protections to disallow that, and require reviews etc. Basic CI/CD.

This incident reflects extremely poorly on PostHog because it demonstrates a lack of thought to security beyond surface level. It tells us that any dev at PostHog has access at any time to publish packages, without review (because we know that the secret to do this is accessible from plain GHA secret which can be read from any GHA run which presumably run on any internal dev's PR). The most charitable interpretation of this is that it's consciously justified by them because it reduces friction, in which case I would say that demonstrates poor judgement, a bad balance.

A casual audit would have revealed this and suggested something like restricting the secret to a specific GHA environment and requiring reviews to push to that env. Or something like that.

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#69

Post by joatmon-snoo »

hrpnk wrote: post #59

TIL: yarn/pnpm has a minimumReleaseAge setting. "We also suggest you make use of the minimumReleaseAge setting present both in yarn and pnpm. By setting this to a high enough value (like 3 days), you can make sure you won't be hit by these vulnerabilities before researchers, package managers, and library maintainers have the chance to wipe the malicious packages."

This setting is new and was introduced in response to the first round of shai hulud attacks.

Re: Post-mortem of Shai-Hulud attack on November 24th, 2025

#70

Post by LtWorf »

woodruffw wrote: post #66
LtWorf wrote: post #24

Earlier quoted context omitted.

Problem is that you might want to have the tests run before even looking at it. I think the mistake was to put secrets in there and allow publishing directly from github's CI. Hilariously the people at pypi advise to use trusted publishers (publishing on pypi from github rather than local upload) as a way to avoid this issue. https://blog.pypi.org/posts/2025-11-26-pypi-and-shai-hulud/

> Problem is that you might want to have the tests run before even looking at it. Why is this a problem? The default `pull_request` trigger isn't dangerous in GitHub Actions; the issue here is specifically with `pull_request_target`. If all you want to do is have PRs run tests, you can do that with `pull_request` without any sort of credential or identity risk. > Hilariously the people at pypi advise to use trusted p…

The kind of argument of "just don't make mistakes, how hard is it" (and we're talking about something very obscure and badly documented here) didn't work for C and in my opinion doesn't work for this either.
Post reply on HN