Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

trigger.dev

Post reply on HN
Print view View original

51–60 of 191 posts

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#51

Post by snickerbockers »

>Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently.

No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packages to act both competently and in good faith ad infinitum.

Also the OP seemingly implies credentials are stored on-filesystem in plaintext but I might be extrapolating too much there.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#52

Post by deepsun »

snickerbockers wrote: post #51

>Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently. No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packag…

Same thing with IDE plugins. At least some are full-featured by the manufacturer, but I couldn't get on with VS Code as for every small feature I had to install some random plugin (even if popular, but still developed by who-knows-who).

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#53

Post by bspammer »

KomoD wrote: post #31
bspammer wrote: post #25

Given that all the stolen credentials were made public, I was hoping that someone would build a haveibeenpwned style site. We know we were compromised on at least a few tokens, but it would be nice to be able to search using a compromised token to find out what else leaked. We’ve rotated everything we could think of but not knowing if we’ve missed something sucks.

Doesn't it publish the repos to your Github account? Just clone and look at what was stolen.

As the sibling comment said, the worm used stolen GitHub credentials from other victims, and randomly distributed the uploads between victims.

Also everything was double base64 encoded which makes it impossible to use GitHub search.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#54

Post by benoau »

t0mas88 wrote: post #26
benoau wrote: post #8

Earlier quoted context omitted.

You can set up your repo to disable pushing directly to branches like main and require MFA to use the org admin account, so something malicious would need to push to a benign branch and separately be merged into one that deploys come from.

But the attacker could just create a branch, merge request and then merge that?

They can't with git by itself, but if you're also signed in to GitHub or BitBucket's CLI with an account able to approve merges they could use those tools.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#55

Post by zozos »

0xbadcafebee wrote: post #41
zozos wrote: post #3

I have been thinking about this. How do I make my git setup on my laptop secure? Currently, I have my ssh key on the laptop, so if I want to push, I just use git push. And I have admin credentials for the org. How do I make it more secure?

1) Get 1Password, 2) use 1Password to hold all your SSH keys and authorize SSH access [1], 3) use 1Password to sign your Git commits and set up your remote VCS to validate them [2], 4) use GitHub OAuth [3] or the GitHub CLI's Login with HTTPS [4] to do repository push/pull. If you don't like 1Password, use BitWarden. With this setup there are two different SSH keys, one for access to GitHub, one is a commit signing k…

I already use 1password and have it already installed. Will try this out. Thanks!

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#56

Post by Rafert »

> This is one of the frustrating realities of these attacks: once the malware runs, identifying the source becomes extremely difficult. The package doesn't announce itself. The pnpm install completes successfully. Everything looks normal.

Sounds like there’s no EDR running on the dev machines? You should have more to investigate if Sentinel One/CrowdStrike/etc were running.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#57

Post by homebrewer »

anthonyryan1 wrote: post #34
zozos wrote: post #3

I have been thinking about this. How do I make my git setup on my laptop secure? Currently, I have my ssh key on the laptop, so if I want to push, I just use git push. And I have admin credentials for the org. How do I make it more secure?

One approach I started using a could of years ago was storing SSH private keys in the TPM, and using it via PKCS11 in SSH agent. One benefit of Microsoft requiring them for Windows 11 support is that nearly every recent computer has a TPM, either hardware or emulated by the CPU firmware. It guarantees that the private key can never be exfiltrated or copied. But it doesn't stop malicious software on your machine from…

Or use a FIDO token to protect your SSH key, which becomes useless without the hardware token.

https://wiki.archlinux.org/title/SSH_keys#FIDO/U2F

That's what I do. For those of us too lazy to read the article, tl;dr:

  ssh-keygen -t ed25519-sk
or, if your FIDO token doesn't support edwards curves:

  ssh-keygen -t ecdsa-sk
tap the token when ssh asks for it, done.

Use the ssh key as usual. OpenSSH will ask you to tap the token every time you use it: silent git pushes without you confirming it by tapping the token become impossible. Extracting the key from your machine does nothing — it's useless without the hardware token.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#58

Post by elif »

snickerbockers wrote: post #51

>Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently. No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packag…

It wasn't in their product. It was just on a devs machine

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#59

Post by h1fra »

We don't have a clear explanation of the destructive behavior, right? It looks like it had no real purpose, and there were much more effective ways of destroying their repos. Very script kiddie-like, which does not really fit the main complexity of the virus. Very surprising.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#60

Post by hnlmorg »

elif wrote: post #58
snickerbockers wrote: post #51

>Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently. No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packag…

It wasn't in their product. It was just on a devs machine

I think the OP is aware of that and I agree with them that it’s bad practice despite how common it is.

For example with AWS, you can use the AWS CLI to sign you in and that goes through the HTTPS auth flow to provide you with temporary access keys. Which means:

1. You don’t have any access keys in plain text

2. Even if your env vars are also stolen, those AWS keys expire within a few hours anyway.

If the cloud service you’re using doesn’t support OIDC or any other ephemeral access keys, then you should store them encrypted. There’s numerous ways you can do this, from password managers to just using PGP/GPG directly. Just make sure you aren’t pasting them into your shell otherwise you’ll then have those keys in plain text in your .history file.

I will agree that It does take effort to get your cloud credentials set up in a convenient way (easy to access, but without those access keys in plain text). But if you’re doing cloud stuff professionally, like the devs in the article, then you really should learn how to use these tools.

Post reply on HN