Live data from Hacker News

Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

trigger.dev

171–180 of 191 posts

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#171
post #50
post #44

Earlier quoted context omitted.

One option is to remember it.

I don’t think that’s considered secure enough, see the other answers and the push for passkeys. I mean, if passphrases were good for anything you’d directly use them for the ssh connection? :)

It's secure enough.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#172
post #33

Earlier quoted context omitted.

Unless the attacker used any one of hundreds of other avenues to access the AWS resource. Are you sure they didn’t get a service account token from some other service then use that to access customer data? I’ve never seen anyone claim in writing all permutations are exhaustively checked in the audit logs.

It depends on what kind of access we're talking about. If we're talking about AWS resource mutations, one can trust CloudTrail to accurately log those actions. CloudTrail can also log data plane events, though you have to turn it on, and it costs extra. Similarly, RDS access logging is pretty trustworthy, though functionality varies by engine.

What do you mean by “trust cloud trail”

So cloud trail shows the compromised account logging into an EC2 instance every day like normal.

Then service account credentials are used to access user data in S3.

How does cloud trail indicate the compromised credentials were used to access the customer data in S3?

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#173

>Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently. No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packag…

> No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. Could you explain how you'd design a package manager that does not allow that? As far as I understand the moment you use third party code you have to trust to some extent the code that you will run.

I'm speaking to the concept of automatic updates in general, which package managers either enable by default or implicitly allow through lack of security measures.

One obvious solution is to host your own repositories so that nothing gets updated without having been signed off by a trusted employee. Another is to check the cryptographic hash of all packages so it cannot change without the knowledge and consent of your employees.

You're right in that this does not completely eliminate the possibility of trojan horses being sneaked in through open-source dependencies but it would at the very least require some degree of finesse on the part of the person making the trojan horse so that they have to manipulate the system into doing something it was not designed to do.

One thing I really hate about the modern cybersecurity obsession is that there's a large contingent of people who aggressively advocate against anything which might present a problem if misused (rust, encryption on everything no matter how inconsequential, deprecating FTP, UEFI secure boot, timing side-channels, etc) yet at the same time there's a massive community of high-level software developers who appear to be under the impression that extremely basic vulnerabilities (trojan package managers, cross-site scripting, letting my cell phone provider steal my identity because my entire life is authenticated by a SIM card, literally just concatenating strings received over the internet into an SQL statement, etc) are unsolved problems which just has to be tolerated for now until somebody figures out a way to not download and execute non-vetted third-party code. Somehow the two groups never seem to cross swords.

TL;DR: Reading HN i feel like im constantly getting criticized for using C because I might fuck up and let a ROP through yet so many of the most severe modern security breaches are coming from people who think turning off automatic updates is like being asked to prove the rieman zeta hypothesis.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#174
post #125

Earlier quoted context omitted.

> No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. Could you explain how you'd design a package manager that does not allow that? As far as I understand the moment you use third party code you have to trust to some extent the code that you will run.

They can't explain, it's just victim blaming. The market currently doesn’t have a proper solution to this. Everyone works with these package managers, I bet the commenter also has installed pip or npm packages without reading its full code, it just feels cool to tell other people they are dumb and it's their own fault for not reading all the code beforehand or for using a package manager, when every single person doe…

>it's just victim blaming

Victim-blaming is when a girl gets raped and you tell her that it's her fault for dressing like a skank and getting drunk at a college fraternity party. Telling the bank they should have put the money in a vault instead of leaving it in an unlocked drawer next to the cash register is not victim-blaming. Telling the CIA that they shouldn't have given Osama Bin-Laden guns and money to fight the soviets in afghanistan is not victim-blaming. Telling president Roosevelt it was a poor decision to park the entire Pacific fleet in a poorly-defended naval base adjacent to an expansionist empire which is already at war with most of America's allies is not victim-blaming. *Telling a well-funded corporation to not download and execute third-party code with privileges is not victim blaming, especially as their customers are often the ones who are actually being targeted.*

>I bet the commenter also has installed pip or npm packages without reading its full code

I think i did use pip at some point about a decade ago but i can't remember what for. In general though you lose that bet because I don't use either of these programs.

> it just feels cool to tell other people they are dumb

it does, yes.

>and it's their own fault for not reading all the code beforehand or for using a package manager, when every single person does the same.

I don't suppose you've ever played an old video game called "Lemmings"?

>Some just are unlucky.

Lol.

>The whole ecosystem is broken, the expectations of trust are not compatible with the current amount of attacks.

that's kind of my point, except it doesn't mitigate responsibility for participating in that ecosystem.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#175

Earlier quoted context omitted.

It depends on what kind of access we're talking about. If we're talking about AWS resource mutations, one can trust CloudTrail to accurately log those actions. CloudTrail can also log data plane events, though you have to turn it on, and it costs extra. Similarly, RDS access logging is pretty trustworthy, though functionality varies by engine.

What do you mean by “trust cloud trail” So cloud trail shows the compromised account logging into an EC2 instance every day like normal. Then service account credentials are used to access user data in S3. How does cloud trail indicate the compromised credentials were used to access the customer data in S3?

If you have data events enabled for your S3 bucket, CloudTrail will log every access to that bucket along with the identity of the principal used to access it. https://docs.aws.amazon.com/awscloudtrail/latest/userguide/l...

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#176

Earlier quoted context omitted.

It isn't victim blaming. People like you make it impossible to avoid attacks like these because you have no appetite for a better security model. I run npm under bubblewrap because npm has a culture of high risk; of using too many dependencies from untrusted authors. But being scrupulous and responsible is a cost I pay with my time and attention. But it is important because if I run some untrusted code and am comprom…

No, you are the problem because you have a higher expectation than reality. People shouldn't have to run npm in containers. You're over simplifying with one case where you have found one solution while ignoring the identical problems elsewhere. You are preventing us from looking at other solutions because you think the one you have is enough and works for everyone.

I agree with you that I shouldn't have to treat my libraries like untrusted code. I don't know what the rest of your comment means. I don't see how I'm preventing anybody from looking at other solutions to npm, they just don't want to do it because it's hard. And I have similar criticisms for cargo as it just copies npm and inherits all of its problems. I hate that.

npm has had a bad ecosystem since its inception. The left-pad thing being some of my earliest memories of it [1]. So none of this is new.

But all of this is still an issue because it's too convenient and that's the most important thing. Even cargo copies npm because they want to be seen as convenient and the risk is acknowledged. Nobody has the appetite to be held accountable for who they put their trust in.

[1] https://en.wikipedia.org/wiki/Npm_left-pad_incident

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#177

Earlier quoted context omitted.

This is not strictly true - most OS keychain stores have methods of authenticating the requesting application before remitting keys (signatures, non-user-writable paths, etc.), even if its running as the correct user. That said, it requires careful design on the part of the application (and its install process) to not allow a non-elevated application to overwrite some part of the trusted application and get the keys…

> This is not strictly true - most OS keychain stores have methods of authenticating the requesting application before remitting keys (signatures, non-user-writable paths, etc.), even if its running as the correct user. Isn't that a smartphone-and-app-store-only thing? As I understand it, no mainstream desktop OS provides the capabilities to, for example, protect a user's browser cookies from a malicious tool launche…

macOS and Windows’s native keychains both support this - they encrypt the secrets with a key that is not accessible to apps that run with user permissions without sudo (macOS) or elevation (Windows). The actual user can still access them, but a normal app (other than the one that stored the secret in the keychain originally) running as that user cannot do so directly.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#178
post #167

Earlier quoted context omitted.

It isn't victim blaming. People like you make it impossible to avoid attacks like these because you have no appetite for a better security model. I run npm under bubblewrap because npm has a culture of high risk; of using too many dependencies from untrusted authors. But being scrupulous and responsible is a cost I pay with my time and attention. But it is important because if I run some untrusted code and am comprom…

What no appetite? I just don't like your solution. The industry needs an answer to this problem stat, and it can't be "just read the code before".

At some point you must be open to being compelled to read code you run or ship. Otherwise, if that's to hard, then I don't know what to tell you. We'll just never agree.

If you find a better solution than being responsible for what you do and who you trust, I'm all for it. Until then, that's part of the job.

When I was a junior, our company payed a commercial license for some of the larger libraries we used and it included support. Or manage risk by using fewer and more trustworthy projects like Django instead of reaching for a new dependency from some random person every time you need to solve a simple problem.

> What no appetite? I just don't like your solution.

When I say "appetite" I am being very deliberate. You are hungry but you won't eat your vegetables. When you say "I just don't like your vegetables", then you aren't that hungry. You don't have the appetite. You'd rather accept the risk. Which is fine but then don't complain when stuff like this happens and everyone is compromised.

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#179

Earlier quoted context omitted.

What do you mean by “trust cloud trail” So cloud trail shows the compromised account logging into an EC2 instance every day like normal. Then service account credentials are used to access user data in S3. How does cloud trail indicate the compromised credentials were used to access the customer data in S3?

If you have data events enabled for your S3 bucket, CloudTrail will log every access to that bucket along with the identity of the principal used to access it. https://docs.aws.amazon.com/awscloudtrail/latest/userguide/l...

Right and in my example it would be the principal of the service account, not the compromised AWS account.

If you ran a cloud trail query that's essentially "Did Alice access user data in S3 ever?" the answer would be "No"

So that brings us back to the question, what is meant by "trust CloudTrail"

Re: Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem

#180

Earlier quoted context omitted.

It isn't victim blaming. People like you make it impossible to avoid attacks like these because you have no appetite for a better security model. I run npm under bubblewrap because npm has a culture of high risk; of using too many dependencies from untrusted authors. But being scrupulous and responsible is a cost I pay with my time and attention. But it is important because if I run some untrusted code and am comprom…

> you have no appetite for a better security model For what it's worth, there are some advancements. PNPM - the packager used in this case - doesn't automatically run postinstall scripts. In this case, either the engineer allowed it explicitly, or a transitive dependency was previously considered safe, and allowed by default, but stopped being safe. PNPM also lets you specify a minimum package age, so you cannot inst…

I had the chance to finish reading and it looks like Trigger were using an older version of PNPM which didn't do any of the above, and have since implemented everything I've mentioned in my post, plus some additional Git security.

So a slight amendment there on the human error side of things.

Post reply on HN