Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinycolor supply chain attack post-mortem

sigh.dev

Post reply on HN
Print view View original

51–60 of 85 posts

Re: Tinycolor supply chain attack post-mortem

#51

Post by groggler »

jerf wrote: post #41
chatmasta wrote: post #18

Earlier quoted context omitted.

These machine-to-machine OIDC flows seem secure, and maybe they are when they’re implemented properly, but they’re really difficult to configure. And I can’t shake the feeling that they’re basically just “tokens with more moving parts,” at least for a big chunk of exploitation paths. Without a human in the loop, there’s still some “thing” that gets compromised, whether it’s a token or something that generates time-li…

"Without a human in the loop, there’s still some “thing” that gets compromised, whether it’s a token or something that generates time-limited tokens." Speaking knowingly reductionistically and with an indeterminate amount of sarcasm, one of the hardest problems in security is how to know something without knowing something. The first "knowing something" is being able to convince a security system to let you do someth…

A sk- key with no user presence test to use and pin to update is pretty perfect in my book.. Anything less and the authentication can too easily be permanently stolen out of pointless soft protections any more and the decisions are overly complicated hoops for whatever they were supposed to deliver.

Re: Tinycolor supply chain attack post-mortem

#52

Post by baobun »

hinkley wrote: post #30
drdrey wrote: post #2

> A while ago, I collaborated on angulartics2, a shared repository where multiple people still had admin rights. That repo still contained a GitHub Actions secret — a npm token with broad publish rights. This collaborator had access to projects with other people which I believe explains some of the other 40 initial packages that were affected. > A new Shai-Hulud branch was force pushed to angulartics2 with a maliciou…

I have admin rights on someone else’s npm repo and I’ve done most of the recent releases. Becoming admin lit a fire under me to fix all of the annoying things and shitty design decisions that have been stuck in the backlog for years so most of the commits are also mine. I don’t want my name on broken code that “works”. I had just about convinced myself that we should be using a GitHub action to publish packages becau…

npm has had support for package-scoped publish tokens (with optional 2FA enforcement) for a few years by now. So in case of compromise, the blast radius would be a single package.

The OP gave the GH repo too broad permissions. There is no good reason for the repo CI workflow to have full access to everything under their account.

Re: Tinycolor supply chain attack post-mortem

#53

Post by [deleted] »

mnahkies wrote: post #32

I think the point around incorporating MFA into the automated publishing flow isn't getting enough attention. I've got no problem with doing an MFA prompt to confirm publish by a CI workflow - but last I looked this was a convoluted process of opening a https tunnel out (using a third party solution) such that you could provide the code. I'd love to see either npm or GitHub provide an easy, out the box way, for me to…

[deleted]

Re: Tinycolor supply chain attack post-mortem

#54

Post by bigiain »

rectang wrote: post #16
tcoff91 wrote: post #7

Earlier quoted context omitted.

Many packages have only 1 author.

The conclusion I'm coming to is that depending on packages which only have a single author is problematic. There are too many ways that packages published by one person can be compromised. Packages which don't have approval and review by a reliable third party shouldn't be visible by default in a package manager.

How many of your dependencies have 2nd level dependencies which have even deeper dependencies on ZX Utils, or NX (or left_pad.js)?

(right now I don't know the answer to that for the stuff I'm responsible for, but I'm in the process of researching and setting up and configuring the sort of tools needed to automate that.)

Re: Tinycolor supply chain attack post-mortem

#55

Post by x0x0 »

rectang wrote: post #40
x0x0 wrote: post #38

Earlier quoted context omitted.

That's a lot of entitlement for things you haven't paid a cent for; not just multiple authors but trusted 3rd parties; approval and review; etc.

I’ve done all those things myself (past ASF member where all that and more was SOP), so I realize what I’m asking for. It’s not crazy for authors of small packages to form small collectives and serve as each others’ trusted third parties. In any case, if the choice is “frequent supply chain compromise, take it or leave it”, the answer is of course “leave it”. If we need to pay for curated packages because the problem…

> It’s not crazy for authors of small packages to form small collectives and serve as each others’ trusted third parties.

Yeah, there's that insane entitlement. More demands for others' time and labor, plus the conflation between you demanding labor vs if people don't agree to your free labor demands, they're pro supply chain compromise.

Re: Tinycolor supply chain attack post-mortem

#56

Post by waterTanuki »

Something somewhere needs to change because the status quo just isn't working. Yes, we can cheer on the benefit of OIDC tokens and zero-trust solutions in CI pipelines on HN all we want, but the fact is there's a significant number of library developers out there with millions of package downloads per week that will refuse to do anything about security until they're compromised or npm blocks them from publishing until they do.

And then there's other non-sensical proposals like spelunking deep into projects some which could be over a decade old and just rip out all the dependencies until there's nothing but a standard library is left. Look, I'm all for a better std lib, I think reducing the number of dependencies we have is good. But just saying "you should reduce dependencies" will do nothing concrete to fix the problem which already exists, because it's much easier said than done.

So either tens of thousands or hundreds of thousands of developers stop using npm, and everyone refactors their projects to add more code and strip dependencies, or npm starts enforcing things like 2FA and OIDC for package developers with over X number of weekly downloads, and blocks publishing for those that don't follow the new security rules. I think it's clear which solution is more practical to implement. The only other option is for npm to completely lose its reputation and then we wind up with XKCD 927 again.

Re: Tinycolor supply chain attack post-mortem

#57

Post by rectang »

x0x0 wrote: post #55
rectang wrote: post #40

Earlier quoted context omitted.

I’ve done all those things myself (past ASF member where all that and more was SOP), so I realize what I’m asking for. It’s not crazy for authors of small packages to form small collectives and serve as each others’ trusted third parties. In any case, if the choice is “frequent supply chain compromise, take it or leave it”, the answer is of course “leave it”. If we need to pay for curated packages because the problem…

> It’s not crazy for authors of small packages to form small collectives and serve as each others’ trusted third parties. Yeah, there's that insane entitlement. More demands for others' time and labor, plus the conflation between you demanding labor vs if people don't agree to your free labor demands, they're pro supply chain compromise.

In a general discussion forum, I have floated some approaches for hardening distribution which have proven effective in other communities. If NPM can harden their systems using other mechanisms, then more power to them.

Re: Tinycolor supply chain attack post-mortem

#58

Post by klysm »

tcoff91 wrote: post #37
mnahkies wrote: post #32

I think the point around incorporating MFA into the automated publishing flow isn't getting enough attention. I've got no problem with doing an MFA prompt to confirm publish by a CI workflow - but last I looked this was a convoluted process of opening a https tunnel out (using a third party solution) such that you could provide the code. I'd love to see either npm or GitHub provide an easy, out the box way, for me to…

Publishing a package involves 2 phases: uploading the package to npmjs, and making it availble to users. Right now these 2 phases are bundled together into 1 operation. I think the right way to approach this is to unbundle uploading the packages & publishing packages so that they're available to end-users. CI systems should be able to build & upload packages in a fully automated manner. Publishing the uploaded packag…

npm should require this with packages that have a large enough blast radius

Re: Tinycolor supply chain attack post-mortem

#59

Post by cyphar »

rectang wrote: post #4

Two-factor auth for publishing is helpful, but requiring cryptographically signed approval by multiple authors would be more helpful. Then compromising a single author wouldn't be enough.

While multiple authors' signatures would be nice, a lot of these kinds of attacks would be solved if there was any signature verification being done of the commits, tags, or generated artefacts at all.

People like to complain about distribution packaging being obtuse, but most distributions have rich support for verifying that package sources were signed by a key in a keyring that is maintained by the distribution. My (somewhat biased) view is that language package managers still do not provide the same set of features for validation that (for instance) rpmbuild does.

The release process for runc has the following safeguards:

    * As the upstream maintainer of runc, our releases are all signed with one of a set of keys that are maintained in our repo[1]. Our tags are also signed by one of the same keys. In my case, my key is stored in a Yubikey and so cannot easily be exfiltrated.
    * Our release scripts include a step which validate that all of the keys in that keyring file are valid (sub)keys registered to the GitHub account of a maintainer[2]. They also prompt the person doing the signing to check that the list looks reasonable before signing anything[3].
    * Distributions such as openSUSE have a copy of the keyring file[4] and the build system will automatically reject the build if the source code archive is not signed. Our official binary releases are also signed and so can be validated in a similar manner.
Maybe there are still gaps in this setup, and I would love to hear them. But I think this setup would have blocked this kind of attack at several stages. I personally don't like the idea of signing releases in CI -- if you really want to build your binaries in CI, that's fine, but you should always require a maintainer to personally sign the binaries at the end of the process.

For language package managers that do not support such a workflow, trusted publishing is a less awful setup than having long-lived publishing keys that may be incorrectly scoped (as happened in this case) but it still allows someone who gains access to your GitHub account (such as by stealing your cookies) to publish updated versions of your package with very little resource. GitHub supports setting a mandatory timeout for trusted publishing but the attacker could easily disable that. If someone got access to my GitHub account, it would be a very bad day but distributions would not accept the new releases because their copy of our keyring would not include the attackers keys (even if they added them to my account).

Disclaimer: I work at SUSE, though I will say that I would like for OBS to have nice support for validating checksums of artefacts like Arch and Gentoo do (you can /theoretically/ do it with OBS services or emulate it with forcelocal -- and most packages actually store the archive in OBS rather than pulling it at build time -- but it would be nice to do both).

[1]: https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/runc... [2]: https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/scri... [3]: https://github.com/opencontainers/runc/blob/v1.4.0-rc.1/scri... [4]: https://build.opensuse.org/projects/openSUSE:Factory/package...

Re: Tinycolor supply chain attack post-mortem

#60

Post by pabs3 »

cyberax wrote: post #13

> exfiltrated a npm token with broad publish rights I freaking HATE tokens. I hate them. There should be a better way to do authentication than a glorified static password. An example of how to do it correctly: Github as a token provider for AWS: https://aws.amazon.com/blogs/security/use-iam-roles-to-conne... But this is an exception, rather than a rule.

mTLS aka TLS client certs seems like the way to go.
Post reply on HN