Live data from Hacker News

70% of new NPM packages in last 6 months were spam

blog.phylum.io

31–40 of 116 posts

Re: 70% of new NPM packages in last 6 months were spam

#31

> Next, because the AI hype train is at full steam, we must point out the obvious. AI models that are trained on these packages will almost certainly skew the outputs in unintended directions. These packages are ultimately garbage, and the mantra of “garbage in, garbage out” holds true. hmm, inspiring thoughts. An answer to "AI is going to replace software developers in the next 10 years" is to create 234876238562856…

Implying that npm wasn’t garbage before AI was mainstream?

Correct.

Re: 70% of new NPM packages in last 6 months were spam

#32
post #12

Earlier quoted context omitted.

A "better" way is to modify the package-lock.json. You can still spoof the package but almost no one actually reviews it as npm will usually modify 1000s of lines. for example take mongoose "resolved": "https://registry.npmjs.org/mongoose/-/mongoose-8.4.4.tgz", "integrity": "sha512-Nya808odIJoHP4JuJKbWA2eIaerXieu59kE8pQlvJpUBoSKWUyhLji0g1WMVaYXWmzPYXP2Jd6XdR4KJE8RELw==", so long as the integrity check passes for the…

Hugely surprising that package.json and package-lock.json don't have to match. The way I would expect it to work is something like: for d in dependencies_from_package_json() get_package(d) if hash_package(d) != package_lock_hash(d) error() end end And not: use_package_lock_and_ignore_package_json_lol_fuck_you_haha_kthxbye() I also discovered that npm doesn't actually verify what's in node_modules when using "npm inst…

I think that's true for most package managers. That if there's a lock file, there's typically a default command to use it for installs and ignore the main config file.

Re: 70% of new NPM packages in last 6 months were spam

#33
post #12

Earlier quoted context omitted.

A "better" way is to modify the package-lock.json. You can still spoof the package but almost no one actually reviews it as npm will usually modify 1000s of lines. for example take mongoose "resolved": "https://registry.npmjs.org/mongoose/-/mongoose-8.4.4.tgz", "integrity": "sha512-Nya808odIJoHP4JuJKbWA2eIaerXieu59kE8pQlvJpUBoSKWUyhLji0g1WMVaYXWmzPYXP2Jd6XdR4KJE8RELw==", so long as the integrity check passes for the…

Hugely surprising that package.json and package-lock.json don't have to match. The way I would expect it to work is something like: for d in dependencies_from_package_json() get_package(d) if hash_package(d) != package_lock_hash(d) error() end end And not: use_package_lock_and_ignore_package_json_lol_fuck_you_haha_kthxbye() I also discovered that npm doesn't actually verify what's in node_modules when using "npm inst…

> But luckily "npm audit" will warn us about 30 "high severity" ReDos "high impact" "vulnerabilities" that can never realistically be triggered and are not really a "vulnerability" in the first place, let alone a "high impact" one.

Yeah, you want to be using a tool that lets you ignore/acknowledge specific entries.

`npm audit` is not an end-all-be-all.

Like and subscribe[0]: https://github.com/npm/rfcs/pull/18

https://www.npmjs.com/package/npm-audit-resolver

[0]: The bottom comment from Jan sums up what happens when Microsoft steps up...

Re: 70% of new NPM packages in last 6 months were spam

#34
post #24
post #8

How about removing the incentive? Take down every package with tea.yaml in it, after say 1 month's warning, so legitimate packages trying to use it don't leave their users in the lurch. The tea protocol is clearly not going to accomplish what it set out to (see below), and is instead incentivising malicious behaviour and damaging the system it set out to support. From https://docs.tea.xyz/tea/i-want-to.../faqs : "tea…

That would be a clear violation of the npm Unpublish Policy[0]. If all it takes is some spam and pissing people off to walk away from principles, they never meant anything. A proper response needs to not break expectations like this. [0]: https://docs.npmjs.com/policies/unpublish

Do principles matter if a registry becomes seen as spam or a security risk due to refusing to take action?

Re: 70% of new NPM packages in last 6 months were spam

#35
post #24
post #8

How about removing the incentive? Take down every package with tea.yaml in it, after say 1 month's warning, so legitimate packages trying to use it don't leave their users in the lurch. The tea protocol is clearly not going to accomplish what it set out to (see below), and is instead incentivising malicious behaviour and damaging the system it set out to support. From https://docs.tea.xyz/tea/i-want-to.../faqs : "tea…

That would be a clear violation of the npm Unpublish Policy[0]. If all it takes is some spam and pissing people off to walk away from principles, they never meant anything. A proper response needs to not break expectations like this. [0]: https://docs.npmjs.com/policies/unpublish

Principles are a means to an end, not an end in themselves. The end here (presumably) is a healthy ecosystem, an end which this principle arguably harms more than it helps. Rigid and unthinking adherence to principles is dogmatic, and dogma has no place in engineering.

Re: 70% of new NPM packages in last 6 months were spam

#36
post #24
post #8

How about removing the incentive? Take down every package with tea.yaml in it, after say 1 month's warning, so legitimate packages trying to use it don't leave their users in the lurch. The tea protocol is clearly not going to accomplish what it set out to (see below), and is instead incentivising malicious behaviour and damaging the system it set out to support. From https://docs.tea.xyz/tea/i-want-to.../faqs : "tea…

That would be a clear violation of the npm Unpublish Policy[0]. If all it takes is some spam and pissing people off to walk away from principles, they never meant anything. A proper response needs to not break expectations like this. [0]: https://docs.npmjs.com/policies/unpublish

How about banning it going forward instead?

Re: 70% of new NPM packages in last 6 months were spam

#37
post #24
post #8

How about removing the incentive? Take down every package with tea.yaml in it, after say 1 month's warning, so legitimate packages trying to use it don't leave their users in the lurch. The tea protocol is clearly not going to accomplish what it set out to (see below), and is instead incentivising malicious behaviour and damaging the system it set out to support. From https://docs.tea.xyz/tea/i-want-to.../faqs : "tea…

That would be a clear violation of the npm Unpublish Policy[0]. If all it takes is some spam and pissing people off to walk away from principles, they never meant anything. A proper response needs to not break expectations like this. [0]: https://docs.npmjs.com/policies/unpublish

Pragmatism trumps principles. In this case, it is better to unpublish these packages, than turn npm registry into a bigger garbage

Re: 70% of new NPM packages in last 6 months were spam

#38

I am really interested if that really matters. Package managers often comes with rating system. npmjs has weekly downloads, pull requests, and other popularity scores. I am layman in AI, but why would anyone think that this would affect anything, like AI? Why would anyone train on noname package, that noone uses? Stats for spam packages can have higher-than-none stats, but that also makes them vulnerable for sweep re…

> Why would anyone train on noname package, that noone uses?

Not that I disagree, but in the same line of thinking: Why would anyone train an LLM on some random blog written in broken English? Why would you train an LLM on the absolute dumpster fire that is Reddit comments? Or why is my Github repos with half-finished projects and horribly insecure coding practises being used as input to CoPilot? Yet here we are, LLMs writing broken, insecure code (just like a real person) and telling people to eat rocks.

Re: 70% of new NPM packages in last 6 months were spam

#39
post #29
post #28

Earlier quoted context omitted.

That's just "rm -r node_modules && npm install". Aside from needlessly taking a long time, it also means we're back to "if something goes wrong we're left with an inconsistent node_modules".

No, it's more than that. Did you read the documentation page linked in the comment you replied to? > But what about after the command has run? If you munge around in node_modules after a successful `npm ci`, that's on you. If you run scripts that do, that's on you. If you depend on packages that run such scripts, that's on you. > What, you mean I'm supposed to audit my dependencies myself? That's too much work! Yes,…

It says exactly that: "If a node_modules is already present, it will be automatically removed before npm ci begins its install".

I didn't "munge around" in node_modules; I said "if something goes wrong". Like I said in my previous comment: "I found this out a few ago after I had some corrupted files due to a flaky internet connection". That's not munging around, that's computers being computers. Network errors happen. Disk errors happen. Memory errors happen. Things like that. I've also had an install ISO corrupted at some point. I always check the sums since, just in case because there was a lot of confusion involved before I found out the ISO was just downloaded wrong for some reason. Stuff doesn't often get randomly corrupted, but it does happen, and with 2GB ISO files (or 2GB node_modules) the chances do grow.

On Go I can do "go mod verify". I think yarn has "yarn check" for this (but didn't verify). I don't know about other package managers off-hand, but if they don't have something for it, they should. You need to be able to verify the content on disk is identical to what's expected.

I never mentioned anything about auditing dependencies or AI.

Your entire post is a masterclass in arguing against things that were never claimed and forceful injection of your own bugbears. I just want to check if node_modules is identical to what's expected, just in case, because computers kind of suck and are unreliable.

Re: 70% of new NPM packages in last 6 months were spam

#40
post #32
post #12

Earlier quoted context omitted.

Hugely surprising that package.json and package-lock.json don't have to match. The way I would expect it to work is something like: for d in dependencies_from_package_json() get_package(d) if hash_package(d) != package_lock_hash(d) error() end end And not: use_package_lock_and_ignore_package_json_lol_fuck_you_haha_kthxbye() I also discovered that npm doesn't actually verify what's in node_modules when using "npm inst…

I think that's true for most package managers. That if there's a lock file, there's typically a default command to use it for installs and ignore the main config file.

Yeah maybe, I don't really know off-hand and I'd have to check. I know it's not possible in Go but not sure about anything else. I'd consider it hugely surprising for other packagers where that's possible too. Who is carefully auditing if the repo URL in the lockfile is actually the correct one?
Post reply on HN