Remember this is a Microsoft product. They certainly have the resources to resolve this if they want to.
50% of new NPM packages are spam
51–60 of 325 posts
Re: 50% of new NPM packages are spam
#52Earlier quoted context omitted.
If the spammers only want to be indexed, then NPM should disable indexing for major search engines. But still allow it to be indexed other ways, which aren't unearthed on Google search. Other ideas include: do not index new packages before they've garnered enough downloads.
As a developer, I want npm package information and docs to show up in search. I frequently prefer pypi or cran results over others because then I can easily tell if it’s a usable package vs just some snippet. Especially cran because it has pretty rigorous entry requirements so being in cran is a signal of at least some minimal quality.
As a developer, I want npm package information and docs to show up in search.
What case is there when you want to find a package in NPM, and information about that package, using Google? If you want information about the package then it's find if the NPM package page is missing from the results - so long as you're getting the package's homepage or git repo then that's plenty. From there you can get to it's NPM page. If you know the package you're looking for, or if you know what you want to do, then searching NPM itself alone is fine.
Essentially, there is no overlap in the Venn diagram of "searching for a package" and "searching for information about a package". You want one or the other, not a results page with links to both.
If people realized this about their searches more then Google could fix a lot of spam problems.
Re: 50% of new NPM packages are spam
#53Spam problems can be solved by - Cross-Internet reputation system for accounts - Small fee on submission
Captcha is an alternative to small fee, cause solving it automatically costs money. Real fee will scare away almost all amateur developers and almost all professional developers who don’t already have a business account available.
I run a big web property with great SEO ranking, and captcha definitely does not deter the spam. A lot of this spam is posted by actual humans.
Re: 50% of new NPM packages are spam
#54I'm afraid it can get worse. What happens when there will be a proliferation of "looking legit npm packages" thanks to AI, full with ransomware? Currently I can't really figure out a one size fits all solution to that. Any idea?
Npmjs can do a lot to fight spam by collecting information about all http requests sent by logged in users (though GDPR may impose some limitations). In many cases this would allow knowing one spam package (e. g. reported by users) to uncover all or most submissions from the same threat actor by making an SQL query to analytical DB with the right parameters. But most abused services AFAIK don't pro-actively fight with spammers. AI will definitely would make it harder one can start with low hanging fruits - most spammers are not that sophisticated.
Re: 50% of new NPM packages are spam
#55Re: 50% of new NPM packages are spam
#56Remember this is a Microsoft product. They certainly have the resources to resolve this if they want to.
A Microsoft product doesn't mean the full capacity of the company will be devoted to resolve it. In a big company almost all products have to fight very hard for additional resources, they are not given resources just because the company as a whole made tons of profits.
Re: 50% of new NPM packages are spam
#57Is this spam not easily mitigated by simple Bayesian approaches and collection of link features by visiting them?
Sure, but removing or unlisting a valid package could break projects. The folks maintaining the package ecosystem need to be careful.
Let’s say there’s 10 spam uploads per hour and it takes you 1 second to verify a package is spam and remove it. That’s 30 minutes a week just dealing with spam. While I was on the .NET package manager, we had the on-call engineer handle this thankless chore.
Could you detect these packages at upload time? Yes, but spammers will change their patterns once the package ecosystem gets too effective at detecting current patterns. Perhaps machine learning could help, but often times package manager teams are small and don’t have expertise in this area. Regardless, package removals require human review.
Re: 50% of new NPM packages are spam
#58Spam problems can be solved by - Cross-Internet reputation system for accounts - Small fee on submission
Do you have an example for cross-community reviews?
Essentially this is what academic journals are doing.
Every paper should be reviewed manually. Of course that costs some money (although the reviewers aren’t paid).
Re: 50% of new NPM packages are spam
#59Earlier quoted context omitted.
I would find amusement if the solution to the spamming of npm turns out to be a genuinely useful use case for blockchain.
> I would find amusement if the solution to the spamming of npm turns out to be a genuinely useful use case for blockchain. I think you can implement a web-of-trust without a blockchain.
You can (because it's been done); this is a use case where "distributed but extremely slow database" is a pretty natural fit for the problem.
Re: 50% of new NPM packages are spam
#60Spam problems can be solved by - Cross-Internet reputation system for accounts - Small fee on submission
> - Cross-Internet reputation system for accounts Gets rid of anonymous spam. > - Small fee on submission Gets rid of amateur spam. I guess that's 98% of the problem. I think this is a good start. What to do about bogus projects sponsored by wealthy companies? What about abandonware? And how do we remain open and inclusive to newbees?
Small fees can't be the same for every country: say, what is small in the US is hefty in Kenya, and what is small in Kenya is negligible in the US.