Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Npm Audit: broken by design?

overreacted.io

Post reply on HN
Print view View original

21–30 of 268 posts

Re: Npm Audit: broken by design?

#21

Post by danabramov »

raziel2p wrote: post #16
mbesto wrote: post #13

> this “vulnerability” is absurd in this context Yes that's exactly the point. The audit tool has no awareness of the context and nor do the people who create severities. If severities were absolute then there would be no reason for anyone to review them. You would simply upgrade your libraries and be done with it, but that can't always be achieved nor may make business sense. I do agree with the author's note about…

Exactly. Why does it matter if it's absurd in this context? Just upgrade it to be on the safe side. Asking vulnerability databases to judge whether vulnerabilities are safe in devDependencies or not is a ridiculous idea, even more so when you consider that the line between static and dynamic have gone blurry long ago.

>Just upgrade it to be on the safe side.

A big part of the problem is there is no reliably way to "just upgrade it" today in npm:

- `npm audit fix --force`, which is supposed to do that, is buggy and doesn't work

- There is no way to override a transitive dependency with npm (there is with Yarn though, so hopefully this feature will come to npm soon)

- Sometimes the fix in transitive dependency _also_ includes breaking changes (e.g. because it wasn't backported), and so updating it subtly breaks the logic

>Asking vulnerability databases to judge whether vulnerabilities are safe in devDependencies or not is a ridiculous idea

I don't think databases can do it, but what I'd like to be able to do is to be able to provide advisory that the way _my package_ uses a concrete transitive dependency is not affected by that vulnerability. Because as the package owner I _do_ have that context. I understand there may be significant issues with this approach though!

Re: Npm Audit: broken by design?

#22

Post by aerojoe23 »

It sounds like npm needs a mitigated and irrelevant flag, these flags should include an explanation field. Security teams would also have to accept this as a solved/fix status.

For projects you own you'd have to flag each dependency path though, because for example, one dependency may not have the input for the regex exposed to the end user, while another dependency could.

Maintainers of libraries should also flag the security issues, and an issue with these two statuses on them wouldn't be raised by default. Options should be available to list them though for auditing.

For more security critical teams/projects, a per project setting to alert about any issues the maintainers have flagged irrelevant or mitigated and you'd have to accept them before it it would stop alerting about them.

Re: Npm Audit: broken by design?

#23

Post by lxe »

I agree with the author. Just like them, I only write code without any bugs that can be affected by the "vulnerabilities". I also never commit to upstream so others won't be able to edit my code. All my projects only run on my machine (which is of course also absolutely exploit-free and it's not connected to the internet).

Re: Npm Audit: broken by design?

#24

Post by danabramov »

richardwhiuk wrote: post #7

I think they are just saying that context matters in security vulnerabilities, and npm audit doesn't have that context. Well yes, correct, well done. By this metric, every security tool ever written is probably pointless.

Note that `npm audit` runs _during every install_ so people who use it don't necessarily consciously understand what's happening. Many of them are beginners and have never used a security tool before (or even want to use it).

Re: Npm Audit: broken by design?

#25

Post by minxomat »

dale_glass wrote: post #20

It's a tricky problem to solve. Ideally you'd want to show only relevant alerts, but... how? You'd need to know which kind of errors are relevant for a particular project, but that'd require solving the halting problem. This is made much worse by that it's JS. Some libraries have an enormous complexity and attack surface. Take a database interface -- there probably is a vulnerability in some obscure corner the typica…

You don't have to go all out doing exhaustive dynamic analysis. Data flow analysis gets rid of 99% of the most popular bugs (injection, validation, defaults, etc.). GitHub CodeQL can do this, and produces much better results than any static analysis tool I've ever used. Feeding this data back into npm (owned by GH) is just the next step.

Re: Npm Audit: broken by design?

#26

Post by unanswered »

gitgud wrote: post #18

Agreed, you shouldn't see low risk security warnings by default, they're more appropriate for larger projects, with something to lose. The [1] yarn package manager is much nicer to work with in many more ways. [1] https://classic.yarnpkg.com/en/

TFA identifies "high risk" false positives too, so this response doesn't seem to have anything to do with them problem as stated by TFA.

Re: Npm Audit: broken by design?

#27

Post by bogota »

nonameiguess wrote: post #19

It isn't broken by design. I sympathize and understand it's annoying to have to comb through false positives and mark them as such, but until a level of AI we're nowhere near being near exists, you can't automate this process, so the only alternative is to ignore all vulnerabilities. Complaining about this is misunderstanding the asymmetrical costs of different types of statistical error. Vulnerability scanners are s…

That is not the cost of false positives. The cost of false positives is ignoring it completely. I have had jobs where we just block the security scanners because they won’t listen to any feedback about why what they are scanning was intentionally setup for the purpose of testing vulnerabilities on an internal only network. Additionally at other jobs security tickets just start to get ignored because they send too many tickets that do not matter. I feel the security field likes to ignore most feedback and play holier than though. At all companies i worked at i have only had one good security team that worked with people instead of just throwing things over the wall.

Re: Npm Audit: broken by design?

#28

Post by danabramov »

nonameiguess wrote: post #19

It isn't broken by design. I sympathize and understand it's annoying to have to comb through false positives and mark them as such, but until a level of AI we're nowhere near being near exists, you can't automate this process, so the only alternative is to ignore all vulnerabilities. Complaining about this is misunderstanding the asymmetrical costs of different types of statistical error. Vulnerability scanners are s…

>it's annoying to have to comb through false positives and mark them as such

There is no way to "mark them as such". That's half of the issue. The other half is that many people reporting these issues have not "opted into" any security tooling and don't understand its tradeoffs. They just ran `npm install`, and npm adopted default behavior of showing these warnings. For a lot of people this is their first programming environment.

>you can't automate this process, so the only alternative is to ignore all vulnerabilities.

I don't think the answer is necessarily automation. But as a package author, I'd like to be able to mark somewhere that a particular transitive vulnerabilities can't affect my users.

Or at least I'd like npm to offer a reliable way to update packages deep in the tree and override the resolved versions. Currently, there isn't such a way.

Re: Npm Audit: broken by design?

#29

Post by esens »

I found that much of the underlying cause is those mass reporting regex denial of services as being high severity bugs.

So many people are reporting these in tons of different projects: https://github.com/search?q=regex+denial+of+service&type=iss...

Anyhow it is just annoying and they broke NPM Audit based on these reports.

It is good to fix all possible bugs, but many of these are not anywhere close to the level of bad that the reports are making them to be.

But maybe this is needed to just get rid of these issues in genera? So a wave of regex vulnerability reports and then we build this type of checking into prettier or similar and we do not have these in the future?

EDIT: It appears there as a project that found 100s of CVE reported Regex vulnerabilities in npm projects -- this is maybe one of the sources of mass reports. See the bottom of this resume: https://yetingli.github.io

Re: Npm Audit: broken by design?

#30

Post by esprehn »

Dan isn't the first person to notice:

https://www.voitanos.io/blog/don-t-be-alarmed-by-vulnerabili...

We disable [1] audit entirely because it's not a good default behavior within a monorepo. It spams the hundreds of developers with the list of "vulnerabilities" on every install, but only a few folks should really be upgrading packages.

We then run audit in non-blocking CI and track the total number of issues and mostly focus on critical ones.

[1] https://docs.npmjs.com/cli/v7/using-npm/config#audit

Post reply on HN