Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Npm Audit: broken by design?

overreacted.io

Post reply on HN
Print view View original

191–200 of 268 posts

Re: Npm Audit: broken by design?

#191

Post by IggleSniggle »

alerighi wrote: post #153
preinheimer wrote: post #58

Honestly, this hit me. I'm not a react developer, I was experimenting with it for a new project. I finished the tic-tac-toe tutorial, then tried to throw bootstrap on top to build from there. It told me there was 97 vulnerabilities (85 moderate, 12 high)... I just deleted the directory and went back to vanilla JS. This is a fun side project, I don't need that. My tweet about it: https://twitter.com/preinheimer/status…

That is exactly the point of the article. Every JS developer knows that these numbers are stupid and doesn't look at them. However a beginner that doesn't know what impact they have of course is scared if when installing a library it tells you that there are all that vulnerabilities.

As a beginner _just to npm_ I can imagine getting totally freaked out and worried that my whole system was _potentially_ compromised after seeing a “Critical” vulnerability reported as installed on my system.

After all, npm can execute any script with the users permissions on install…except often (compared to bash) it’s less easily inspected due to the common use of nested dependencies!

I, too, would delete my node_modules, and if I even wanted to move forward at that point, would probably waste at least half a day looking up the Critical vulns and discovering that they are probably not at all critical in my particular scenario. Like not at all for the 99.99% use case.

After experiencing something like that, it’s just like the article says. “The boy who called wolf.” Really terrible use of the labels “Critical” and “High”. The labels are fine, but the way they are applied is just stupid.

Re: Npm Audit: broken by design?

#193

Post by jrockway »

DaiPlusPlus wrote: post #188
jrockway wrote: post #174

Earlier quoted context omitted.

I wouldn't call a CVE a scarlet letter. Given the current state of software engineering, it's more like "my project is valuable enough to be used by someone that cares about security". You fixed it, one less bug to worry about. No doubt there are many less popular products with many worse vulnerabilities that don't have a CVE. Even OpenBSD had to change their tagline to "Only two remote holes in the default install,…

> You fixed it, one less bug to worry about. Those "bugs" can be features though - or the work involved to fix the bug meant that high-impact feature work - or other bugfixes, had to be postponed or even cancelled. Our SaaS frequently gets security "researchers" (read: people running online scanners) submitting emails through our contact-form informing us about click-jacking attacks on our login-page - the problem fo…

> there is no easy solution that makes the "I ran a tool, gimme $100" people go-away

I use the "mark as spam" button :)

Re: Npm Audit: broken by design?

#194

Post by staticassertion »

cratermoon wrote: post #34
esens wrote: post #29

I found that much of the underlying cause is those mass reporting regex denial of services as being high severity bugs. So many people are reporting these in tons of different projects: https://github.com/search?q=regex+denial+of+service&type=iss... Anyhow it is just annoying and they broke NPM Audit based on these reports. It is good to fix all possible bugs, but many of these are not anywhere close to the level of…

This kind of nonsense really goes back to the broken CVE process. https://opensourcesecurity.io/2021/03/30/its-time-to-fix-cve... Linux kernel maintainer Greg Kroah-Hartman has a similar opinion. https://github.com/gregkh/presentation-cve-is-dead/blob/mast... Edit: LWN mention https://lwn.net/Articles/801157/

I wouldn't take Greg's opinions on security too seriously.

Spender has a much more nuanced, informed view. I think it covers the issues of the CVE process well, but doesn't make the same mistakes that Greg does.

https://www.grsecurity.com/reports_of_cves_death_greatly_exa...

Re: Npm Audit: broken by design?

#195

Post by staticassertion »

DaiPlusPlus wrote: post #188
jrockway wrote: post #174

Earlier quoted context omitted.

I wouldn't call a CVE a scarlet letter. Given the current state of software engineering, it's more like "my project is valuable enough to be used by someone that cares about security". You fixed it, one less bug to worry about. No doubt there are many less popular products with many worse vulnerabilities that don't have a CVE. Even OpenBSD had to change their tagline to "Only two remote holes in the default install,…

> You fixed it, one less bug to worry about. Those "bugs" can be features though - or the work involved to fix the bug meant that high-impact feature work - or other bugfixes, had to be postponed or even cancelled. Our SaaS frequently gets security "researchers" (read: people running online scanners) submitting emails through our contact-form informing us about click-jacking attacks on our login-page - the problem fo…

> so there is no easy solution that makes the "I ran a tool, gimme $100" people go-away without causing a much bigger problem to now exist.

Maybe consider setting up a free-tier HackerOne bounty program? I think they triage to some degree on your behalf.

Re: Npm Audit: broken by design?

#196

Post by schneidmaster »

eropple wrote: post #35
danabramov wrote: post #28

Earlier quoted context omitted.

>it's annoying to have to comb through false positives and mark them as such There is no way to "mark them as such". That's half of the issue. The other half is that many people reporting these issues have not "opted into" any security tooling and don't understand its tradeoffs. They just ran `npm install`, and npm adopted default behavior of showing these warnings. For a lot of people this is their first programming…

> The other half is that many people reporting these issues have not "opted into" any security tooling and don't understand its tradeoffs. ... For a lot of people this is their first programming environment. This attitude makes me kind of uncomfortable. Like, I have taught software development to a decent number of folks, but I've always done so in a relatively isolated environment. If one is buying into web programm…

> This attitude makes me kind of uncomfortable.

I think the point though is that security warnings need to be actionable and high-signal. Experienced folks are absolutely tuning out the security warnings on npm install, because 95% of the warnings are like the examples in the post -- I know they don't affect me/my use case and there's nothing I can do about them anyway. The effect is only compounded for novices who run "npx create-react-app hello-world" and immediately see something incomprehensible about a vulnerability in react-scripts > webpack > watchpack > watchpack-chokidar2 > chokidar > glob-parent. It either discourages them from programming entirely or it teaches them to ignore security warnings.

I don't disagree with your overall point -- e.g. we should absolutely teach novices "here's what XSS is and how to avoid it" early and often. But if a dependency manager is going to surface a vulnerability alert every time I install dependencies, the alerts should be 1) high severity (to the point where I should actually stop using the package if I am unable to patch/upgrade) or 2) at least immediately actionable. The current npm audit implementation does the opposite -- 95% of the alerts are totally irrelevant to my actual security posture, and the suggested command to upgrade a vulnerable dependency is unreliable and can actually downgrade to an older, even-less-secure version (!).

Re: Npm Audit: broken by design?

#197

Post by zerkten »

indigochill wrote: post #163
zerkten wrote: post #142

Earlier quoted context omitted.

Isn't this an area where gamification and machine learning could actually be useful, if applied carefully? If people are competing for CVEs, then why not work out a way to better differentiate them them through scoring and make this visible. The goal would be for attention to shift to the scoring instead of only a CVE count. Offer both views of the world, so tools could still fall back on the problematic listings the…

We're kind of already doing scoring in that CVEs are usually graded on severity, but researchers are motivated to inflate the severity of CVEs they find. So the question you'd need to tackle is how does one apply a universal standard to measure the real impact of a CVE? I suspect it's an impossible challenge, but I only dip into this domain casually so maybe someone has better ideas.

I'm not making the claim it's a universal standard, but there are likely indications that some researchers are a different pedigree from others. A researcher reporting the same kind of low grade vulnerability probably shouldn't carry the same reputation score as other researchers.

I don't think there is a perfect way to do this, and I don't think there is an absolute standard that can be applied. It will be unfair to some people, but the system should have options for resolution when there are egregious mistakes. I'm not making the claim either, that the views of the data you are interested in are the ones I might be interested in. A good system would provide some different levels which itself is an incentive towards better research that would break through.

Re: Npm Audit: broken by design?

#198

Post by nirvdrum »

trunnell wrote: post #118
nirvdrum wrote: post #110

Earlier quoted context omitted.

If you, as an attacker, are going to make a fraudulent commit and change package.json, you could just pull in a bogus dependency or add a new script/command that gets run on the build/dev machine. I agree the bar to submitting a fraudulent commit is unfortunately too low for many teams. But, this also extends to package publishing, too. We've seen packages published through compromised dev keys or the maintainer gran…

I agree with all of your points except for the very last one. Are we shooting the messenger (npm audit) here? Seems like the problem is lack of trust and lack of information about security of dependencies. Npm audit is just pointing that out. The size of the problem makes it very uncomfortable.

I think where we disagree is I don't think npm audit is pointing out useful information. Or, at least, it's including a lot of useless information that makes finding the useful information hard to see. Now that it's the default when installing packages, most JS developers are going to have to deal with it. Given a lot of the data is not applicable, unnecessarily repetitive, and has severity mismatch with reality, I think most people are just going to ignore the output. If not that, they're going to blindly update packages and when that doesn't work due to dependency chains and such, they're going to start saddling open source maintainers with busy work. The net result is that when the next real mass security issue hits, it's going to get overlooked or ignored with all the other noise.

What could have been a useful tool for CI is not going to be enabled because who wants their build failing for a week while trying to orchestrate a mass dependency upgrade? I just took a look a Create React App (CRA)-based I have and "yarn audit" indicates 2,288 packages were scanned with 337 vulnerabilities (that's pretty good, I've seen it list over 500 vulnerabilities in the past). This is an app using CRA, with Relay as a code generating GraphQL client, and Antd as a component framework. There's also supporting tooling such as TypeScript, ESLint, and Prettier. Beyond that, very few dependencies are being used. Almost none of them are actually shipping in the product, whether that be due to them being build-related tools or because tree-shaking just strips most of the library out. It's a SPA hosted in CloudFront, so even DoS attacks have a very different profile than products where the server is rendering views. In this particular case, there's no shared user-generated content and XSS mitigations are in place, so the best a customer could do is DoS themselves.

I try to be very diligent about auditing dependencies before upgrading, but I can't reasonably extend that to transitive dependencies. 2,288 packages is an absurd number and this app isn't even doing all that much. I believe it's quite likely that it'd be impossible to get the npm/yarn audit report down to 0, given how quickly the JS ecosystem moves -- upgrade one dependency, deal with whatever API changes are needed, and now another has a vulnerability of some sort listed. It's certainly a burden on a small development team.

Just to be clear, I'm not advocating sticking your head in the sand and pretend everything's fine. I'm saying when you have vulnerabilities listed as high severity and you've verified they're not problematic in a project, then you've addressed the issue. But, now you have a tool that you need to satisfy and if you don't then you're going to miss the next security issue because CI shut off the "npm audit" check.

Re: Npm Audit: broken by design?

#199

Post by pmoleri »

esens wrote: post #29

I found that much of the underlying cause is those mass reporting regex denial of services as being high severity bugs. So many people are reporting these in tons of different projects: https://github.com/search?q=regex+denial+of+service&type=iss... Anyhow it is just annoying and they broke NPM Audit based on these reports. It is good to fix all possible bugs, but many of these are not anywhere close to the level of…

Having 20 supposeddly high risk issues about possible DoS when all come from a build dev-dependency is just totally useless. If only I could add whitelists, then it would be bearable. Like "I don't care about such kind of issue in a dev dependency".

Re: Npm Audit: broken by design?

#200

Post by pmoleri »

IggleSniggle wrote: post #186
JonathonW wrote: post #46

Earlier quoted context omitted.

A regex "denial of service" "vulnerability" could be important, if it shows up in code that processes untrusted input from end users. But NPM Audit has no idea of context-- a "critical" bug in `browserlist`, which, in this context, is never used outside the development process and never takes input outside of what's in my package.json, gets the same prominence (or more so, since it's early in alphabetical order) as a…

Agreed! A “Critical” bug in a dev context should mean something very different from a “Critical” bug in a prod context. A “Critical” devDependency bug should be either a direct threat to the developer’s context, either by infecting the dev machine or by injecting a supply-chain problem, worming it’s way into downstream contexts. npm audit is just not granular OR careful enough to address these issues appropriately.

Agree 100%
Post reply on HN