Npm Audit: broken by design? • Hacker News Board

Re: Npm Audit: broken by design?

#91

Post by riho » Wed, Jul 07, 2021, 4:49 PM UTC

The the main problem is the fact that this audit happens with no context, and the audit results offer no information about the context an issue applies to either. Every issue should have a clear explanation about why and where it's an issue, and be tagged. Then we'd just need a way to hint npm what context a package will be used in, similarly to what we already do for devDependencies.

Also going through an audit result in a CLI isn't really the best experience. I wish I could just click a link and open up the report in a browser to drill down into issues.

Re: Npm Audit: broken by design?

#92

Post by cratermoon » Wed, Jul 07, 2021, 4:51 PM UTC

G3rn0ti wrote: post #76

cratermoon wrote: post #55

Earlier quoted context omitted.

Beautifully succinct. This quote: "Grey-hat hackers are rewarded based on the number and severity of CVEs that they write. This results in a proliferation of CVEs that have minor impact, or no impact at all, but which make exaggerated impact claims." Alignment of incentives is messed up. Goodhart-Strathern's and Campbell's laws apply.

Maybe then writing and submitting a CVE should cost some money that’s payed back together with the reward if the vulnerability is found to be „reasonable“ upon review?

I'm always suspicious of just throwing money at a problem, particularly in things like open source where money isn't always the motivator and can often be a corrupting influence. In some cases this will reduce the ability of genuinely well-intentioned people to participate, simply because they don't have the money up front, and for well-funded organizations the money would have to be quite a lot.

I'd like to ask what, other than money directly motivates people? Is it prestige? A line on their resume? A requirement for a bootcamp class? In addition, we should re-evaluate the difficulty of submitting a CVE. Is it too easy? The story about a mass of "hey your regex parser could choke on this weird expression[1]" reports suggest that perhaps so. What can we do to make it so that CVEs and equivalents are truly meaningful? Also, just the fact that CVE reports are given a great deal of respect could be the problem, although at this point that seems to be self-correcting.

[1] Some classes of regex parsers are known to be vulnerable by nature, those that do backtracking for example, because their worst-case runtime grows exponentially and can run in unbounded time. This has been known since at least 2009. There are other implementations with better worst-case runtimes, but worse performance in typical cases. The fact that it's trivially easy to look at a regex parser to see if it does backtracking and construct an "evil" expression that breaks it means it's trivially easy to file a DOS report against any such parser.

Re: Npm Audit: broken by design?

#93

Post by cratermoon » Wed, Jul 07, 2021, 4:53 PM UTC

watwut wrote: post #79

cratermoon wrote: post #51

Earlier quoted context omitted.

I agree about the false positive problem. Boy who cried wolf and all. I've also worked with security vendors who offer to run "free" vulnerability scans for you, and to absolutely nobody's surprise, they find vulnerabilities that just happen to be the ones that they can fix, if you buy what they are selling. Still, your example is problematic. Beware the "internal-only network". Such a thing has mostly lost meaning t…

The issue was "intentionally setup for testing purposes".

What testing purposes? How long was it up? Without knowing more, that doesn't eliminate it as a problem. There are plenty of cases where something set up "just for testing" ended up being the entry point for attackers.

Re: Npm Audit: broken by design?

#94

Post by jmull » Wed, Jul 07, 2021, 5:00 PM UTC

preinheimer wrote: post #58

Honestly, this hit me. I'm not a react developer, I was experimenting with it for a new project. I finished the tic-tac-toe tutorial, then tried to throw bootstrap on top to build from there. It told me there was 97 vulnerabilities (85 moderate, 12 high)... I just deleted the directory and went back to vanilla JS. This is a fun side project, I don't need that. My tweet about it: https://twitter.com/preinheimer/status…

FYI, it's probably not react-bootstrap or bootstrap or any of their dependencies being flagged. It's some of the other ~2K packages already installed.

Re: Npm Audit: broken by design?

#95

Post by bscphil » Wed, Jul 07, 2021, 5:02 PM UTC

jrochkind1 wrote: post #65

> Inline all dependencies during publish… From a maintainer’s point of view, the upsides are clear: you get faster boot time, smaller downloads, and — as a nice bonus — no bogus vulnerability reports from your users. And when you inlined a version that later really does have a vulnerability, it is not easily flagged or fixed by your consumers. The tension between "upgrades (especially of indirect dependencies) might…

Well said.

I would add that I think the devDependencies solution is underrated. Not using it is a bad practice. npm has a nice feature, `npm prune --production` which will remove all the dev dependencies for you resulting in a clean build of the program. You can easily have things set up so that none of the development dependencies that have all these audit issues are ever present on your production machines if you do things right.

Furthermore, if you have a project with end users, devDependencies allows you to make clear in your issue template that audit issues that don't show up in a production build are very likely to be false positives and will probably be closed without comment. If you aren't properly isolating your dependencies, you can't take advantage of this.

At the end of the day, as you say, the issue is largely with Node projects having too many dependencies and a large number of relatively minor issues that affect very specific use cases get reported. That's a hard ecosystem problem to solve, not necessarily something that indicates an inherent problem with npm audit; but if someone isn't using devDependencies, that alone could constitute an enormous improvement in their workflow.

Re: Npm Audit: broken by design?

#96

Post by smsm42 » Wed, Jul 07, 2021, 5:02 PM UTC

G3rn0ti wrote: post #76

cratermoon wrote: post #55

Earlier quoted context omitted.

Beautifully succinct. This quote: "Grey-hat hackers are rewarded based on the number and severity of CVEs that they write. This results in a proliferation of CVEs that have minor impact, or no impact at all, but which make exaggerated impact claims." Alignment of incentives is messed up. Goodhart-Strathern's and Campbell's laws apply.

Maybe then writing and submitting a CVE should cost some money that’s payed back together with the reward if the vulnerability is found to be „reasonable“ upon review?

AFAIK MITRE has a process for an organization to register as vendor, and then it would accept CVEs for their products only from the vendor, not from random people. Of course this has an opposite failure mode that may have unscrupulous vendors hide issues or just be lazy in issuing CVEs for existing bugs, but it eliminates the problem of random people issuing a ton of CVEs for non-issue bugs.

Re: Npm Audit: broken by design?

#97

Post by vbezhenar » Wed, Jul 07, 2021, 5:04 PM UTC

nih0 wrote: post #9

quintex wrote: post #6

As someone who only had just gotten into front end programming after years of backend work, npm has been a nightmare. I haven't experienced the same level of frustration with other package managers (pip, cargo, go mod, etc) as I have with npm. Is yarn the better option? What is our path forward?

same xd, npm even managed to make me appreciate maven and gradle

I think that there's difference between Java and JS landscapes.

Java has rich standard library. Some would say - too rich. But anyway - very few people would need customized collections, standard library covers all the needs. And where it does not cover all the needs, there are few commonly accepted additions like apache-commons or google guava. So that one solves `leftpad`-like issues.

Another difference is that Java is old. Most needs were covered by many libraries and few libraries survived which are good enough. It's again some commonly accepted wisdom, so you don't really need to search for many options. You have one good enough option that you'll use and move on.

So it's not about maven/gradle vs npm, it's more about ecosystems. I don't think that porting maven to JS world would change anything.

Re: Npm Audit: broken by design?

#98

Post by planb » Wed, Jul 07, 2021, 5:09 PM UTC

Funny. I had a discussion about the same topic in context of docker image security scanners last week. 100% agree with the author here!

Re: Npm Audit: broken by design?

#99

Post by munificent » Wed, Jul 07, 2021, 5:10 PM UTC

theptip wrote: post #85

jrochkind1 wrote: post #65

> Inline all dependencies during publish… From a maintainer’s point of view, the upsides are clear: you get faster boot time, smaller downloads, and — as a nice bonus — no bogus vulnerability reports from your users. And when you inlined a version that later really does have a vulnerability, it is not easily flagged or fixed by your consumers. The tension between "upgrades (especially of indirect dependencies) might…

> I'm not totally sure why they are so insane I think a big part of it is that due to much stronger pressure on bundle size than most other environments, each library tends to be small, so there have to be more to carry the same amount of functionality. Duplicates are certainly a contributing factor as well, and small bundles compound with allowed-duplication to further increase the tree size. I think that small pack…

I agree with all of this. Also JavaScript's "standard library" is nearly nonexistent (or at least was when Node first got big). That built a culture of people assuming they needed to reach for third-party dependencies for nearly everything (see: leftpad).

Re: Npm Audit: broken by design?

#100

Post by TheDong » Wed, Jul 07, 2021, 5:13 PM UTC

cphoover wrote: post #72

danabramov wrote: post #21

Earlier quoted context omitted.

>Just upgrade it to be on the safe side. A big part of the problem is there is no reliably way to "just upgrade it" today in npm: - `npm audit fix --force`, which is supposed to do that, is buggy and doesn't work - There is no way to override a transitive dependency with npm (there is with Yarn though, so hopefully this feature will come to npm soon) - Sometimes the fix in transitive dependency _also_ includes breaki…

> Because as the package owner I _do_ have that context. I understand there may be significant issues with this approach though! But what if one of your contributors slips in a merge that uses the vulnerable code path of your dependency... Does this "not affected" marker still exist, and now you have vulnerable code? Does it disappear with each version? What if someone maliciously adds a "not affected" marker? To a p…

> what if one of your contributors slips in a merge that uses the vulnerable code path of your dependency

If your threat model is a contributor submitting malicious code, your problem is not something npm audit will help with either way.

If a malicious actor is able to add the "not affected" marker, you have bigger problems.

The threat model you're talking about neither seems realistic, nor like something npm audit can help with. The attack vector of a contributor sneaking in malicious code is dealt with by only giving the commit bit to trusted people, and reviewing code yourself.