Always Review Your Dependencies, AGPL Edition • Hacker News Board

Re: Always Review Your Dependencies, AGPL Edition

#141

Post by oliwarner » Mon, Jan 06, 2020, 1:11 PM UTC

lmm wrote: post #98

cygned wrote: post #41

I try to convince my team that the node.js ecosystem has gotten into a stage where it cannot be used for security/financial applications because the sheer amount of dependencies pose an inherent threat. I advocate for Go because of the tendency to less and easier reviewable dependencies. Nobody except me seems to see a problem there, despite me being able to point out specific security incidents. I am wondering if I…

Dependencies are cattle, not pets. There's nothing wrong with having zillions of them; what you need is good tools to manage them in bulk. In the JVM ecosystem, projects are only allowed in the Maven Central repository if they have their license documented in a machine readable fashion there; it's then trivial to check what licenses you're depending on (and there are several plugins available for doing so). I'm amaze…

> Dependencies are cattle...

... Which also happen keep their own cattle that you're still responsible for.

It's not so bad in languages with solid standard libraries. In Python projects I might have 20 direct deps, ~50 indirect.

In a real JS project I'm building, I have 17 direct, 3829 indirect. The JS standard library is so damned thin that everything pulls in some random version of the kitchen sink.

    yarn list | sed -E 's/.*─ //' | sort -u | wc -l  # minus 2

In situations like that your job of auditing licenses, updates, sec issues, etc balloons exponentially with each new dependency.

Tooling should absolutely be used, but it still doesn't perform the job of working out whether or not you want to upgrade a component, or whether or not you're likely to have suffered a security breach, or how to report on how well audited your dependencies are.

Re: Always Review Your Dependencies, AGPL Edition

#142

Post by zxcmx » Mon, Jan 06, 2020, 1:17 PM UTC

cygned wrote: post #126

anton_gogolev wrote: post #116

Earlier quoted context omitted.

> There's nothing wrong with having zillions of them... There's nothing wrong until something goes wrong an now you're royally screwed. With zillion dependencies you are at a mercy of zillion maintainers, and none of them has any obligation to you. They can break backwards compatibility in patch releases, introduce subtle behavior changes, steer the project in an unexpected direction or abandon it altogether.

That goes into the direction of my thinking. I am concerned about transitive security issues. It is impossible to check in node dependencies into version control (size/binaries). They have a lock file to pin versions, but dependencies that are downloaded upon each build are are not reproducible from my point of view. With Go, it’s easy to vendor and check in, it’s also straight forward to review them. There have been…

Not a huge fan of node or anything but npm lock files do pin to a hash. Also in commercial world you're going to be pulling through nexus or some other cache to reduce bandwidth use and developer downtime.

Are there other reproducibility concerns I should be worrying about? Are you thinking npm modules with native code or that (this does happen!) actively pull other stuff during build? Most of those do their own pinning but agree the whole thing is messy.

Re: Always Review Your Dependencies, AGPL Edition

#143

Post by csdreamer7 » Mon, Jan 06, 2020, 1:20 PM UTC

cygned wrote: post #41

I try to convince my team that the node.js ecosystem has gotten into a stage where it cannot be used for security/financial applications because the sheer amount of dependencies pose an inherent threat. I advocate for Go because of the tendency to less and easier reviewable dependencies. Nobody except me seems to see a problem there, despite me being able to point out specific security incidents. I am wondering if I…

My biggest issue with the massive dependencies is not licensing or security. Both are important, but there is an even bigger block.

My biggest issue is with things in the node ecosystem is breakage, especially React libraries. I was following a NetNinja tutorial series on Youtube. I had to use a Firebase-Redux library for a combined React/Redux/Firebase series. There was a security vulnerability with something down the stack only a few months after the tutorial was out. So to use the updated version I had to use the beta version of a Firebase library which broke the tutorial code despite the library change being a point release like 1.4.0-beta2. Either 1.3.0 to 1.4.0 or 1.2.0 to 1.4.0 would be consider a breaking release for this particular dev.

We have a large group of dependencies chained upon each other with maintainers that have different support commitments and styles. React might be the de facto JS system, but it relies on a large system of these 3rd party packages to do deeper functionality which can so easily break things. I already had to peg React itself to an older version to use this tutorial when it was months old.

I realized at this point how much of a pain it would be to maintain this code in my portfolio if I did something with it. Let alone a paid product. I stopped learning React at this point (or at least learning React+Redux+Firebase).

Ruby sorta has this issue too, but it feels the Node ecosystem is much more accelerated. The Rails framework dominates the gem ecosystem so much usually gems break functionality by the version of Rails they support. So gems often will not update or warn of breaking changes based on the Rails version or ActiveRecord which follows Rails versioning. This makes gem dependencies much more manageable.

The node ecosystem does not have a large dominating framework like Rails. Even if you considered React that framework, Facebook breaks it whenever they please. 16.4 which sound like a minor version update break things. While Rails 4.2 was supported for 4 years. jQuery might be years before it breaks support.

Re: Always Review Your Dependencies, AGPL Edition

#144

Post by fenomas » Mon, Jan 06, 2020, 1:20 PM UTC

colonwqbang wrote: post #110

fenomas wrote: post #64

Earlier quoted context omitted.

Well, he talks about reviewing when he upgrades a dependency, but the tricksy thing about event-stream was that people could get the malicious change without having intended to upgrade anything. The only thing that really prevents such issues is version locking of transitive dependencies (which the author doesn't mention, but it could be that his package manager does it by default, or similar..).

I had missed that npm doesn't (didn't?) lock package versions by default. That's really scary.

It does today, and I could be crazy but I think it did even before event-stream. But of course some users will have been on old versions of npm, or not checking in their lockfiles, etc.

I should add, I was probably wrong to talk about lockfiles preventing such issues. With event-stream the malicious code was hidden deviously enough to evade a pretty rigorous check - if a Node update hadn't deprecated one of the functions used in the payload, I suppose we might still not know about it. In such cases a lockfile is at least a layer of defense, but naturally it only helps if you're lucky enough to have installed the library before it got corrupted..

Re: Always Review Your Dependencies, AGPL Edition

#145

Post by grammarxcore » Mon, Jan 06, 2020, 1:24 PM UTC

marcan_42 wrote: post #72

> The bulk of my trust is consolidated in the Go project, and thanks to their stellar reputation and solid operating procedures, I don't feel a need to review the source code of the Go compiler and standard libraries. Well... I have reviewed the Go runtime, after I ran into a bug in it... And that's one of the reasons I no longer use Go unless forced to. That thing is ugly under the covers. There is seriously quite a…

I ran into a speed issue last year messing around with Go RSA keys. Turns out it's an open bug. Even though fixes have been made (see links in the thread), it's not a global fix and Go ciphers can be absysmally slow.

https://github.com/golang/go/issues/20058

The recommended fix is to use a library like this one. However, that means your containers blow up with complicated dependency trees so it's not really a good solution for a distributed container architecture (eg Kubernetes).

https://github.com/ncw/gmp

I love working with Go because of its simple binaries and small containers, but there are some things that it just does not do well.

Re: Always Review Your Dependencies, AGPL Edition

#146

Post by sime2009 » Mon, Jan 06, 2020, 1:36 PM UTC

iforgotpassword wrote: post #76

cygned wrote: post #41

I try to convince my team that the node.js ecosystem has gotten into a stage where it cannot be used for security/financial applications because the sheer amount of dependencies pose an inherent threat. I advocate for Go because of the tendency to less and easier reviewable dependencies. Nobody except me seems to see a problem there, despite me being able to point out specific security incidents. I am wondering if I…

I'm not a web dev, not a JavaScript fan, but out of pure curiosity I've been playing with the idea of doing my next side project in node, just to get an idea of what modern Js feels like. But the whole dependency hell, left-pad and the likes are a real turnoff. So what I'd probably end up with is coding in pure Js without a package manager or "build system". Basically like you did PHP in the 90s. The question is whet…

It is certainly possible to do JS development in a more controlled way.

My advice is to avoid most of the JS build tools (i.e. grunt, gulp) by using npm scripts (basically shell commands defines in `package.json`) to trigger build actions, or to start smaller, more specialised tools. At least then you will understand how your build works and won't have an extra layer of buggy plugins screwing things up.

Also being conservative and critical of the dependencies you take on is a great idea. Many smaller dependencies are not worth it. Bigger popular libraries like lodash are often a better deal. Unfortunately this micro-module philosophy and "publish random trash to npmjs.com" attitude has created a huge amount of crud packages.

I've had success organising a bigger project into multiple node packages inside the same git repository and using yarn's workspaces feature to tie it altogether. I avoid the complexity of publishing those (private) packages on npmjs.com. yarn's workspaces make it possible for my main app to find its (internal) package dependencies in the same git repo. is certainly possible to do JS development in a more controlled way.

My advice is to avoid most of the JS build tools (i.e. grunt, gulp) by using npm scripts (basically shell commands defines in `package.json`) to trigger build actions, or to start smaller, more specialised tools. At least then you will understand how your build works and won't have an extra layer of buggy plugins screwing things up.

Also being conservative and critical of the dependencies you take on is a great idea. Many smaller dependencies are not worth it. Bigger popular libraries like lodash are often a better deal.

I've had success organising a bigger project into multiple node modules inside the same git repository and using yarn's workspaces feature to tie it altogether. I avoid the complexity of publishing those (private) modules on npmjs.com. yarn's workspaces make it possible for my main app to find its (sub-)module dependencies in the same git repo.

Re: Always Review Your Dependencies, AGPL Edition

#147

Post by jannes » Mon, Jan 06, 2020, 1:38 PM UTC

pas wrote: post #114

krapp wrote: post #94

Earlier quoted context omitted.

Modern JS is the current ECMAScript standard. Typescript is not "modern JS", any more than Coffeescript was, any more than the compile-to-js language that will probably replace it in a week. You're confusing the tool for the language the tool operates on, akin to claiming modern C++ is Visual Studio.

Not using TS means leaving a lot of useful tools on the floor. The ECMAScript standardization is process is invaluable, after all TS builds upon it. But TS provides a saner subset (and adds a very productive type system), without that I wouldn't touch "modern" JS even with a stick.

What do you mean with TS provides a subset of JS? TS is actually a superset of JS.

Quoting the TypeScript website:

> TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.

https://www.typescriptlang.org/

Re: Always Review Your Dependencies, AGPL Edition

#148

Post by pjc50 » Mon, Jan 06, 2020, 1:51 PM UTC

lmm wrote: post #98

cygned wrote: post #41

I try to convince my team that the node.js ecosystem has gotten into a stage where it cannot be used for security/financial applications because the sheer amount of dependencies pose an inherent threat. I advocate for Go because of the tendency to less and easier reviewable dependencies. Nobody except me seems to see a problem there, despite me being able to point out specific security incidents. I am wondering if I…

Dependencies are cattle, not pets. There's nothing wrong with having zillions of them; what you need is good tools to manage them in bulk. In the JVM ecosystem, projects are only allowed in the Maven Central repository if they have their license documented in a machine readable fashion there; it's then trivial to check what licenses you're depending on (and there are several plugins available for doing so). I'm amaze…

"Servers are cattle, not pets" relies on the servers being substantially identical, cloned like bananas from a single source where each is as good as the other.

Dependencies are, if they are to be useful at all, all different. Dependencies are suppliers, in the business sense. Having lots of dependencies loaded at runtime is like a modern just-in-time giant supply chain; it lets you take advantage of efficiencies in exchange for being more brittle.

Or they are like BOM items on a circuit board. Part of the original drive to "componentise" software came from people experienced in electronic engineering; you don't have to reinvent the transistor, you just buy them at a cost of a few dollars for a reel of thousands. But experienced designers will still try to:

- choose more-common components wherever possible

- ensure there are multiple sources for a component

- reduce the overall number of BOM lines, which reduces supply risk and inventory cost

The software world would go completely bananas if the cost for dependencies was not exactly zero. Imagine having to license left-pad.

Re: Always Review Your Dependencies, AGPL Edition

#149

Post by zeveb » Mon, Jan 06, 2020, 1:51 PM UTC

marcan_42 wrote: post #105

self wrote: post #88

Earlier quoted context omitted.

> That thing is ugly under the covers. Was the code you looked at in Go 1.5 and above? Is it cleaner in 1.4? They used automated tools to convert the C code into Go for 1.5.

They were halfway through the process when I made my contribution, and in fact I had to rewrite it in Go. The Go version was just a straight conversion; being Go didn't make it inherently cleaner.

I think his point is that it stands to reason that machine-converted C→Go code may be pretty ugly. Maybe it was better when it was clean, Plan-9-style C? Maybe it is better now, after some number of years of cleanup?

Re: Always Review Your Dependencies, AGPL Edition

#150

Post by iSnow » Mon, Jan 06, 2020, 1:55 PM UTC

codeflo wrote: post #121

anton_gogolev wrote: post #116

Earlier quoted context omitted.

> There's nothing wrong with having zillions of them... There's nothing wrong until something goes wrong an now you're royally screwed. With zillion dependencies you are at a mercy of zillion maintainers, and none of them has any obligation to you. They can break backwards compatibility in patch releases, introduce subtle behavior changes, steer the project in an unexpected direction or abandon it altogether.

I’m a bit torn on this. I have most of my experience in the .NET ecosystem, where dependencies are a lot more manageable. However, if something breaks, you’re screwed a lot harder, because it’s not so easy to replace a large library, and there are very likely fewer well-maintained alternatives than there would be on NPM. In total, I find it hard to deny how productive the NPM ecosystem can be, despite my philosophica…

You aren't alone in this. The Node/NPM/JS scene is churning out code and innovations like there's no tomorrow, that's something to admire.

What I feel they are missing is a community process to consolidate things. You don't need three generations of ten incompatible solutions for a given problem - after some iterations, things should consolidate into one or two more or less standardized libs that don't break existing code at every damn point release.