Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ride down into JavaScript dependency hell

blog.appsignal.com

Post reply on HN
Print view View original

61–70 of 149 posts

Re: Ride down into JavaScript dependency hell

#61

Post by BiteCode_dev »

matheusmoreira wrote: post #48
dwheeler wrote: post #13

Part of the issue is that JavaScript packages are often far far smaller than packages in other ecosystems. "small packages are extremely common in the JavaScript npm package system. For example, in npm, 47% of the packages have 0 or 1 functions, and the average npm package has 112 physical lines of code. In contrast, the average Python module in the PyPI repository has 2,232 physical lines of code." Source: "Vulnerab…

Why are small packages bad? Independent functions should be versioned and distributed independently. Otherwise we get several utility packages which are nothing but collections of independent functions. Widely used packages should form the basis of a standard library that is distributed with the language itself.

Small packages means more packages, and more packages compounds the dependency graph which:

- makes dependency resolution harder

- means more points of failure

- makes auditing difficult

- makes install time longer

- leads to harder maintenance and upgrade

This also causes heterogeneity in the mass of your dependencies:

- it splits the resources for documentation, testing, tutorial, etc

- it ensures very weak integration between various building blocks, forcing everyone to rebuild glue code again and again

- it makes discovering the proper solution to your problem harder as you must chose a combination of dependencies instead of one

- it makes contributing to the project harder, especially to junior profiles, and increases the price of on-boarding

- eventually, it leads to a culture that pushes the "libs versus frameworks" so far you never get a decent framework for anything. This is why there is no JS equivalent to RoR or Django.

There is, of course, a balance to reach. You don't want a lib that does everything.

Re: Ride down into JavaScript dependency hell

#62

Post by _bxg1 »

BiteCode_dev wrote: post #43
dwheeler wrote: post #13

Part of the issue is that JavaScript packages are often far far smaller than packages in other ecosystems. "small packages are extremely common in the JavaScript npm package system. For example, in npm, 47% of the packages have 0 or 1 functions, and the average npm package has 112 physical lines of code. In contrast, the average Python module in the PyPI repository has 2,232 physical lines of code." Source: "Vulnerab…

Which is weird. I would have expected have the leftpad scandal that the community would have found a balance. To me, culture is a feature, and despite using JS a lot since it has a monopoly on the browser, the community culture never clicked with me. I often feel like they are reinventing the wheel, not learning from other techs solving the same problems, under and over-engineering, regularly guilty of the XKCD 927 s…

There are several explanations:

1) JavaScript is a first language for many, many new developers, especially the ones not coming from a formal computer science program. This is because it's multi-paradigm, extremely hireable, runs on everything, runs on the web specifically, and other reasons. If you're only going to learn one language, it's a very good and versatile choice. So its community has a lot of inexperienced members.

2) Because it's such a focal point of the industry, it gets a lot of attention from major players as well and has been evolving rapidly in terms of core features. These features usually get tested out first in the form of packages.

3) Being a dynamic language it's very easy to extend and hack. The prototype system even allows you to modify the behavior of its core primitives.

4) The more experienced devs who use it are often dissatisfied with the qualities of the base language, and so take advantage of its flexibility to try and extend it into what they think a good language is (see TypeScript, Ramda, RxJS, etc).

The response to the leftpad scandal was for NPM to disallow packages being unpublished once they've been out for 72 hours, which really should've always been the case.

Re: Ride down into JavaScript dependency hell

#64

Post by inakarmacoma »

dep_b wrote: post #36
dwheeler wrote: post #13

Part of the issue is that JavaScript packages are often far far smaller than packages in other ecosystems. "small packages are extremely common in the JavaScript npm package system. For example, in npm, 47% of the packages have 0 or 1 functions, and the average npm package has 112 physical lines of code. In contrast, the average Python module in the PyPI repository has 2,232 physical lines of code." Source: "Vulnerab…

iOS dependencies in Cocoapods are including less than 1 other packages on average (1). The most I see are Alamofire (a networking package people keep using for reasons that I do not fully understand) and Starscream (a Websocket implementation, that's pretty hard to write from scratch to be honest). Having unnecessary dependencies is pretty much frowned upon. One of the reasons is that the iOS SDK is really complete b…

Google Apps Script! Heh...

Re: Ride down into JavaScript dependency hell

#65

Post by oefrha »

matheusmoreira wrote: post #59
mschuetz wrote: post #56

Earlier quoted context omitted.

Every separate package adds overhead and another maintainer that you've got to put your trust in. I'd rather have few packages of well trusted maintainers instead of a thousand packages from god knows who. Personally, I've made it a habit to not add anything that requires trivial one liner garbage packages, which amounts to not installing any dependency that uses anything from Jon Schlinkert (so no Webpack). Unfortun…

> I'd rather have few packages of well trusted maintainers instead of a thousand packages from god knows who. Then the problem is the fact anybody can submit a package, not small packages. Linux distributions solve this problem by having dedicated maintainers. Users of a distribution trust its maintainers when they use it. Software developers want the complete opposite: language-specific packages, instant and unrestr…

> What if Node's maintainers decided to distribute a curated set of packages alongside Node itself?

First, front-end has nothing to do with Node. Secondly, let’s say there’s some sort of front-end foundation who decides to provide a curated set of packages. A natural candidate would be create-react-app, which pulls over a thousand dependencies. That’s just one package you want to include. How the hell do you even start to curate such a beast?

Re: Ride down into JavaScript dependency hell

#66

Post by tln »

BiteCode_dev wrote: post #61
matheusmoreira wrote: post #48

Earlier quoted context omitted.

Why are small packages bad? Independent functions should be versioned and distributed independently. Otherwise we get several utility packages which are nothing but collections of independent functions. Widely used packages should form the basis of a standard library that is distributed with the language itself.

Small packages means more packages, and more packages compounds the dependency graph which: - makes dependency resolution harder - means more points of failure - makes auditing difficult - makes install time longer - leads to harder maintenance and upgrade This also causes heterogeneity in the mass of your dependencies: - it splits the resources for documentation, testing, tutorial, etc - it ensures very weak integra…

The package manager also has evolved to handle lots of dependencies.

In the article 19000 packages are installed in 40 seconds... My Django project with 100 times fewer dependencies takes longer in CI

Npm comes with an audit tool

I have never seen dependency resolution fail (since packages can have private copies), unlike pypi or rubygems.

So some of the downsides to a large dependency tree are mitigated. I'll add one more downside:

- more chances for shenanigans like left-pad to cause issues.

Don't get me wrong, I think 19000 dependencies is fucking nuts. NINETEEN THOUSAND.

Re: Ride down into JavaScript dependency hell

#67

Post by wffurr »

dgellow wrote: post #11
awinter-py wrote: post #3

express (common web framework) seems to have only 51 deps in lockfile jekyll has 13 I don't disagree that lots of deps = supply chain risk But there have always been variations between projects & kinds of projects re how many deps they pull in. Try following someone's ipython data science tutorial, it's requirements.txt for days I think lack of a standard lib early on plus hipster functional culture made the '10s JS…

A python equivalent to "express" would be Flask https://palletsprojects.com/p/flask/ I see 4 dependencies in the setup.py, and they are all from the same team of maintainers (i.e the Pallet team): - Werkzeug - Jinja2 - itsdangerous - click https://github.com/pallets/flask/blob/master/setup.py https://github.com/pallets/flask/network/dependencies

The Python standard library is a lot more capable than the JS equivalents (Node or browser API). That eliminates the need for a lot of dependencies.

There's been a TC39 proposal for a JS stdlib for about 2 years now: https://github.com/tc39/proposal-javascript-standard-library

Re: Ride down into JavaScript dependency hell

#68

Post by overgard »

Sadly the alternative isn't much better: the old school model of not having packages often leads to libraries never getting updated at all. In C++ often adding a library is such a chore once you get through all the linking issues, as you battle the 4000 conflicting compiler switches, and compiler versions, that once it works you never want to change it again.

I think maybe the best solution we have so far are robust "batteries included" standard libraries like python, where even if they're huge theyre at least curated and maintained by a central group.

Re: Ride down into JavaScript dependency hell

#69

Post by matheusmoreira »

tln wrote: post #66
BiteCode_dev wrote: post #61

Earlier quoted context omitted.

Small packages means more packages, and more packages compounds the dependency graph which: - makes dependency resolution harder - means more points of failure - makes auditing difficult - makes install time longer - leads to harder maintenance and upgrade This also causes heterogeneity in the mass of your dependencies: - it splits the resources for documentation, testing, tutorial, etc - it ensures very weak integra…

The package manager also has evolved to handle lots of dependencies. In the article 19000 packages are installed in 40 seconds... My Django project with 100 times fewer dependencies takes longer in CI Npm comes with an audit tool I have never seen dependency resolution fail (since packages can have private copies), unlike pypi or rubygems. So some of the downsides to a large dependency tree are mitigated. I'll add on…

> 19000 dependencies

Doesn't this happen because of package manager duplication? I think npm lets packages have their own copies of their dependencies. Since we have a lots of small, widely used packages, they get duplicated at numerous points in the dependency graph. The number of files and installation size explodes.

Re: Ride down into JavaScript dependency hell

#70

Post by IggleSniggle »

dragonwriter wrote: post #53
matheusmoreira wrote: post #48

Earlier quoted context omitted.

Why are small packages bad? Independent functions should be versioned and distributed independently. Otherwise we get several utility packages which are nothing but collections of independent functions. Widely used packages should form the basis of a standard library that is distributed with the language itself.

> Why are small packages bad? Small, separately maintained packages are bad in a an ecosystem where a final system can incorporate only one version of any given package because it increases the number of opportunities for version conflicts. Now, it's true that there are some other concerns which weigh in favor of packages being at the minimum useful size, but those necessarily also weigh in favor of a package managem…

I was under the impression that in the npm ecosystem, packages are reused if the version dependencies are compatible, and split if they are not.
Post reply on HN