Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We check our node_modules folder into source control

jackfranklin.co.uk

Post reply on HN
Print view View original

151–160 of 241 posts

Re: We check our node_modules folder into source control

#151

Post by dtgriscom »

jayflux wrote: post #143
jdlshore wrote: post #112

Disappointed to see so many knee-jerk reactions to this. Vendoring dependencies is a simple way to ensure consistent build inputs, and has the bonus effect of decreasing build times. To respond the two major criticisms: 1) “It takes a lot of space” Don’t be so sure. Text diffs and compresses well. I have a 9-year old Node repo that I’ve been vendoring from the beginning and it’s only grown 200MB over that time. (Gran…

> Vendoring dependencies is a simple way to ensure consistent build inputs If consistent build inputs is your concern may I ask why using lock files wasn’t enough? That’s a problem they were designed to solve.

npm has long had a problem respecting lock files. The concept is easy: have a fixed lock file, get a reproducible build. But no: npm will change your lock file (I believe it's framed as "optimizing") without notice.

(Perhaps they've solved this in the last couple of years. I've been staying away from that ecosystem... too much growing in it...)

Re: We check our node_modules folder into source control

#152

Post by tentacleuno »

> Once you check your node_modules in, there's no need to run an install step before you can get up and running on the codebase. This isn't just useful for developers locally, but a big boost for any bots you might have running on a Continuous Integration platform (e.g. CircleCI, GitHub Actions, and so on). That's now a step that the bots can miss out entirely. I've seen projects easily need at least 1-2 minutes to run a complete npm install from scratch [...]

Couldn't this issue be solved by using caching? If I remember correctly, Travis CI has the option to cache certain folders between builds, meaning an npm install doesn't have to start from scratch, and can just incrementally update the cached node_modules folder (any changes are then copied to the next CI build).

Re: We check our node_modules folder into source control

#153

Post by kevincox »

dtgriscom wrote: post #151
jayflux wrote: post #143

Earlier quoted context omitted.

> Vendoring dependencies is a simple way to ensure consistent build inputs If consistent build inputs is your concern may I ask why using lock files wasn’t enough? That’s a problem they were designed to solve.

npm has long had a problem respecting lock files. The concept is easy: have a fixed lock file, get a reproducible build. But no: npm will change your lock file (I believe it's framed as "optimizing") without notice. (Perhaps they've solved this in the last couple of years. I've been staying away from that ecosystem... too much growing in it...)

I think the trick is that you should use `npm ci` instead of `npm install` in most cases.

Re: We check our node_modules folder into source control

#154

Post by tentacleuno »

3np wrote: post #56
DaiPlusPlus wrote: post #45

Earlier quoted context omitted.

Now we just need to convince all those startups burning cycles trying to get Kubernetes working instead of working on their actual product...

TFW you realize k8s isn't even used that widely inside Google.

What's your source on that? Would be interested to hear more.

Aside: They seemed to like their Linux containers in 2014[0]. No idea if they're still using the same technique, or have maybe moved onto Kubernetes?

[0]: https://www.theregister.com/2014/05/23/google_containerizati...

Re: We check our node_modules folder into source control

#155

Post by fivea »

raverbashing wrote: post #127
fivea wrote: post #115

Earlier quoted context omitted.

> He offers a long list of arguments and even though I don't check in node_modules, some of the arguments are compelling (...) Are they, really? The less debatable point is arguing that making CICD pipelines slightly faster, but this feels like an appeal to microoptimization. Any free tier CICD system out there let's you do a single npm install and move these dependencies as far as you'd like into the pipeline as art…

Here's a better idea, can't you have an npm cache/clone that keeps all the artifacts you use in your code? So you pull from it, it pulls from npm and caches?

> Here's a better idea, can't you have an npm cache/clone that keeps all the artifacts you use in your code? So you pull from it, it pulls from npm and caches?

Not only is that possible, that's also expected to be mandatory in any company that is required to monitor ad control dependencies. I know for a fact that some FANGs do manage and enforce the use of internal npm repositories, mainly because of infosec audits, and I doubt Google is not one of them.

Re: We check our node_modules folder into source control

#156

Post by tentacleuno »

Chyzwar wrote: post #142

This is idiotic and could only be written by someone that have no idea about node package managements. 1. Most of CI systems have better options for caching node modules[1][2]. When you check node_modules, you add fixed cost(increasing) for every commit. When you use CI caching you add fixed cost(static) to only small number of builds. 2. Once you start upgrading packages repo size will continue to grow. At some poin…

> This is idiotic and could only be written by someone that have no idea about node package managements.

Some of your points are valid, but the first sentence seems really hand-wavy and rude.

> 1. Most of CI systems have better options for caching node modules[1][2]. When you check node_modules, you add fixed cost(increasing) for every commit. When you use CI caching you add fixed cost(static) to only small number of builds.

I was wondering about this too[0], mostly because it's the setup I tended to gravitate towards for my projects. Gitlab CI and Travis can do this AFAIK, although I'm not sure how long the cache folders would be kept (especially on the Free tier).

[0]: https://news.ycombinator.com/item?id=29529154

Re: We check our node_modules folder into source control

#157

Post by TomSwirly »

jdlshore wrote: post #112

Disappointed to see so many knee-jerk reactions to this. Vendoring dependencies is a simple way to ensure consistent build inputs, and has the bonus effect of decreasing build times. To respond the two major criticisms: 1) “It takes a lot of space” Don’t be so sure. Text diffs and compresses well. I have a 9-year old Node repo that I’ve been vendoring from the beginning and it’s only grown 200MB over that time. (Gran…

Came here to say this but you said it better.

I don't check in my dependencies in my current project because I don't need to; but in earlier projects, I or we did, for various good reasons; and it worked perfectly well, and was extremely convenient for new developers.

Re: We check our node_modules folder into source control

#158

Post by Tajnymag »

tiotempestade wrote: post #149

I know there’s no other way, but I wish JS could just die!

Then you might be happy to hear that a pure JavasScript has been dead for quite some time.

Nowadays, most new large codebases run on TypeScript or some recent version of EcmaScript.

The rant the article is based on is not really about language itself but the tooling and ecosystem around it.

Re: We check our node_modules folder into source control

#159

Post by paulbjensen »

I understand what he's getting at, but if you are trying to have an exact copy of code in order to guarantee an exact behaviour, then you would want to extend that to the operating system used to run the software.

One example was having code where the test suite ran fine on my local MacBook, but would fail on CI (Linux). It turned out that on Linux finding files by name is case sensitive, whereas on Mac OS it isn't, and a require statement in Node.js was referencing a file path with casing that was different to the file name's spelling.

Re: We check our node_modules folder into source control

#160

Post by HelloNurse »

I currently have a more concrete problem than speed with leaving node_modules out of version control: buggy libraries that, as downloaded, need small corrections. Doing the right thing and fixing libraries for everyone on npm requires significantly more effort and it could be impossible.
Post reply on HN