Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We check our node_modules folder into source control

jackfranklin.co.uk

Post reply on HN
Print view View original

141–150 of 241 posts

Re: We check our node_modules folder into source control

#141

Post by skytreader »

> Better awareness of the code you're shipping.

Then somewhere down the line...

> One of our dependencies that we check in is TypeScript, and every time we update that, the git diff is huge and frankly not worth looking at (beyond the CHANGELOG)

Which I assume is the official attitude towards any dependency of the same magnitude. How are you more aware of the code you are shipping? Okay you managed to give yourself a visual on how much LOC your dependencies are but is that a relevant awareness? Do I not get the same thing with a `du -h node_modules`, with a matching pretty GUI on top?

The one thing I haven't seen addressed so far is: doesn't this make them susceptible to poisoned dependencies? Say they have a dependency to a large well-known library, what's stopping a malicious contributor from adding an HTTP call, thinly disguised to prevent grep, to some server in MiddleOfFucking, Nowhere? Even if they manage to flag this from blackbox testing, they now have a problem that only they have.

I can try to answer my own question: they're Google, they can afford a team scanning for vulnerabilities like this, a team dedicated to analyzing codebases that are found to be compromised, a legal and PR team to handle the fallout if this kind of vulnerability makes it to the public.

In short: horrible advice to follow if you are not Google.

Re: We check our node_modules folder into source control

#142

Post by Chyzwar »

This is idiotic and could only be written by someone that have no idea about node package managements.

1. Most of CI systems have better options for caching node modules[1][2]. When you check node_modules, you add fixed cost(increasing) for every commit. When you use CI caching you add fixed cost(static) to only small number of builds.

2. Once you start upgrading packages repo size will continue to grow. At some point you will be forced to git filter out node_modules. You will lose the ability to run locally older commits.

3. You will need to pin version of npm/yarn because structure of node_modules depends on hoisting algorithm. Every upgrade of node will be extra painful because you also potentially need to upgrade all yours packages.

4. Platform dependent modules like fsevents, node-sass can be broken if you use different OS. You will be forced to only support a single platform(linux).

5. Impossible to resolve node_modules conflicts. Modern package manager have git conflict resolution build in. If two people update the same module to the same version they can still create a merge conflict when node_modules are checked.

6. Currently, you have plenty of good options that can achieve the same with smaller effort. You can use yarn2 with node_modules linker and local cache. This would create .yarn folder in the repo that have all modules as zip files. During install it would use these to hydrate node_modules. Alternatively, You can use pnp and have zero install but with proper support[3]

7. You lose automatic audit and dependency management. Current best practice is to use something like dependabot or renovate bot. Once you commit your package, you will no longer be able to use this effectively.

8. Most people commenting on left-pad are maybe not aware, but today npm is immutable, and you simply cannot unpublish public package[4]. Because npm.com is such vital infrastructure, it unlikely that it would ever stop working.

[1] https://docs.github.com/en/actions/advanced-guides/caching-d... [2] https://circleci.com/docs/2.0/caching/ [3] https://yarnpkg.com/features/pnp [4] https://docs.npmjs.com/policies/unpublish

Re: We check our node_modules folder into source control

#143

Post by jayflux »

jdlshore wrote: post #112

Disappointed to see so many knee-jerk reactions to this. Vendoring dependencies is a simple way to ensure consistent build inputs, and has the bonus effect of decreasing build times. To respond the two major criticisms: 1) “It takes a lot of space” Don’t be so sure. Text diffs and compresses well. I have a 9-year old Node repo that I’ve been vendoring from the beginning and it’s only grown 200MB over that time. (Gran…

> Vendoring dependencies is a simple way to ensure consistent build inputs

If consistent build inputs is your concern may I ask why using lock files wasn’t enough? That’s a problem they were designed to solve.

Re: We check our node_modules folder into source control

#144

Post by fctorial »

jdlshore wrote: post #112

Disappointed to see so many knee-jerk reactions to this. Vendoring dependencies is a simple way to ensure consistent build inputs, and has the bonus effect of decreasing build times. To respond the two major criticisms: 1) “It takes a lot of space” Don’t be so sure. Text diffs and compresses well. I have a 9-year old Node repo that I’ve been vendoring from the beginning and it’s only grown 200MB over that time. (Gran…

> Vendoring dependencies is a simple way to ensure consistent build inputs

It wouldn't be necessary if the dependency tree was a pure function of package manifest.

https://developer.okta.com/blog/2019/12/16/semantic-versioni...

Re: We check our node_modules folder into source control

#145

Post by Goz3rr »

caymanjim wrote: post #125
mohanmcgeek wrote: post #55

The listed reasons are insufficient and we could achieve many of these by just pinning our dependencies versions. If we did this, our git repo after a few commits will tend towards a gazillion GBs. Costs outweigh the benefits, if at all there are any. Horrible advice. Don't break the industry practice and check-in your node_modules

You're drastically overestimating the storage requirements. Time is far, far more valuable than storage savings. The point about CI builds running faster is enough to sell the idea all by itself. Oddly not mentioned in the article is that it allows you to build when npmjs.org is down or unreachable, which happens often enough to be frustrating, and if it happens when you're trying to deal with an emergency, it's down…

Git is good at storing text diffs, any binary file in your node_modules (images, natives etc) are permanently stored in your repo, including old versions or deleted files. I've seen times where this had meaningful impact on both disk use and the speed of Git itself

Re: We check our node_modules folder into source control

#147

Post by avgcorrection »

klodolph wrote: post #84
mohanmcgeek wrote: post #55

The listed reasons are insufficient and we could achieve many of these by just pinning our dependencies versions. If we did this, our git repo after a few commits will tend towards a gazillion GBs. Costs outweigh the benefits, if at all there are any. Horrible advice. Don't break the industry practice and check-in your node_modules

I’d say that the main link in that reasoning is that “Git can’t handle this without making the repo a gazillion GBs” which, can of course, can be solved if you weren’t using Git in the first place. Certain other SCMs, like Perforce, allow you to trim history and don’t require you to clone the whole history in the first place.

The author thinks that Git is so obiquitous that they just use “Git” as a stand-in for “VCS” throughout after the first paragraph. So the author definitely thought that Git would be suitable for this.

Re: We check our node_modules folder into source control

#148

Post by maccard »

sktrdie wrote: post #133
maccard wrote: post #128

People here complaining about the size of the node modules folder - this is one of the reasons people use other solutions like perforce. Checking a 2GB folder into p4 is an absolute no brainer, and for all the flexibility people talk about with git, it's inability to handle this is pretty damning after so many years.

Can you expand on this? Why is the size smaller on p4 compared to git?

As the other commenter replied p4 doesn't download the entire history locally, so storage is only a concern on the server. Depending on what you're storing in p4, you can use p4 archive and/or p4 obliterate to manage the actual storage used on the server. E.g. you might use p4 obliterate to only keep the latest version of your node modules folder in p4 if you're trying to optimize for CI performance, or if you're using p4 for build artifacts, you might use p4 archive and store the older revsions on slower bulk storage.

Re: We check our node_modules folder into source control

#150

Post by davedx »

DaiPlusPlus wrote: post #140
dijit wrote: post #123

Earlier quoted context omitted.

The main reason to use perforce in the first place is because of assets. Locking of files too, but that’s only necessary on non-diffable assets.

I've only ever used Perforce through a command-line interface. What's the UX like when using Perforce for blobs/assets? Does everyone have to use the command-line or is there a GUI experience (with fast-rendering thumbnails?)? Forgive the questions, I'm just curious about the minutia and peculiarities of the gaming-biz because I've never worked in the field.

I used Perforce 10-15 years ago, it had a pretty solid GUI application. I never used the CLI.
Post reply on HN