Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We check our node_modules folder into source control

jackfranklin.co.uk

Post reply on HN
Print view View original

111–120 of 241 posts

Re: We check our node_modules folder into source control

#111

Post by cookiengineer »

StopHammoTime wrote: post #109

I don't agree that it should be checked into source control. I do believe it should be cached somewhere. How much bandwidth of popular sites are used by redundant actions. A single request for a 2GB archive is much better than a 1,000,000 small requests that are all a few KBs or MBs in size.

Nodejs has a node-gyp problem. Every node_module that somewhere down the dependency tree requires a "native" module will require recompilation on the target machine (or in worst case: the user machine).

I really would have hoped that the NaN module related problems will be fixed over time, but here we are in 2021 and nothing's been fixed.

As long as npm doesn't use binaries and headers, those things will stay broken. The thing that they argue with to use "always source" is kinda ridiculous when considering that probably the most of all npm packages are using webpack or another bundler before pushing their own package to npm - because npm itself has become impossible to use as a package manager alone.

I mean, a couple MB of libraries with the wrong dependencies can lead to multiple phantomjs installation, which is an inactive, deprecated, and unsecure project for years already... just because of some unit tests that have no place in a production npm package.

My hopes are that more sane developers come together, switch to ESM and implement better policies for evaluating their dependencies (e.g. blocking sources from people that have more than 1000 npm packages and brag about it).

Pikapkg was a great idea in my opinion, and I was using it before they moved the project to building astro as a platform :-/

Re: We check our node_modules folder into source control

#112

Post by jdlshore »

Disappointed to see so many knee-jerk reactions to this. Vendoring dependencies is a simple way to ensure consistent build inputs, and has the bonus effect of decreasing build times.

To respond the two major criticisms:

1) “It takes a lot of space”

Don’t be so sure. Text diffs and compresses well. I have a 9-year old Node repo that I’ve been vendoring from the beginning and it’s only grown 200MB over that time. (Granted, I’m fairly restrained in my use of dependencies. But I do update them regularly.)

But even if it does take a lot of space… so what? If your dependencies are genuinely so huge that this is a problem, then vendoring may not be right for you. But you could also use one of the many techniques for managing the size of your repo. Or just acknowledge that practices are contextual, and there’s no such thing as “best practice”—just a bunch of trade-offs.

2) “It doesn’t work well with platform-specific code”

This can cause some pain if you’re in a multi-platform environment. The way I deal with it (in Node) is by installing modules with --ignore-scripts, comitting the files, running “npm rebuild”, and then adding whatever shows up to .gitignore. I have a little shell script that makes this easier.

This is only an issue for modules that have a platform-specific build, which I try to avoid anyway. But when it comes up, it can be a pain in the butt. I find its pain to be less frequent and more predictable than the pain that comes from not vendoring modules, though, so I put up with it.

Bonus) “It’s not best practice”

Sez who? Dogma is for juniors. “Best practices” are all situational, and the only way to know if a practice is a good idea is to examine its tradeoffs in the context of your situation.

Re: We check our node_modules folder into source control

#113

Post by DecoPerson »

Jarred wrote: post #54
ilumanty wrote: post #13

Yarn offers "Plug'n'play" mode since v2, which basically promotes what the author says. It takes the idea further: dependencies are stored as zip archives instead of thousands of small files, which reduces the "git noise" and actually makes this viable as a performant workflow. https://yarnpkg.com/features/pnp

Unfortunately, reading files from .zip can be considerably slower than the filesystem. On an Ubuntu x86_64 machine with an SSD, require("react") with Yarn PnP ends up 4x slower than when installed from npm (Node 14.17.6).

I might be misremembering but at runtime, the Node process loads .pnp.js, which is a kind of monolithic “compiled” modules file containing all the modules installed. No reading of the .zip files occurs at runtime. (Again, I might be misremembering or have misunderstood. Please confirm/deny this if you know.)

Re: We check our node_modules folder into source control

#114

Post by clintonb »

lf-non wrote: post #96
emteycz wrote: post #91

Earlier quoted context omitted.

You can have multiple commits in a PR

Yes, but a lot of people prefer to have every commit in master in a buildable state. Otherwise things like git bisect become a lot more tedious.

Some teams force all pull requests to be squashed to a single commit.

Re: We check our node_modules folder into source control

#115

Post by fivea »

locallost wrote: post #95
mohanmcgeek wrote: post #55

The listed reasons are insufficient and we could achieve many of these by just pinning our dependencies versions. If we did this, our git repo after a few commits will tend towards a gazillion GBs. Costs outweigh the benefits, if at all there are any. Horrible advice. Don't break the industry practice and check-in your node_modules

Whether he's right or wrong can be debated, but reducing the argument to "follow industry practice" is a perfect example of cargo culting. Industry practice needs to be based on something today, not something from ten years ago that might or might not be valid anymore. He offers a long list of arguments and even though I don't check in node_modules, some of the arguments are compelling - e.g. we already want reproduc…

> He offers a long list of arguments and even though I don't check in node_modules, some of the arguments are compelling (...)

Are they, really?

The less debatable point is arguing that making CICD pipelines slightly faster, but this feels like an appeal to microoptimization. Any free tier CICD system out there let's you do a single npm install and move these dependencies as far as you'd like into the pipeline as artifacts. Is a git checkout really faster that a npm install?

For example, is adding a npm dependency really invisible if you already track package.json and even package-lock.json? Those files show up in diffs, and it's hard to miss them.

Also, if the goal is to get replicated builds, isn't this handled by pinning versions and tracking package-lock?

The left_pad example is particularly ridiculous as I highly doubt that a company like Google, like any company that cares about auditing and vending dependencies, does not run its own npm proxy with cherry-picked packages.

Re: We check our node_modules folder into source control

#116

Post by sofixa »

klodolph wrote: post #84
mohanmcgeek wrote: post #55

The listed reasons are insufficient and we could achieve many of these by just pinning our dependencies versions. If we did this, our git repo after a few commits will tend towards a gazillion GBs. Costs outweigh the benefits, if at all there are any. Horrible advice. Don't break the industry practice and check-in your node_modules

I’d say that the main link in that reasoning is that “Git can’t handle this without making the repo a gazillion GBs” which, can of course, can be solved if you weren’t using Git in the first place. Certain other SCMs, like Perforce, allow you to trim history and don’t require you to clone the whole history in the first place.

With git you can specify the clone depth and only get the latest X versions. And there are ways to trim history with external plugins ( git filter repo).

Re: We check our node_modules folder into source control

#117

Post by andreineculau »

unqueued wrote: post #64
mohanmcgeek wrote: post #55

The listed reasons are insufficient and we could achieve many of these by just pinning our dependencies versions. If we did this, our git repo after a few commits will tend towards a gazillion GBs. Costs outweigh the benefits, if at all there are any. Horrible advice. Don't break the industry practice and check-in your node_modules

I think that having rapid access to node_modules can be very helpful sometimes. The solution I came up with was this: https://github.com/unqueued/git-cache-tag It copies all untracked stuff (including node_modules) into a leaf tag. It is fairly easy to manage them, or find the latest one. And because they are leaves, they can be pruned and completely garbage collected when they aren't useful anymore. I have been burn…

Funny but I've also done this. Thanks for sharing!

Depending on the context, if you don't want this in git history, and want to handle git submodules, there's also git-archive-all https://github.com/roehling/git-archive-all (if you like shell scripts, it is using bats for testing - it was the first time I heard of it)

Re: We check our node_modules folder into source control

#118

Post by po1nt »

None of the benefits mentioned in article won't benefit well maintained codebase. If you're forced into such solution you're doing something wrong.

Just use locking, cache and reduce your dependencies to bare minimum you actually need.

In JS you can often find yourself pulling a massive dependency tree just for simple usecase that would be replicable in your codebase under 3 hours.

Re: We check our node_modules folder into source control

#119

Post by awill88 »

20after4 wrote: post #105
awill88 wrote: post #72

Can’t you like, cache the node_modules folder on CI builds? I dunno, seems gross, unless you’re on a project with minimal deps or very meticulous about which deps you leverage. I am just one of those people who are constantly trying to upgrade dependencies anyways (cautiously of course) as to avoid vulnerabilities. That said, I see the point, it’s interesting..

Upgrading could introduce vulnerabilities just as easily as fixing them.

it’s possible, kind of a moot point as doing nothing induces a similar level of risk in my experience. Don’t touch your code for two weeks, I guarantee you that ‘npm audit’ will complain about some new issue.

Re: We check our node_modules folder into source control

#120

Post by bennyp101 »

mohanmcgeek wrote: post #55

The listed reasons are insufficient and we could achieve many of these by just pinning our dependencies versions. If we did this, our git repo after a few commits will tend towards a gazillion GBs. Costs outweigh the benefits, if at all there are any. Horrible advice. Don't break the industry practice and check-in your node_modules

I've just checked a largish repo, and the node_modules folder is just under 500mb - I could check that in, and be done with it. Updates aren't constant, so it would be every now and then.

That's really not a lot for knowing the code that is being used in your codebase hasn't changed, and the bonus of having everything available should the registry go down, or something.

So I don't think it's 'horrible advice' - it's do what suits your needs best. Some people want to have everything they need to build their application in their control, on the off chance everything hits the fan.

(Also, this: "Don't break the industry practice and check-in your node_modules" - does not necessarily mean it is the best way, it just happens to be the advice from the start.

Post reply on HN