Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We check our node_modules folder into source control

jackfranklin.co.uk

Post reply on HN
Print view View original

181–190 of 241 posts

Re: We check our node_modules folder into source control

#181

Post by NaN1352 »

I don’t check in node_modules but do a backup once in a while.

With frontend nowadays it’s sad but after six months it’s highly unlikely that my project will compile, not to mention the tooling like Vue, Vite, etc that has breaking changes.

I mean it’s scary. You write a program and it WONT run if you just give it enough time. Locking versions is not really a solution since often times the tooling itself, and IDE extensions require newer versions of packages.

Maybe you wanted to fix a typo a year down the line but oh no, now you need to figure out why Vite won’t start, why eslint dropped support for xyz, spend hpurs figuring out what you need to change in your configs, etc.

Re: We check our node_modules folder into source control

#182

Post by onion2k »

NaN1352 wrote: post #181

I don’t check in node_modules but do a backup once in a while. With frontend nowadays it’s sad but after six months it’s highly unlikely that my project will compile, not to mention the tooling like Vue, Vite, etc that has breaking changes. I mean it’s scary. You write a program and it WONT run if you just give it enough time. Locking versions is not really a solution since often times the tooling itself, and IDE ext…

I don't see how this would work in practise. You could use module-alias[1] to actually switch to the backup copy during dev or a local build, but that will only work when the backup is on the same machine you're building on, and then everyone on the team will need to have the same backup (or use a network drive for it I guess). If you don't check in your backup then nothing will get through CI or make it to production. Why not just check in node_modules and let git handle the 'backup' process?

[1] https://www.npmjs.com/package/module-alias

Re: We check our node_modules folder into source control

#183

Post by gausswho »

eyelidlessness wrote: post #46
gausswho wrote: post #36

Earlier quoted context omitted.

Anyone recommend a build check to block PR's from including fully rebuilt lockfiles? Other than a high LOC, it's too easy for this to slip through.

In my experience this is probably not viable. A single, valid update to a direct dependency can bring dozens or even hundreds of updated sub-dependencies. And an audit fix will often update just the lockfile. Maybe I lack imagination, but I can’t think of a workable heuristic for determining how the lockfile was changed. I think, instead, it would be good to move in a different direction for sub-dependencies generall…

What you're describing is more in spirit with the intent of a lockfile and worth exploring by package managers. But I do think a heuristic could be conceived for the lockfile-was-rebuilt situation: If a single top-level dependency declared with ^ or ~ was present in the lockfile when the existing resolution was still valid.

Ideally the onus is on the package manager to provide metadata in the lockfile for the strategies it took when generating.

Re: We check our node_modules folder into source control

#185

Post by rpadovani »

gravypod wrote: post #179
somehnacct3757 wrote: post #169

Upsides from storing node_modules in repo are outweighed by the downsides. Unless of course you're Google-scale and can afford to contribute filesize fixes upstream, write fancy tooling to enforce commit-time workarounds, etc. Nobody working finger-to-feature has time for this. For your average npm shop which doesn't have infinite internet oil money, here is why the article recommendations won't work for you. Your CI…

> Your CI will pay the time penalty during git clone instead of npm ci. Things like GitLab's CI runners will do a single clone then do `fetch`, `checkout`, and `clean` to checkout your repo. Git repo size isn't a huge bottleneck in CI performance.

Only if you have long-living runners: if you use a dynamic fleet to save money, then almost every time you clone the repo. However, this is why you can do sparse checkouts and limit the git depth.

Re: We check our node_modules folder into source control

#186

Post by onei »

jayflux wrote: post #143
jdlshore wrote: post #112

Disappointed to see so many knee-jerk reactions to this. Vendoring dependencies is a simple way to ensure consistent build inputs, and has the bonus effect of decreasing build times. To respond the two major criticisms: 1) “It takes a lot of space” Don’t be so sure. Text diffs and compresses well. I have a 9-year old Node repo that I’ve been vendoring from the beginning and it’s only grown 200MB over that time. (Gran…

> Vendoring dependencies is a simple way to ensure consistent build inputs If consistent build inputs is your concern may I ask why using lock files wasn’t enough? That’s a problem they were designed to solve.

I use vendoring in Go because my team's builds happen within a huge, complicated corporate network that has been known to break arbitrarily in new and interesting ways (or rather when something gets changed unexpectedly and then it takes days/weeks to navigate outsourced IT and change it back). Vendoring deps doesn't save me all the time, but I've generally found it helps. Plus builds are a bit quicker because I can download everything in one go (via git clone) rather than pulling everything in at build time. It also helps when the linter decides it wants all the dependencies downloaded before doing anything and then we find it has a relatively short timeout when the network gains a lot of latency without notice.

On reflection, it seems more like I'm papering over network issues. Perks of working in an enterprise company I guess.

Re: We check our node_modules folder into source control

#188

Post by oftenwrong »

AtlasBarfed wrote: post #48

Maven, Gradle and the like always download dependencies as part of the build, and if a server is down (or flaky like jitpack) good lord is it annoying. What I don't understand is why dependency download isn't a separate task you do before compilation. FIRST you download all the dependencies to stabilize that, then you build the code. The reproducibility is the big one. IT drives me nuts that the dependencies are hidd…

For Maven, you can use this plugin to move downloading dependencies into a separate step:

https://github.com/qaware/go-offline-maven-plugin

It's not perfect, but it can be useful.

Re: We check our node_modules folder into source control

#189

Post by hasmolo »

wruza wrote: post #110
ilumanty wrote: post #13

Yarn offers "Plug'n'play" mode since v2, which basically promotes what the author says. It takes the idea further: dependencies are stored as zip archives instead of thousands of small files, which reduces the "git noise" and actually makes this viable as a performant workflow. https://yarnpkg.com/features/pnp

Yarn pnp was in a poor state tooling-wise last time I checked it a year or so ago. Too many tools depend on ./node_modules/ (or actually an entire module.paths thing) to exist on a filesystem. Was it resolved somehow?

by and large it has been solved with editor extensions and pnp plugins for various tools, i've used yarn 2/3 at enterprise scale for a few years now and am happy with it

Re: We check our node_modules folder into source control

#190

Post by hasmolo »

ilumanty wrote: post #13

Yarn offers "Plug'n'play" mode since v2, which basically promotes what the author says. It takes the idea further: dependencies are stored as zip archives instead of thousands of small files, which reduces the "git noise" and actually makes this viable as a performant workflow. https://yarnpkg.com/features/pnp

and yarn 3 will even allow for node_module linker usage instead of cjs require hook overload
Post reply on HN