Live data from Hacker News

We check our node_modules folder into source control

jackfranklin.co.uk

171–180 of 241 posts

Re: We check our node_modules folder into source control

#171

Upsides from storing node_modules in repo are outweighed by the downsides. Unless of course you're Google-scale and can afford to contribute filesize fixes upstream, write fancy tooling to enforce commit-time workarounds, etc. Nobody working finger-to-feature has time for this. For your average npm shop which doesn't have infinite internet oil money, here is why the article recommendations won't work for you. Your CI…

Isn't --depth=1 all you need to avoid downloading every binary?

Re: We check our node_modules folder into source control

#172
post #70

This is basically a way of saying: to hell with those "package managers". It's a sentiment that I'm actually in agreement with. I've been coding mostly in Java for the past 22 years. Somewhere around 2010 Maven became the prevalent build tool quickly displacing the venerable Ant. With Ant we had builds that used checked in jar file dependencies. It was obvious what your builds consisted of and they were very fast onc…

It’s been a long while since I saw a POM file with unpinned dependencies… ca 12 years of Java experience. Had to deal with an Ant project recently that didn’t check in the JARs into Git, I thought my hair would become gray by the time I figure out how to pull the right dependency versions transitively to get the Ant build to work.

But I share the overall sentiment. What if Maven Central goes down one day? It’s just a web server like any other.

Re: We check our node_modules folder into source control

#173

Earlier quoted context omitted.

> Vendoring dependencies is a simple way to ensure consistent build inputs If consistent build inputs is your concern may I ask why using lock files wasn’t enough? That’s a problem they were designed to solve.

It says so in the post, in case another left-pad removal happens.

Not that I'm a fan but didn't npm resolve that problem right after? You can't yank entire packages anymore 24 hours after they're published

Re: We check our node_modules folder into source control

#174
post #54

Earlier quoted context omitted.

Unfortunately, reading files from .zip can be considerably slower than the filesystem. On an Ubuntu x86_64 machine with an SSD, require("react") with Yarn PnP ends up 4x slower than when installed from npm (Node 14.17.6).

I might be misremembering but at runtime, the Node process loads .pnp.js, which is a kind of monolithic “compiled” modules file containing all the modules installed. No reading of the .zip files occurs at runtime. (Again, I might be misremembering or have misunderstood. Please confirm/deny this if you know.)

The pnp.js file has a list of modules, not the modules themselves. The modules are still read from zip files at runtime.

Yarn doesn't do any kind of compilation itself - that would be done by build tools like webpack.

Re: We check our node_modules folder into source control

#175

Earlier quoted context omitted.

It says so in the post, in case another left-pad removal happens.

Not that I'm a fan but didn't npm resolve that problem right after? You can't yank entire packages anymore 24 hours after they're published

Yes, you are right. But there are people who trust npm (now and esp. in the future) and other free infrastructure and there are those who prefer to be a bit more self reliant after getting burned once.

Re: We check our node_modules folder into source control

#176

Earlier quoted context omitted.

Isn't that really a problem with npm and package maintainers? Node is really just a runtime.

When you install nodejs, you have npm installed. So in this case, there's not much you can do though. So it's not just npm issue, it's NodeJS issue, too.

Not necessarily. On Arch, npm isn't bundled with the node package.

Also, according to the official npm documentation, npm should be installed through nvm, thus having the ability to specify its version separately from node.

https://docs.npmjs.com/downloading-and-installing-node-js-an...

Re: We check our node_modules folder into source control

#177

Earlier quoted context omitted.

Can you expand on this? Why is the size smaller on p4 compared to git?

P4 does not store the entire history locally, so your only limit is the amount of space on the server. Git will pack objects after a while (initially they’re stored as-is, just compressed), but large files or enormous amounts of changes can make the repository grow to unwieldy amounts, and while git is able to perform “shallow” clones (clones which only store part of the history), not everything handles them well

Git also has "partial" clones, which can avoid some of the downsides of shallow clones. You can, for example, filter out all historical objects (--filter=blob:none). Historical objects are fetched lazily as needed, such as checking out an old revision. The same can be done with unreachable trees.

This article has excellent diagrams that depict the different types of clones:

https://github.blog/2020-12-21-get-up-to-speed-with-partial-...

Re: We check our node_modules folder into source control

#178

This is idiotic and could only be written by someone that have no idea about node package managements. 1. Most of CI systems have better options for caching node modules[1][2]. When you check node_modules, you add fixed cost(increasing) for every commit. When you use CI caching you add fixed cost(static) to only small number of builds. 2. Once you start upgrading packages repo size will continue to grow. At some poin…

"Because npm.com is such vital infrastructure, it unlikely that it would ever stop working."

Except that is has? It's a service, like everything else on the internet, it can and will go down. The choice here is whether you can carry on, or wait until it comes back up. THAT is the main benefit I see in this. (Not everybody wants to manage a local copy/proxy/internal NPM registry etc)

Re: We check our node_modules folder into source control

#179

Upsides from storing node_modules in repo are outweighed by the downsides. Unless of course you're Google-scale and can afford to contribute filesize fixes upstream, write fancy tooling to enforce commit-time workarounds, etc. Nobody working finger-to-feature has time for this. For your average npm shop which doesn't have infinite internet oil money, here is why the article recommendations won't work for you. Your CI…

> Your CI will pay the time penalty during git clone instead of npm ci.

Things like GitLab's CI runners will do a single clone then do `fetch`, `checkout`, and `clean` to checkout your repo. Git repo size isn't a huge bottleneck in CI performance.

Re: We check our node_modules folder into source control

#180

Yarn offers "Plug'n'play" mode since v2, which basically promotes what the author says. It takes the idea further: dependencies are stored as zip archives instead of thousands of small files, which reduces the "git noise" and actually makes this viable as a performant workflow. https://yarnpkg.com/features/pnp

Wanted to suggest the same thing. Used Yarn 2 PnP with zero installs in the past company and it worked great. Whenever I switched to projects not using it install seemed to take ages. Used that only on backend, YMMV.
Post reply on HN