Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We check our node_modules folder into source control

jackfranklin.co.uk

Post reply on HN
Print view View original

211–220 of 241 posts

Re: We check our node_modules folder into source control

#211

Post by cxr »

jayflux wrote: post #143
jdlshore wrote: post #112

Disappointed to see so many knee-jerk reactions to this. Vendoring dependencies is a simple way to ensure consistent build inputs, and has the bonus effect of decreasing build times. To respond the two major criticisms: 1) “It takes a lot of space” Don’t be so sure. Text diffs and compresses well. I have a 9-year old Node repo that I’ve been vendoring from the beginning and it’s only grown 200MB over that time. (Gran…

> Vendoring dependencies is a simple way to ensure consistent build inputs If consistent build inputs is your concern may I ask why using lock files wasn’t enough? That’s a problem they were designed to solve.

Onus confusion strikes again https://news.ycombinator.com/item?id=29276656>. The (mediocre) tooling for lockfiles isn't bedrock.

In a discussion about Skub, no one need to explain why the Skub-powered approach isn't good enough. It is the duty of anyone pushing Skub to explain exactly what makes Skub so special to the point that we need to have Skub in our lives.

Re: We check our node_modules folder into source control

#212

Post by ng12 »

NavinF wrote: post #134
ng12 wrote: post #87

> We've come up with a rule that helps us here: a change that updates node_modules may not touch any other code in the codebase What? What if the dependency upgrade requires code changes? Master is just broker until the second MR merges? This all sounds like terrible advice.

Dude he addressed that in the next few lines: > There are times where this doesn't work; updating TypeScript may require us to update some code to fix errors that the new version of TypeScript is now detecting. In that case we have the ability to override the rule. As with anything in software engineering, most "rules" are guidelines, and we're able to side-step them when required.

Ah, I glossed over that. But that just makes me more confused, not less. On a long lived project most changes to node_modules will require code changes... just not sure what the point is.

Re: We check our node_modules folder into source control

#213

Post by [deleted] »

abraxas wrote: post #70

This is basically a way of saying: to hell with those "package managers". It's a sentiment that I'm actually in agreement with. I've been coding mostly in Java for the past 22 years. Somewhere around 2010 Maven became the prevalent build tool quickly displacing the venerable Ant. With Ant we had builds that used checked in jar file dependencies. It was obvious what your builds consisted of and they were very fast onc…

[deleted]

Re: We check our node_modules folder into source control

#214

Post by somehnacct3757 »

ownagefool wrote: post #203
somehnacct3757 wrote: post #169

Upsides from storing node_modules in repo are outweighed by the downsides. Unless of course you're Google-scale and can afford to contribute filesize fixes upstream, write fancy tooling to enforce commit-time workarounds, etc. Nobody working finger-to-feature has time for this. For your average npm shop which doesn't have infinite internet oil money, here is why the article recommendations won't work for you. Your CI…

Counter point, and ignoring download size as your typical CI probably doesn't download the entire history, when are we supposed to pretend we've reviewed our dependencies? I'll admit I don't believe everyone always need to check every dep, but we're skating close nobody checking them ever.

My team guards up front when introducing a new dependency. You fill out a little template with security assessment as well as some other stuff, just to do a dirt simple build vs 'buy' analysis. left-pad for example would fail because the build time cost savings are not worth the ongoing maintenance cost. (In fact doing this assessment at all rarely makes sense for microlibs, by design.)

Once something's in package.json I don't believe anyone who says they can vouch for the security of that over time. We're all doing security theater with npm audit, dependabot, etc. Don't use npm at all if anyone's life depends on your code.

Re: We check our node_modules folder into source control

#215

Post by manicdee »

0des wrote: post #76
manicdee wrote: post #71

Earlier quoted context omitted.

I guarantee they are not aware of the ramifications of what they wrote. They live in the now, following the contemporary paradigm. I live in the five years future where the propeller head rock star programmer has moved to greener pastures (the ones where he doesn’t have to write project planning documents of any kind). They are different worlds.

Hey it's me, 10-15yrs in the future guy. Pass me a stack of punch cards and let me babble a little bit about legacy code. I found some dusty bourbon in rockstar guy's old desk. Don't let it get to you, I'll happily cash checks to work on whatever legacy spaghetti tech is in play. Hours are hours, dollars are dollars, and as long as I'm maintaining a happy ratio of those two, I don't mind what code I'm working on. I'l…

I wish I had the time to sit down for a bourbon and a chat with you but in order to earn my $26k salary I have to repeatedly unearth these ancient systems, divine how they were supposed to work, repair them to a state where they don't break as much anymore, and then move to the next emergency.

I'd love to join the smoking-jacket crowd but I have student debts to pay, and my other job as a janitor actually sweeping floors to get to.

The world isn't the same place it was when punched cards, smoking jackets and bourbon in the library were a thing.

Re: We check our node_modules folder into source control

#216

Post by manicdee »

winrid wrote: post #98
manicdee wrote: post #80

Earlier quoted context omitted.

Just another service that needs to be maintained but isn’t on the books as something that needs to be maintained leading to a wonderful day a couple of years in the future when a license cull of abandoned VMs means all your code stops building successfully on the same day.

I mean, yeah, gotta maintain it. If it breaks one day every 5 years but speeds up build times 2x every day until then, worth it.

My complaint isn't about build times, it's about dark repositories which aren't just mirrors of offical repositories but also contain home-grown packages such as "Company X custom VirtualBox Ubuntu box for VMWare" which contains an Ubuntu machine with up-to-date guest tools for the version of VMWare we use, along with the versions of Puppet, NTPd, Samba, and so forth that we use for all our Vagrant-ified infrastructure. Thus we save time over building the guest VM from scratch (about 20 minutes for each box we spin up) but someone has to maintain that repository.

Re: We check our node_modules folder into source control

#217

Post by cxr »

jakear wrote: post #86

So instead of downloading the current set of dependencies after cloning, you download every dependency ever used while cloning? And you do this to "save bandwidth"? Doesn't make much sense. (Yes I know about shallow clones, but it's often nice to have the full history around) Any reasonable CI tool will have a way to cache generated assets based on file contents, that's the way to go here IMO.

Consider if you had written, "So instead of downloading the current project source tree, you download every version..?"

That is what happens in a DVCS, after all—in fact, it's sort of the whole point. If you're so uncomfortable with this, it might be worth asking yourself whether it was ever really the case that you agreed that DVCSes were the right approach. (Even then, still no reason to embrace package managers like some kind of paramilitary force that's subject to its own rules—better to just improve your version control system to handle things the right way, right?)

Re: We check our node_modules folder into source control

#218

Post by cxr »

xtian wrote: post #124
fivea wrote: post #115

Earlier quoted context omitted.

> He offers a long list of arguments and even though I don't check in node_modules, some of the arguments are compelling (...) Are they, really? The less debatable point is arguing that making CICD pipelines slightly faster, but this feels like an appeal to microoptimization. Any free tier CICD system out there let's you do a single npm install and move these dependencies as far as you'd like into the pipeline as art…

The fact that there’s no mention of the `npm ci` command (or yarn or pnpm) makes me wonder how deeply this problem was investigated before using it as a justification for this hacky workaround. One wonders why they aren’t checking in the binaries for their database and language runtimes. Surely this would save crucial seconds in project setup.

There's a tremendous irony here, which is that these package managers are little more than a hack to let people manage parts of their project tree (and the accompanying shame) outside the harsh and knowing gaze of their version control system.

Re: We check our node_modules folder into source control

#219

Post by xtian »

cxr wrote: post #218
xtian wrote: post #124

Earlier quoted context omitted.

The fact that there’s no mention of the `npm ci` command (or yarn or pnpm) makes me wonder how deeply this problem was investigated before using it as a justification for this hacky workaround. One wonders why they aren’t checking in the binaries for their database and language runtimes. Surely this would save crucial seconds in project setup.

There's a tremendous irony here, which is that these package managers are little more than a hack to let people manage parts of their project tree (and the accompanying shame) outside the harsh and knowing gaze of their version control system.

So I imagine that if you ship an app on Linux you check in the source of every system library and utility it depends on, right? Anything less is little more than a hack to escape the harsh and knowing gaze of your version control system.

Re: We check our node_modules folder into source control

#220

Post by jakear »

cxr wrote: post #217
jakear wrote: post #86

So instead of downloading the current set of dependencies after cloning, you download every dependency ever used while cloning? And you do this to "save bandwidth"? Doesn't make much sense. (Yes I know about shallow clones, but it's often nice to have the full history around) Any reasonable CI tool will have a way to cache generated assets based on file contents, that's the way to go here IMO.

Consider if you had written, "So instead of downloading the current project source tree, you download every version..?" That is what happens in a DVCS, after all—in fact, it's sort of the whole point. If you're so uncomfortable with this, it might be worth asking yourself whether it was ever really the case that you agreed that DVCSes were the right approach. (Even then, still no reason to embrace package managers li…

Theres cost-benefit tradeoffs for sure. It'd be nice to be able to do a big bisect without needing to flash the dependencies every iteration, but if it comes at the cost of making everything else slower (more bandwidth, more disk I/O, larger diffs to parse, etc.), it just isn't worth it.

Sure you can say "well why not just make git better then it will handle any operation in any size repo imperceptibly quickly", but I think you and I both know that isn't anywhere near as easy to implement as it is to type.

Post reply on HN