Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We check our node_modules folder into source control

jackfranklin.co.uk

Post reply on HN
Print view View original

221–230 of 241 posts

Re: We check our node_modules folder into source control

#221

Post by notthathardbro »

smarx007 wrote: post #167
paulbjensen wrote: post #159

I understand what he's getting at, but if you are trying to have an exact copy of code in order to guarantee an exact behaviour, then you would want to extend that to the operating system used to run the software. One example was having code where the test suite ran fine on my local MacBook, but would fail on CI (Linux). It turned out that on Linux finding files by name is case sensitive, whereas on Mac OS it isn't,…

Isn’t it where we are going with Nix and co?

This entire comment thread is just torture for anyone who uses real dep management tools like Nix and Guix. Sorry, but it's just exhausting watching the same asinine conversations play out over and over. I guarantee there are multiple people who have spent more time pontificating over pointless ways to microscopically improve the node tooling situation who could've picked up Nix in 1/10th the amount of time.

But then again, I know there's a bunch of Nixers that pick their head up slightly, shake it, and then just go back to work on actually interesting problems instead of a millionth discussion about dealing with npm. Christ, the stuff people put up with.

EDIT: Ironic, this being here along with the CISA/log4j post where everyone is yammering about SBOM (software bill of matearials). Again, I just glance over at Nix and go, "sure, what do you want to know, I can tell you instantly if log4j is anywhere and if it's a vulnerable version (excluding non-source-built packages in nixpkgs)".

Re: We check our node_modules folder into source control

#222

Post by NavinF »

ng12 wrote: post #212
NavinF wrote: post #134

Earlier quoted context omitted.

Dude he addressed that in the next few lines: > There are times where this doesn't work; updating TypeScript may require us to update some code to fix errors that the new version of TypeScript is now detecting. In that case we have the ability to override the rule. As with anything in software engineering, most "rules" are guidelines, and we're able to side-step them when required.

Ah, I glossed over that. But that just makes me more confused, not less. On a long lived project most changes to node_modules will require code changes... just not sure what the point is.

If your libraries have breaking changes on every update, sure. Most of the libs I use haven’t had one in decades, but I don’t write js so maybe my experiences aren’t that relevant.

Re: We check our node_modules folder into source control

#223

Post by Gigachad »

nickkell wrote: post #206
Gigachad wrote: post #106

Earlier quoted context omitted.

That’s already exactly the same risk almost all web developers take currently. Yes it is a real threat, but it’s too hard to deal with and not often exploited.

We all hope and rely on the fact that popular OSS projects have enough eyeballs on them to make sure nothing malicious slips through. What is proposed here allows a load of changes to be made that completely bypass the normal review process

This doesn't bypass the review process because no ones review process includes auditing the code of all things in the package lock files. This is no more or less secure than the current way of doing things.

Re: We check our node_modules folder into source control

#224

Post by xdennis »

nhoughto wrote: post #77

I like the benefits but our node_modules is 1.9GB, def not checking that in. A Git bot that parses changes to yarn lock and comments with size/loc/files of all the deps etc, kinda like how coverage bots work. get all the observability benefits without the checkin cost, also wouldn’t have to split npm changed and code changes..

What do you have in there that it's that big? I assume there are quite a few binary files which probably shouldn't be in Git.

Re: We check our node_modules folder into source control

#225

Post by cxr »

jakear wrote: post #220
cxr wrote: post #217

Earlier quoted context omitted.

Consider if you had written, "So instead of downloading the current project source tree, you download every version..?" That is what happens in a DVCS, after all—in fact, it's sort of the whole point. If you're so uncomfortable with this, it might be worth asking yourself whether it was ever really the case that you agreed that DVCSes were the right approach. (Even then, still no reason to embrace package managers li…

Theres cost-benefit tradeoffs for sure. It'd be nice to be able to do a big bisect without needing to flash the dependencies every iteration, but if it comes at the cost of making everything else slower (more bandwidth, more disk I/O, larger diffs to parse, etc.), it just isn't worth it. Sure you can say "well why not just make git better then it will handle any operation in any size repo imperceptibly quickly", but…

> you can say "well why not just make git better then it will handle any operation in any size repo imperceptibly quickly", but I think you and I both know that isn't anywhere near as easy to implement as it is to type

Good thing I didn't type that, then. That's not what the shape of my argument looks like at all.

There's a massive leap between, "we don't want every clone of our repo per se to carry all the baggage of our dependencies' histories, so it would by nice to have some scheme for handling lightweight, shallow copies" and "... so we decided the right way to do that, rather than making that a first class feature of the version control system we're using, is to create a hack in the form of a new set of unrelated tools meant to circumvent our VCS's fundamentals completely—so from our its point of view, these controlled objects and the scheme we use for managing them are invisible and might as well not even exist."

Re: We check our node_modules folder into source control

#226

Post by dijit »

DaiPlusPlus wrote: post #140
dijit wrote: post #123

Earlier quoted context omitted.

The main reason to use perforce in the first place is because of assets. Locking of files too, but that’s only necessary on non-diffable assets.

I've only ever used Perforce through a command-line interface. What's the UX like when using Perforce for blobs/assets? Does everyone have to use the command-line or is there a GUI experience (with fast-rendering thumbnails?)? Forgive the questions, I'm just curious about the minutia and peculiarities of the gaming-biz because I've never worked in the field.

The CLI is punishing as hell.

Most game engines natively integrate with perforce so it's largely invisible.

Those that don't use the engine typically use an IDE which automatically checks out files; then they just check them in using the p4 GUI.

Even I don't use the commandline and I'm a CLI-Junkie.

Re: We check our node_modules folder into source control

#227

Post by nhoughto »

xdennis wrote: post #224
nhoughto wrote: post #77

I like the benefits but our node_modules is 1.9GB, def not checking that in. A Git bot that parses changes to yarn lock and comments with size/loc/files of all the deps etc, kinda like how coverage bots work. get all the observability benefits without the checkin cost, also wouldn’t have to split npm changed and code changes..

What do you have in there that it's that big? I assume there are quite a few binary files which probably shouldn't be in Git.

Tbh haven’t gone deep into it, now you ask maybe I should. It compresses ok, zstd to 240MB. React + react native, we lack etc tons of stuff. My guess/hope is mostly dev dependencies.

Re: We check our node_modules folder into source control

#228

Post by littlecranky67 »

The article contains some false assumptions and false statements

> Having your node_modules checked in guarantees that two developers running the code are running the exact same code with the exact same set of dependencies.

No, it doesn't, your system environment is important. Any code executed can check for environment parameters and branch accordingly. For example, a simple "if (macOSVersion === "10.10") {} else { }" would run different code branches and possibly produce different results, even when executing the same code binaries.

The reason you don't check in node_modules is that differences in system env during build time produces different build results - checking in node_modules fixes that, but does not handle system differences at runtime.

Re: We check our node_modules folder into source control

#229

Post by penguin_booze »

Can't speak for elsewhere, but at $WORK, the development and CI environments sit behind a package resolver (don't know if that's the right word) which transparently caches and proxies package downloads. So dependencies are implicitly persisted permanently by virtue of usage, even if only once.

Re: We check our node_modules folder into source control

#230

Post by cxr »

xtian wrote: post #219
cxr wrote: post #218

Earlier quoted context omitted.

There's a tremendous irony here, which is that these package managers are little more than a hack to let people manage parts of their project tree (and the accompanying shame) outside the harsh and knowing gaze of their version control system.

So I imagine that if you ship an app on Linux you check in the source of every system library and utility it depends on, right? Anything less is little more than a hack to escape the harsh and knowing gaze of your version control system.

> I imagine that if you ship an app on Linux you check in the source of every system library and utility it depends on, right?

Dumb false equivalence, since that ("every system library and utility it depends on") is not the argument of the side you're trying to appear to offer a response to. Please refer to the HN guidelines.

A less dishonest retort would be to ask if one should check in the dependencies that are analogous to what ends up in node_modules, and the response would be, "welp, that's exactly how many app developers have been known to approach things, so 'yes'."

Post reply on HN