Live data from Hacker News

We check our node_modules folder into source control

jackfranklin.co.uk

41–50 of 241 posts

Re: We check our node_modules folder into source control

#41

The GitHub mirror of the Chrome DevTools repo has the node_modules folder here: https://github.com/ChromeDevTools/devtools-frontend/blob/mai... But there is nuance (there always is...), the README file in node_modules is here: https://github.com/ChromeDevTools/devtools-frontend/blob/mai... - and it makes it clear the only NPM dependencies used by the build-system or infrastructure is meant to be checked-in. Other NPM…

Maybe I'm wrong, but I see quite a lot of node_modules here - https://source.chromium.org/chromium/chromium/src/+/main:thi...

Here are all "node_modules" folders I can find (sorry I'm not much familiar with npm/js to know more) https://source.chromium.org/search?q=file:node_modules%2F$%2...

Re: We check our node_modules folder into source control

#42

Does it work across x86, 64, arm, linux, mac and windows? Some modules straight up download binaries so I don’t see being so straightforward.

No, and that detail is dangerously missing from the advice. lol Anyone that does this and has teams on Mac, Windows and Linux will find out how crappy this is very quickly.

It’s even worse than that. If you support all LTS/current Node versions—as is, and should be, very common for libraries—it'll break even on a single platform. I’m sure it’s solvable, but the solution would be so complex it’s tantamount to building a new package manager.

Re: We check our node_modules folder into source control

#43
post #41

The GitHub mirror of the Chrome DevTools repo has the node_modules folder here: https://github.com/ChromeDevTools/devtools-frontend/blob/mai... But there is nuance (there always is...), the README file in node_modules is here: https://github.com/ChromeDevTools/devtools-frontend/blob/mai... - and it makes it clear the only NPM dependencies used by the build-system or infrastructure is meant to be checked-in. Other NPM…

Maybe I'm wrong, but I see quite a lot of node_modules here - https://source.chromium.org/chromium/chromium/src/+/main:thi... Here are all "node_modules" folders I can find (sorry I'm not much familiar with npm/js to know more) https://source.chromium.org/search?q=file:node_modules%2F$%2...

This file seems to be controlling what gets ("checked-in") - https://source.chromium.org/chromium/chromium/src/+/main:thi...

In our industry (games) we often do that - checkin prebuilt code in the depots (typically "p4"). I'm not saying it's wrong/right, it's just what we do (not 100% fully, but almost, though people in IT/infra tend to do otherwise).

Re: We check our node_modules folder into source control

#44
post #20
post #15

Earlier quoted context omitted.

Check in your lock file.

Done. Then I delete it, and `npm install`. Then commit. The majority of people I have worked with do that. On Friday some dude was saying "shrinkwrap v2 is not shrinkwrap v1" and the advice was "delete it and npm install" then commit. (Payment company software manager).

Yes, this is fairly common in my experience. Whenever local build fails, a lot of troubleshooting advices says to nuke node_modules and lock file.

Re: We check our node_modules folder into source control

#45

Earlier quoted context omitted.

RTFA. It says, quite succinctly: > I currently work at Google on the Chrome DevTools team and we check our node_modules folder into source control

tbf Google is so far ahead that it’s kinda not a real company. Aka you don’t have to adopt every best practice google does since you aren’t that big, mature, secure or rich.

Now we just need to convince all those startups burning cycles trying to get Kubernetes working instead of working on their actual product...

Re: We check our node_modules folder into source control

#46
post #20

Earlier quoted context omitted.

Done. Then I delete it, and `npm install`. Then commit. The majority of people I have worked with do that. On Friday some dude was saying "shrinkwrap v2 is not shrinkwrap v1" and the advice was "delete it and npm install" then commit. (Payment company software manager).

Anyone recommend a build check to block PR's from including fully rebuilt lockfiles? Other than a high LOC, it's too easy for this to slip through.

In my experience this is probably not viable. A single, valid update to a direct dependency can bring dozens or even hundreds of updated sub-dependencies. And an audit fix will often update just the lockfile. Maybe I lack imagination, but I can’t think of a workable heuristic for determining how the lockfile was changed.

I think, instead, it would be good to move in a different direction for sub-dependencies generally. A rough sketch:

1. Packages state their dependencies as they do currently.

2. When released, they’re built and bundled by the package manager host.

3. Included in the bundle is a manifest of the dependencies used, specific imports used, and a hash for each (recursively until exhausted).

4. On install, identical code (same hash/same bundled result) is deduplicated.

5. A human readable record of dependencies and imports used is produced. It’s important that it’s human readable, because:

6. This should not be filtered out in diffs. It should be subject to review just like any other change.

All of this is pretty complex, and there are probably ways to reduce that complexity. But it has some obvious advantages:

- Only your direct dependencies are installed. Tons of bloat can be stripped out.

- Even deduplication can be performed on the package manager’s servers. And hashes aren’t a particularly expensive lookup.

- It would go a long way towards addressing audit fatigue: if your dependencies’ bundles don’t include affected code, the audit doesn’t apply; if they do, you can be reasonably confident the audit is valid.

- A (wild guess) huge amount of the time, sub-dependency changes will require little to no review. Their stable parts will seldom change, and the parts shared among several dependencies could be reviewed as one unit.

- Lock files themselves just need to track direct dependencies (and even then, only to support semver ranges).

Re: We check our node_modules folder into source control

#47
post #43
post #41

Earlier quoted context omitted.

Maybe I'm wrong, but I see quite a lot of node_modules here - https://source.chromium.org/chromium/chromium/src/+/main:thi... Here are all "node_modules" folders I can find (sorry I'm not much familiar with npm/js to know more) https://source.chromium.org/search?q=file:node_modules%2F$%2...

This file seems to be controlling what gets ("checked-in") - https://source.chromium.org/chromium/chromium/src/+/main:thi... In our industry (games) we often do that - checkin prebuilt code in the depots (typically "p4"). I'm not saying it's wrong/right, it's just what we do (not 100% fully, but almost, though people in IT/infra tend to do otherwise).

Ah Perforce... you have my sympathy.

Does your team/company store art and asset blobs (even rendered FMVs?) in Perforce too, or do you have a separate asset-management system for that? If you do have two separate systems, how do you keep the asset-store and source-control in-sync?

It's been a long time (easily 20 years now) since I dabbled in any high-end creative software (like 3ds, etc), but I remember they generally all had large binary singleton project files (like Flash .fla and Photoshop .psd files) that couldn't be deconstructed and effectively diffed by any source-control system (though Flash eventually supported external ActionScript source-files), I'm curious how that affects your org's asset storage needs.

Re: We check our node_modules folder into source control

#48
Maven, Gradle and the like always download dependencies as part of the build, and if a server is down (or flaky like jitpack) good lord is it annoying.

What I don't understand is why dependency download isn't a separate task you do before compilation. FIRST you download all the dependencies to stabilize that, then you build the code. The reproducibility is the big one.

IT drives me nuts that the dependencies are hidden away in maven and gradle. I have to lookup obscure "download dependencies to a lib" task configuration. Still the obsession with massive jars/wars/whatever when all you should have to update in a deploy is the difference in the libs and the main code jar.

The reason for this obscurement is pretty much "uh, it saves disk space?" which is a laughable consideration given the bloat in war files, docker images, and the like.

I agree with the poster.

Re: We check our node_modules folder into source control

#49

No, this is what the lock file is for.

Note the other child comments of this; NPM has failed to make the lockfile reliable across systems and versions. Here's an example from March that isn't fixed: https://github.com/npm/cli/issues/2846

The fact that you have so many people believing "nuke node_modules and delete package-lock.json" is a reasonable step in diagnosing an error is damning to NPM.

We don't check in our node_modules, but "use the lockfile" is not a valid counter to this article's points.

Re: We check our node_modules folder into source control

#50
post #20

Earlier quoted context omitted.

Done. Then I delete it, and `npm install`. Then commit. The majority of people I have worked with do that. On Friday some dude was saying "shrinkwrap v2 is not shrinkwrap v1" and the advice was "delete it and npm install" then commit. (Payment company software manager).

use yarn instead, the lockfile is much better.

Problem with yarn is that it doesn't actually use dependencies' lock files... So once you publish your library to npm your lock file doesn't do anything whatsoever.
Post reply on HN