Why not use fixed version dependencies and an internal npm registery such as JFrog Artifactory or Sonatype Nexus?
We check our node_modules folder into source control
231–240 of 241 posts
Re: We check our node_modules folder into source control
#232Earlier quoted context omitted.
What do you have in there that it's that big? I assume there are quite a few binary files which probably shouldn't be in Git.
Tbh haven’t gone deep into it, now you ask maybe I should. It compresses ok, zstd to 240MB. React + react native, we lack etc tons of stuff. My guess/hope is mostly dev dependencies.
mostly dev dependencies, i guess you really pay the price for not having a compressed intermediate format like a JAR:
- storybook is 420MB (! will look into this one!)
- babel is 110MB
- sentry sdk is 92MB (?)
- aws-sdk is 63MB
- typescript 60MB
- react-native 51MB
.. on and on, 40MB..30MB..20MB.. hundreds of them just adds up! 147 dependencies over 1M. pretty incredible/wasteful really when you look at it.
Re: We check our node_modules folder into source control
#233Upsides from storing node_modules in repo are outweighed by the downsides. Unless of course you're Google-scale and can afford to contribute filesize fixes upstream, write fancy tooling to enforce commit-time workarounds, etc. Nobody working finger-to-feature has time for this. For your average npm shop which doesn't have infinite internet oil money, here is why the article recommendations won't work for you. Your CI…
Counter point, and ignoring download size as your typical CI probably doesn't download the entire history, when are we supposed to pretend we've reviewed our dependencies? I'll admit I don't believe everyone always need to check every dep, but we're skating close nobody checking them ever.
GH actions doesnt, Azure devops does
Re: We check our node_modules folder into source control
#234Earlier quoted context omitted.
So I imagine that if you ship an app on Linux you check in the source of every system library and utility it depends on, right? Anything less is little more than a hack to escape the harsh and knowing gaze of your version control system.
> I imagine that if you ship an app on Linux you check in the source of every system library and utility it depends on, right? Dumb false equivalence, since that ("every system library and utility it depends on") is not the argument of the side you're trying to appear to offer a response to. Please refer to the HN guidelines. A less dishonest retort would be to ask if one should check in the dependencies that are ana…
Despite the sanctimony, I'm sure there's at least a slim chance of you understanding my point: the distinction between "what ends up in node_modules" and any other application dependency is arbitrary and purely conventional. There are legitimate technical reasons to check dependencies into source control, but neither the reasons cited in the article nor any pompous ascriptions of moral judgment to software tools are among them.
Re: We check our node_modules folder into source control
#235Earlier quoted context omitted.
Ah Perforce... you have my sympathy. Does your team/company store art and asset blobs (even rendered FMVs?) in Perforce too, or do you have a separate asset-management system for that? If you do have two separate systems, how do you keep the asset-store and source-control in-sync? It's been a long time (easily 20 years now) since I dabbled in any high-end creative software (like 3ds, etc), but I remember they general…
We ... commit them. Probably some really exceptional cases stay behind some "ftp"-like service (not really ftp, something like this). It maybe wasteful, but it's the established practice (it seems). People coming from other game companies pretty much use it too (some rare exceptions). Also the automotive/chip design industry uses them - and yes mainly for the big blobby non-diffable/mergeable assets. Locks are terrib…
> Locks are terrible, and yet you gotta doit sometimes, as how else would you prevent people working on the same asset.
I'm unfamiliar with P4's locking semantics; when files are "locked" does that prohibit other users from even getting a copy of the centralized file, or merely prevent users from overwriting the centralized file on push/upload? How does branching work?
If I were designing a centralized asset management system then I'd definitely add support for git-style (i.e. "many-worlds") branches instead of SVN/TFS-style "spatial" branches - but I'd also add support for some kind of "mini-branch" or deferred-conflict-resolution, whereby a file, or entire directory, can still be pushed to central storage but have multiple different representations that can be resolved/merged later, rather than immediately. So if two artists are working on the same "texture123.psd" file without realizing it then the system would let them both push (so the first artist to push would get to overwrite the file, and the second artist's push would see their file saved as "texture123.psd.v2").
There are good business reasons for having the ability to disallow changes to files in central storage, but that doesn't mean locks need to be used: it could be done by instead directing all updates/pushes to separate mini-branches, thus allowing users to push-and-forget and allowing them to defer conflict resolution while still protecting files from unwanted changes.
Re: We check our node_modules folder into source control
#236Earlier quoted context omitted.
Counter point, and ignoring download size as your typical CI probably doesn't download the entire history, when are we supposed to pretend we've reviewed our dependencies? I'll admit I don't believe everyone always need to check every dep, but we're skating close nobody checking them ever.
GH actions doesnt, Azure devops does
I think you mean they're automatically doing SCA and maybe SAST. I don't think there's a human working at microsoft reading the code for you though, is there?
Re: We check our node_modules folder into source control
#237Earlier quoted context omitted.
Counter point, and ignoring download size as your typical CI probably doesn't download the entire history, when are we supposed to pretend we've reviewed our dependencies? I'll admit I don't believe everyone always need to check every dep, but we're skating close nobody checking them ever.
My team guards up front when introducing a new dependency. You fill out a little template with security assessment as well as some other stuff, just to do a dirt simple build vs 'buy' analysis. left-pad for example would fail because the build time cost savings are not worth the ongoing maintenance cost. (In fact doing this assessment at all rarely makes sense for microlibs, by design.) Once something's in package.js…
I think formlising an assessment like that makes some sense, but the question was more around what the assurances are. So it probably works like this:
#1 You look at the dependency and do an assessment on whether it's worth including. Check.
#2 You probably require some automated checks. SAST, Depedency Scanning / SCA, maybe some DAST, etc. Check
The outstanding question though...
#1 Did anyone actually read the code of the depedency?
#2 Did anyone actually look at what the depedency itself pulls in?
#3 Are these checks re-done when you update the lock files?
#4 If nobody is doing it, who's updating the lists and rules we use to scan from?
#5 Where possible do you have the monitoring to check when an app is doing something weird? i.e. network ACLs that when they fail, cause an event, that alerts a person to investigate?
I think we're mostly agreeing here, but the wider question is why is it that folks writing the app and including the depdency don't feel responsible for these things?
Re: We check our node_modules folder into source control
#238Earlier quoted context omitted.
> I imagine that if you ship an app on Linux you check in the source of every system library and utility it depends on, right? Dumb false equivalence, since that ("every system library and utility it depends on") is not the argument of the side you're trying to appear to offer a response to. Please refer to the HN guidelines. A less dishonest retort would be to ask if one should check in the dependencies that are ana…
Despite the sanctimony, I'm sure there's at least a slim chance of you understanding my point: the distinction between "what ends up in node_modules" and any other application dependency is arbitrary and purely conventional. There are legitimate technical reasons to check dependencies into source control, but neither the reasons cited in the article nor any pompous ascriptions of moral judgment to software tools are…
As a point of fact, the sanctimony of referring to something as a "hacky workaround" began here, where _you_ were the one to (unironically) introduce the phrase: https://news.ycombinator.com/item?id=29528285> Pointing out the logical inconsistency of a strong claim is not provocation, no matter how much you feel like you are the one who is being attacked.
I don't recognize your claim that the distinction is arbitrary. Is the NPM world's distinction between package.json's "dependencies" vs "devDependencies" arbitrary? (Answer: no.)
Re: We check our node_modules folder into source control
#239I don’t check in node_modules but do a backup once in a while. With frontend nowadays it’s sad but after six months it’s highly unlikely that my project will compile, not to mention the tooling like Vue, Vite, etc that has breaking changes. I mean it’s scary. You write a program and it WONT run if you just give it enough time. Locking versions is not really a solution since often times the tooling itself, and IDE ext…
I don't see how this would work in practise. You could use module-alias[1] to actually switch to the backup copy during dev or a local build, but that will only work when the backup is on the same machine you're building on, and then everyone on the team will need to have the same backup (or use a network drive for it I guess). If you don't check in your backup then nothing will get through CI or make it to productio…
Sure I’m talking solo projects.
Re: We check our node_modules folder into source control
#240Adding my voice to say this is not a bad idea at all in practice. Before anyone trying to apply the same approach to other ecosystems, however, be aware that (frontend) JavaScript is kind of unique in this. In most languages you’d likely have at least one package somewhere in the dependency chain that is in binary form, and those tend to need a bit more vendoring mechanism to handle correctly, unless everyone in your team have exactly the same setup.