NPMprune: Remove unnecessary files from node_modules to optimize storage
1–10 of 52 posts
Re: NPMprune: Remove unnecessary files from node_modules to optimize storage
#2We did exactly this when packaging and deploying large node manifests at one of my former companies.
Be super careful of removing large swaths of files. Out of 150,000 node modules in your manifest, I'm willing to bet at least one of them is doing something by reading one of these non-source files.
Re: NPMprune: Remove unnecessary files from node_modules to optimize storage
#3I use https://pnpm.io whenever possible. It has many benefits, the main one is that the node modules are symlinked to one big repo in your home directory, so there isn’t nearly as much duplication.
Re: NPMprune: Remove unnecessary files from node_modules to optimize storage
#4I use https://pnpm.io whenever possible. It has many benefits, the main one is that the node modules are symlinked to one big repo in your home directory, so there isn’t nearly as much duplication.
Yup, same here. I’ve saved 40gb of data by recursively removing all node_modules directories from my Mac and replacing npm with pnpm.
I did notice small issues with some libraries (react testing library IIRC)
Re: NPMprune: Remove unnecessary files from node_modules to optimize storage
#5> In deployment scripts:
>
> wget -qO- https://raw.githubusercontent.com/xthezealot/npmprune/master... | sh -- -p
Serious question: Is this the norm now? Are people actually executing unversioned wget'd shell scripts from random github users as part of their deployment workflow?
Re: NPMprune: Remove unnecessary files from node_modules to optimize storage
#6We did exactly this when packaging and deploying large node manifests at one of my former companies. Be super careful of removing large swaths of files. Out of 150,000 node modules in your manifest, I'm willing to bet at least one of them is doing something by reading one of these non-source files.
This was my concern as well.
Looking at the script source, it's just matching globs, so there isn't much smarts to this. I'm sure it works most of the time, but yeah..
Do JS packages need some kind of .prodignore file similar to other .ignore files?
So with a flag passed, after doing an npm install, there's a extra cleanup step that removes explicitly marked files that aren't needed for running in prod?
(Not a fully formed idea, I'm sure I'm not thinking of drawbacks with this)
Edit: this sort of exists as the .npmignore file?
https://docs.npmjs.com/cli/v10/using-npm/developers#keeping-...
Re: NPMprune: Remove unnecessary files from node_modules to optimize storage
#7Just for package authors (or people looking for some easy pull requests) out there that might not know this exists.
NPMs package.json has a `files` field which allows you to define which files are included on an npm install: https://docs.npmjs.com/cli/v6/configuring-npm/package-json#f....
This also extends to an .npmignore file that works similar to a .gitignore file.
Re: NPMprune: Remove unnecessary files from node_modules to optimize storage
#8This is wildly unsafe.
- Some packages contain non-JS files for good reasons, and they may break in subtle unpredictable ways when you mess with the contents of their package.
- Node.js will happily run JavaScript files even if they're not "*.js": A file like "hello.alsdfhlshdfl" works just fine as long as its content parses. There is no guarantee that your dependencies (and their recursive dependencies) don't statically or dynamically load files with completely arbitrary filenames.
- If you distribute packages with license files stripped this way, you are violating licenses that require the license to be distributed along with the code.
If this is actually a major issue for you, consider instead sending PRs to upstream to tidy up their package. This will also benefit other users.
Re: NPMprune: Remove unnecessary files from node_modules to optimize storage
#9Just for package authors (or people looking for some easy pull requests) out there that might not know this exists. NPMs package.json has a `files` field which allows you to define which files are included on an npm install: https://docs.npmjs.com/cli/v6/configuring-npm/package-json#f... . This also extends to an .npmignore file that works similar to a .gitignore file.
Just beware that some files may seem unnecessary but are expected from an idiomatic npm package. Three things that come to mind -- a markdown file named README.md, any generated typescript definitions, and typescript/babel sourcemaps. And something I've seen far too often: please don't give a minified, rolled up bundle as the only option, otherwise you are chucking your library's users back into the dark ages of Bower.js.
Re: NPMprune: Remove unnecessary files from node_modules to optimize storage
#10> In deployment scripts: > > wget -qO- https://raw.githubusercontent.com/xthezealot/npmprune/master... | sh -- -p Serious question: Is this the norm now? Are people actually executing unversioned wget'd shell scripts from random github users as part of their deployment workflow?
> now
For about the last 15 years