Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NPMprune: Remove unnecessary files from node_modules to optimize storage

github.com

Post reply on HN
Print view View original

11–20 of 52 posts

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#11

Post by nailer »

swatcoder wrote: post #5

> In deployment scripts: > > wget -qO- https://raw.githubusercontent.com/xthezealot/npmprune/master... | sh -- -p Serious question: Is this the norm now? Are people actually executing unversioned wget'd shell scripts from random github users as part of their deployment workflow?

The threat model is exactly the same as executing untrusted, uninspected content you've downloaded locally.

I could do some tricks where I sent different files based on user agent, but still... most people aren't inspecting the download anyway before running it.

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#13

Post by paulddraper »

armchairhacker wrote: post #3

I use https://pnpm.io whenever possible. It has many benefits, the main one is that the node modules are symlinked to one big repo in your home directory, so there isn’t nearly as much duplication.

*hard linked

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#14

Post by arthurwhite »

an_ko wrote: post #8

This is wildly unsafe. - Some packages contain non-JS files for good reasons, and they may break in subtle unpredictable ways when you mess with the contents of their package. - Node.js will happily run JavaScript files even if they're not "*.js": A file like "hello.alsdfhlshdfl" works just fine as long as its content parses. There is no guarantee that your dependencies (and their recursive dependencies) don't static…

- Of course, this entails the risk of occasional breakage. But for 99% of modules, this has no impact at runtime.

- The patterns used to find files are specific enough to target only those files that are well known to be useless at runtime.

- The license texts of these libraries can be copied and merged into a main LICENSE file.

- Have you seen the number of modules installed by most major libraries? Making a pull request for each of them is humanly impossible and counter-productive. It's easier to use a simple script that releases dozens of MB in a few seconds.

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#16

Post by arthurwhite »

c0n5pir4cy wrote: post #7

Just for package authors (or people looking for some easy pull requests) out there that might not know this exists. NPMs package.json has a `files` field which allows you to define which files are included on an npm install: https://docs.npmjs.com/cli/v6/configuring-npm/package-json#f... . This also extends to an .npmignore file that works similar to a .gitignore file.

If everyone used the `file` field, the world would be a better place, for sure.

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#17

Post by nfriedly »

creatonez wrote: post #9
c0n5pir4cy wrote: post #7

Just for package authors (or people looking for some easy pull requests) out there that might not know this exists. NPMs package.json has a `files` field which allows you to define which files are included on an npm install: https://docs.npmjs.com/cli/v6/configuring-npm/package-json#f... . This also extends to an .npmignore file that works similar to a .gitignore file.

Just beware that some files may seem unnecessary but are expected from an idiomatic npm package. Three things that come to mind -- a markdown file named README.md, any generated typescript definitions, and typescript/babel sourcemaps. And something I've seen far too often: please don't give a minified, rolled up bundle as the only option, otherwise you are chucking your library's users back into the dark ages of Bowe…

The readme gets included automatically even if you don't specify it in the files field, ditto for the changelog license, and package.json.

Compare https://github.com/express-rate-limit/express-rate-limit/blo... to https://www.npmjs.com/package/express-rate-limit?activeTab=c...

Agree with you about the other points.

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#18

Post by QuadmasterXLII »

lxe wrote: post #2

We did exactly this when packaging and deploying large node manifests at one of my former companies. Be super careful of removing large swaths of files. Out of 150,000 node modules in your manifest, I'm willing to bet at least one of them is doing something by reading one of these non-source files.

Test and bifurcate I guess.

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#20

Post by Ayesh »

Yeah no.

Npm already does it at the package registry with ignore/npmignore files, and that's the package authors choice. How much storage can you really save? 50MB? 200MB? is it really worth the risk of running rm on some glob pattern and cross your fingers the packages don't require any of the deleted files?

Post reply on HN