Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NPMprune: Remove unnecessary files from node_modules to optimize storage

github.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#31

Post by josephg »

creatonez wrote: post #9
c0n5pir4cy wrote: post #7

Just for package authors (or people looking for some easy pull requests) out there that might not know this exists. NPMs package.json has a `files` field which allows you to define which files are included on an npm install: https://docs.npmjs.com/cli/v6/configuring-npm/package-json#f... . This also extends to an .npmignore file that works similar to a .gitignore file.

Just beware that some files may seem unnecessary but are expected from an idiomatic npm package. Three things that come to mind -- a markdown file named README.md, any generated typescript definitions, and typescript/babel sourcemaps. And something I've seen far too often: please don't give a minified, rolled up bundle as the only option, otherwise you are chucking your library's users back into the dark ages of Bowe…

Yep. And if you’re writing typescript, please include type definitions, source maps, type definition source maps and the original typescript source.

Having all of this stuff makes it possible to ctrl+click on functions in my libraries and read the corresponding source code. That’s a godsend during development - well worth a few extra kb of files in the npm module.

tsconfig.json:

    "declaration": true,
    "declarationMap": true,
    "sourceMap": true,
    ...
package.json (assuming typescript compiles src/ to dist/):

    "files": [
      "dist/*",
      "src/*"
    ],

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#34

Post by shepherdjerred »

josephg wrote: post #31
creatonez wrote: post #9

Earlier quoted context omitted.

Just beware that some files may seem unnecessary but are expected from an idiomatic npm package. Three things that come to mind -- a markdown file named README.md, any generated typescript definitions, and typescript/babel sourcemaps. And something I've seen far too often: please don't give a minified, rolled up bundle as the only option, otherwise you are chucking your library's users back into the dark ages of Bowe…

Yep. And if you’re writing typescript, please include type definitions, source maps, type definition source maps and the original typescript source. Having all of this stuff makes it possible to ctrl+click on functions in my libraries and read the corresponding source code. That’s a godsend during development - well worth a few extra kb of files in the npm module. tsconfig.json: "declaration": true, "declarationMap":…

It makes me wonder why these are even configurable. These should all be emitted by default.

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#35

Post by lxe »

QuadmasterXLII wrote: post #18
lxe wrote: post #2

We did exactly this when packaging and deploying large node manifests at one of my former companies. Be super careful of removing large swaths of files. Out of 150,000 node modules in your manifest, I'm willing to bet at least one of them is doing something by reading one of these non-source files.

Test and bifurcate I guess.

In production, preferably. This way you'll immediately find any issues and will have top priority allocated to fixing them.

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#36

Post by butshouldyou »

arthurwhite wrote: post #21
pavlov wrote: post #12

Why not use yarn? It has a much more reliable solution: https://yarnpkg.com/features/pnp

Because its primary focus is on redefining how dependencies are stored and accessed, rather than modifying the contents of these dependencies. Useless files will still be there. Also, when you create a Docker image, you avoid packing in dev tools that aren't absolutely essential (such as Yarn).

FYI: The default node Docker images already include yarn.

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#37

Post by nusmella »

arthurwhite wrote: post #23
Alifatisk wrote: post #15

Just use pnpm

While pnpm optimizes storage and reduces duplication, it does not inherently remove non-essential files (like documentation, Markdown, or test files) within the dependencies. Also, when you create a Docker image, you avoid packing in dev tools that aren't absolutely essential (such as pnpm).

Even the alpine nodejs images have pnpm and yarn nowadays

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#38

Post by akdor1154 »

arthurwhite wrote: post #14
an_ko wrote: post #8

This is wildly unsafe. - Some packages contain non-JS files for good reasons, and they may break in subtle unpredictable ways when you mess with the contents of their package. - Node.js will happily run JavaScript files even if they're not "*.js": A file like "hello.alsdfhlshdfl" works just fine as long as its content parses. There is no guarantee that your dependencies (and their recursive dependencies) don't static…

- Of course, this entails the risk of occasional breakage. But for 99% of modules, this has no impact at runtime. - The patterns used to find files are specific enough to target only those files that are well known to be useless at runtime. - The license texts of these libraries can be copied and merged into a main LICENSE file. - Have you seen the number of modules installed by most major libraries? Making a pull re…

> but for 99% of modules, this has no impact at runtime

So for the typical enterprise crapware where the app template installs about 2,000 packages for a React Hello World, how many broken modules is that?

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#39

Post by dunham »

petesergeant wrote: post #10
swatcoder wrote: post #5

> In deployment scripts: > > wget -qO- https://raw.githubusercontent.com/xthezealot/npmprune/master... | sh -- -p Serious question: Is this the norm now? Are people actually executing unversioned wget'd shell scripts from random github users as part of their deployment workflow?

> now For about the last 15 years

And before the web existed, people would distribute software packaged inside executable shell scripts (https://en.wikipedia.org/wiki/Shar).

It looks like that practice goes back at least 40 years.

Re: NPMprune: Remove unnecessary files from node_modules to optimize storage

#40

Post by filterfiber »

arthurwhite wrote: post #14
an_ko wrote: post #8

This is wildly unsafe. - Some packages contain non-JS files for good reasons, and they may break in subtle unpredictable ways when you mess with the contents of their package. - Node.js will happily run JavaScript files even if they're not "*.js": A file like "hello.alsdfhlshdfl" works just fine as long as its content parses. There is no guarantee that your dependencies (and their recursive dependencies) don't static…

- Of course, this entails the risk of occasional breakage. But for 99% of modules, this has no impact at runtime. - The patterns used to find files are specific enough to target only those files that are well known to be useless at runtime. - The license texts of these libraries can be copied and merged into a main LICENSE file. - Have you seen the number of modules installed by most major libraries? Making a pull re…

> Of course, this entails the risk of occasional breakage. But for 99% of modules, this has no impact at runtime.

Right so most projects end up with 100's (random one I have is 700+) modules. Which would mean multiple breakages.

The worst part isn't the breakage - it's not knowing where or when it breaks, and because it could be missed when it's being bundled it can happen in production.

The bundling step should effectively be doing the file pruning for you (or even parts of files) and you can be a lot more confident that won't miss things.

node_modules are generally big (580MB in my case), but I don't know why you'd trade 580MB of storage for reliability. For us the 580MB will get bundled under 1MB for our web application, essentially all dev machines will be 512GB+ at this point anyway.

Post reply on HN