Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

131–140 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#131

Post by syncsynchalt »

pif wrote: post #48
jerf wrote: post #29

Earlier quoted context omitted.

"I don't find the cons all that compelling to be honest" This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm. You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to…

> This is a solid example of how things change at scale. 5% is 5% at any scale.

In some scenarios the equation flips, and the enterprise is looking for _more_ scale.

The more bandwidth that Cloudflare needs, the more leverage they have at the peering table. As GitHub's largest repo (the @types / DefinitelyTyped repo owned by Microsoft) gets larger, the more experience the owner of GitHub (also Microsoft) gets in hosting the world's largest git repos.

I would say this qualifies as one of those cases, as npmjs is hosted on Azure. The more resources that NPM needs, the more Microsoft can build towards parity with AWS's footprint.

Re: My failed attempt to shrink all NPM packages by 5%

#132

Post by jeffhuys »

jefozabuss wrote: post #71
Alifatisk wrote: post #39

Earlier quoted context omitted.

At least, switch to pnpm minimize the bloat

I just installed a project with pnpm about 120 packages mostly react/webpack/eslint/redux related with prod env: 700MB without prod env: 900MB sadly the bloat cannot be avoided that well :/

pnpm stores them in a central place and symlinks them. You’ll see the benefits when you have multiple projects with a lot of the same packages.

Re: My failed attempt to shrink all NPM packages by 5%

#133

Post by JZerf »

gregmac wrote: post #65
pif wrote: post #48

Earlier quoted context omitted.

> This is a solid example of how things change at scale. 5% is 5% at any scale.

5% off your next lunch and 5% off your next car are very much not the same thing.

Those lunches could add up to something significant over time. If you're paying $10 per lunch for 10 years, that's $36,500 which is pretty comparable to the cost of a car.

Re: My failed attempt to shrink all NPM packages by 5%

#134

Post by eitau_1 »

cedws wrote: post #25

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

It's not even funny:

  $ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/*
  /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux:
  .r-xr-xr-x 129k root  1 Jan  1970 xsel

  /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows:
  .r-xr-xr-x 444k root  1 Jan  1970 clipboard_i686.exe
  .r-xr-xr-x 331k root  1 Jan  1970 clipboard_x86_64.exe
(clipboardy ships executables and none of them can be run on NixOS btw)

Re: My failed attempt to shrink all NPM packages by 5%

#135

Post by michaelmior »

Null-Set wrote: post #64
michaelmior wrote: post #59

Earlier quoted context omitted.

Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.

No, it can't because the checksums won't match.

I don't think that's actually a problem, but it would require continuing to host both versions (at distinct URLs) for any users who may have installed the package before the Zopfli-compressed version completed. Although I think you could also get around this by tracking whether the newly-released package was ever served by the API. If not, which is probably the common case, the old gzip-compressed version could be deleted.

Re: My failed attempt to shrink all NPM packages by 5%

#136

Post by syncsynchalt »

jeffhuys wrote: post #132
jefozabuss wrote: post #71

Earlier quoted context omitted.

I just installed a project with pnpm about 120 packages mostly react/webpack/eslint/redux related with prod env: 700MB without prod env: 900MB sadly the bloat cannot be avoided that well :/

pnpm stores them in a central place and symlinks them. You’ll see the benefits when you have multiple projects with a lot of the same packages.

You'll also see the benefit when `rm -rf`ing a `node_modules` and re-installing, as pnpm still has a local copy that it can re-link after validating its integrity.

Re: My failed attempt to shrink all NPM packages by 5%

#137

Post by ziddoap »

dgfitz wrote: post #128
gregmac wrote: post #65

Earlier quoted context omitted.

5% off your next lunch and 5% off your next car are very much not the same thing.

So what, instead of 50k for a car you spend 47.5k? If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it. 5% is 5%.

Why do so many people take illustrative examples literally?

I'm sure you can use your imagination to substitute "lunch" and "car" with other examples where the absolute change makes a difference despite the percent change being the same.

Even taking it literally... The 5% might not tip the scale of whether or not I can purchase the car, but I'll spend a few hours of my time comparing prices at different dealers to save $2500. Most people would consider it dumb if you didn't shop around when making a large purchase.

On the other hand, I'm not going to spend a few hours of my time at lunch so that I can save an extra $1 on a meal.

Re: My failed attempt to shrink all NPM packages by 5%

#138

Post by syncsynchalt »

hinkley wrote: post #111
avodonosov wrote: post #13

My experiment on how to reduce javascript size of every web app by 30-50% : https://github.com/avodonosov/pocl Working approach, but in the end I abandoned the project - I doubt people care about such js size savings.

I got measurable decreases in deployment time by shrinking the node_modules directory in our docker images. I think people forget that, when you’re copying the same images to dozens and dozens of boxes, any improvement starts to add up to real numbers.

I've not done it, but have you considered using `pnpm` and volume-mounting a shared persistent `pnpm-store` into the containers? It seems like you'd get near-instant npm installs that way.

Re: My failed attempt to shrink all NPM packages by 5%

#139

Post by kmacdough »

woadwarrior01 wrote: post #117

From the RFC on github[1]. > Zopfli is written in C, which presents challenges. Unless it was added to Node core, the CLI would need to (1) rewrite Zopfli in JS, possibly impacting performance (2) rely on a native module, impacting reliability (3) rely on a WebAssembly module. All of these options add complexity. Wow! Who's going to tell them that V8 is written in C++? :) [1]: https://github.com/npm/rfcs/pull/595

It's not about C per-se, as much as each native compiled dependency creates additional maintenance concerns. Changes to hardware/OS can require a recompile or even fixes. NPM build system already requires a JavaScript runtime, so is already handled as part of existing maintenance. The point is that Zopfli either needs to be rewritten for a platform-agnostic abstraction they already support, or else Zopfli will be added to a list of native modules to maintain.

Re: My failed attempt to shrink all NPM packages by 5%

#140

Post by bangaladore »

I think the main TLDR here [1]:

> For example, I tried recompressing the latest version of the typescript package. GNU tar was able to completely compress the archive in about 1.2 seconds on my machine. Zopfli, with just 1 iteration, took 2.5 minutes.

[1] https://github.com/npm/rfcs/pull/595#issuecomment-1200480148

My question of course would be, what about LZ4, or Zstd or Brotli? Or is backwards compatibility strictly necessary? I understand that GZIP is still a good compressor, so those others may not produce meaningful gains. But, as the author suggests, even small gains can produce huge results in bandwidth reduction.

Post reply on HN