Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kill Your Dependencies

mikeperham.com

Post reply on HN
Print view View original

111–120 of 229 posts

Re: Kill Your Dependencies

#111

Post by shoover »

mwcampbell wrote: post #98
shoover wrote: post #84

Earlier quoted context omitted.

Yeah, but it seems like you listed most of the platforms most people actually use, all in the X column. How do we get from 100MB Electron deployment and fat, partially used jars and dlls to this magical tree-shaking world?

You're right; most of the languages and platforms used for developing applications don't have reliable support for tree-shaking or fine-grained static linking. But there's hope for some of them. When building Android applications, it's common to process the JVM bytecode with ProGuard before converting it to Dex bytecode. ProGuard includes a tree-shaking step. Sometimes it's necessary to tell ProGuard about specific c…

Good point about Android and .NET.

Regarding the Closure Compiler, ClojureScript touts whole program optimization as an advantage; the ClojureScript compiler just has to play by the Google Closure rules when emitting JS to take advantage. I'm sure it is harder for users of any arbitrary off-the-shelf library to gain the benefits.

Re: Kill Your Dependencies

#112

Post by mwcampbell »

skybrian wrote: post #96
mwcampbell wrote: post #50

A large number of dependencies is only a problem in environments that aren't amenable to per-function static linking or tree-shaking. These include dynamically typed languages like Python, Ruby, and JavaScript (except when using the Google Closure Compiler in advanced mode), but also platforms like the JVM and .NET when reflection is allowed. Where static linking or tree-shaking is feasible, the run-time impact of br…

Tree shaking is helpful but not enough. It makes dependencies more fine-grained and binaries smaller by removing some false sharing. But library maintainers still have to be careful about true sharing, where a function calls another function, which in turn pulls in something big (like a lot of data stored in a constant). You need both tree shaking and a community dedicated to keeping code small. Javascript has the la…

You're right; tree-shaking doesn't eliminate all instances of dependency bloat. Come to think of it, I've even seen a counter-example in C++. On Windows, a hello-world application using the wxWidgets GUI toolkit is ~2.4 MB, even statically linked. I think the problem, or at least part of it, is that the WindowProc implementation uses a big switch statement to handle all of the Win32 window messages that wx supports. So, for example, the handler for the WM_PRINTCLIENT message has to be included even if your application doesn't do any printing. Same for drag and drop. It would be better if the WindowProc implementation looked up message handlers in a table, and the application could ask wx to register the handlers for just the features that are actually used. I wouldn't be surprised if similar concerns apply to frameworks like Angular, even in Dart.

Re: Kill Your Dependencies

#113

Post by jerf »

mwcampbell wrote: post #50

A large number of dependencies is only a problem in environments that aren't amenable to per-function static linking or tree-shaking. These include dynamically typed languages like Python, Ruby, and JavaScript (except when using the Google Closure Compiler in advanced mode), but also platforms like the JVM and .NET when reflection is allowed. Where static linking or tree-shaking is feasible, the run-time impact of br…

The run time resource usage risks are only one element of deep dependency trees, and certainly the one I worry about least. The biggest risks are the fact that you've handed every person in your dependency tree developer status in your project. That includes not just potential maliciousness, though that is a factor, but potentially disappearing the project entirely, potentially taking it in an unexpected direction, potentially introducing bugs (oh that all projects merely monotonically got better), potentially introducing security vulnerabilities in deep parts of the stack you wouldn't even think to audit, diamond dependencies, difficult-to-replicate builds, etc. There are then tools that can strike back at some of these, but there's something to be said for avoiding the problem.

For that matter, there's no guarantee that tree-shaking would even have fixed the referenced issue; if the library preloaded 10MB of stuff, like a Unicode definition table, that you didn't use, but the tree shaker couldn't quite prove you never would, you'll still end up with it loaded at runtime. (For that matter, you may very well be using such code even though you don't mean to, if, for instance, you have code that attempts to load the table, and uses it if it loads for something trivial, but will just keep going without it if it is not present. The tree shaker will determine (correctly!) that you're using it.)

Basically, tree shaking only sort of kind of addresses one particular problem that deep dependencies can introduce, and that one not even necessarily reliably and well.

Re: Kill Your Dependencies

#114

Post by dceddia »

mwcampbell wrote: post #98
shoover wrote: post #84

Earlier quoted context omitted.

Yeah, but it seems like you listed most of the platforms most people actually use, all in the X column. How do we get from 100MB Electron deployment and fat, partially used jars and dlls to this magical tree-shaking world?

You're right; most of the languages and platforms used for developing applications don't have reliable support for tree-shaking or fine-grained static linking. But there's hope for some of them. When building Android applications, it's common to process the JVM bytecode with ProGuard before converting it to Dex bytecode. ProGuard includes a tree-shaking step. Sometimes it's necessary to tell ProGuard about specific c…

It looks like Webpack 2 will have support for tree-shaking: https://github.com/webpack/webpack/pull/861#issuecomment-149...

There's also rollup.js (another module bundler) that supports tree-shaking: http://rollupjs.org/

Either way, it seems like the code needs to be using ES6 modules to make it all work.

Re: Kill Your Dependencies

#115

Post by cdnsteve »

Double edge sword. Deps are great! Functionality added quickly. Deps are terrible! They broke my app.

If your app has a long shelf time, the less deps you rely on, the easier to manage from what I've seen.

For some reason Golang feels like it makes sense here. Pretty much everything you need is in core. *Disclaimer, I don't have any Golang apps in prod but I'd love to hear from those that do.

Re: Kill Your Dependencies

#116

Post by mwcampbell »

jerf wrote: post #113
mwcampbell wrote: post #50

A large number of dependencies is only a problem in environments that aren't amenable to per-function static linking or tree-shaking. These include dynamically typed languages like Python, Ruby, and JavaScript (except when using the Google Closure Compiler in advanced mode), but also platforms like the JVM and .NET when reflection is allowed. Where static linking or tree-shaking is feasible, the run-time impact of br…

The run time resource usage risks are only one element of deep dependency trees, and certainly the one I worry about least. The biggest risks are the fact that you've handed every person in your dependency tree developer status in your project. That includes not just potential maliciousness, though that is a factor, but potentially disappearing the project entirely, potentially taking it in an unexpected direction, p…

You're right; I over-stated the benefits of tree-shaking. My comment came out of long frustration that so many of the languages and platforms we use don't even support tree-shaking, so if we care about binary size, we have to do micro-management of dependencies that the machine should be able to do for us. But you're right; the other issues with dependencies remain.

About your 10 MB Unicode table example, did you have in mind the ICU data table that's included with Chromium and, more recently, all the Electron-based apps?

Re: Kill Your Dependencies

#117

Post by sytse »

Rafert wrote: post #76

Excellent article. I tried to develop on GitLab once but the sheer amount of gems it pulls in (~100 directly declared, 350+ including dependencies if I remember correctly) with a bunch of installation problems made me decide it was not worth the hassle.

I'm sorry to hear you had installation problems trying to develop for GitLab. Have you tried the GitLab development kit https://gitlab.com/gitlab-org/gitlab-development-kit? If still are interested and experience problems please email support@gitlab.com and reference this comment for help.

I agree with the article, the less dependencies the better. GitLab's gemfile.lock https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Gemfile.... has over 1000 lines and GitLab uses a lot of memory. We try to be careful what we pull in but if anyone has suggestions which can be removed please let us know. Recently we found out that we still had to remove Redcloth as a dependency, it will be gone in GitLab 8.5.

Re: Kill Your Dependencies

#118

Post by geodel »

davexunit wrote: post #59
greggman wrote: post #36

This reminds me of an example I ran into yesterday. I haven't used webpack yet but I saw a question on SO of someone wanting to use some package called glslify. I thought I'd take a look and maybe learn webpage in the process. From the description all glslify does is look for files with the extensions .glsl, .frag, and .vert and lets you get their contents with `content = require(filename)`. Sounds like it would be a…

NodeJS developers ought to be embarrassed at how absurdly huge their dependency trees are.

I think they are a proud bunch.

Re: Kill Your Dependencies

#119

Post by alexose »

BinaryIdiot wrote: post #25

This is a great read that can be applied to node.js very much. I've seen apps that include 10, maybe 20 dependencies but when you flatten out the full dependency tree? Thousands. It's incredible and if one of those dependencies screws up semantic versioning or just screws up in general it can be a nightmare to debug and fix. This is why every 1.0 product I work on I include every dependency that speeds up my developm…

The worst offender, IMO, is request. I've seen more than a few projects pull it in just to make a single HTTP call. Just look at its package.json:

https://github.com/request/request/blob/master/package.json

Re: Kill Your Dependencies

#120

Post by JBReefer »

davexunit wrote: post #59
greggman wrote: post #36

This reminds me of an example I ran into yesterday. I haven't used webpack yet but I saw a question on SO of someone wanting to use some package called glslify. I thought I'd take a look and maybe learn webpage in the process. From the description all glslify does is look for files with the extensions .glsl, .frag, and .vert and lets you get their contents with `content = require(filename)`. Sounds like it would be a…

NodeJS developers ought to be embarrassed at how absurdly huge their dependency trees are.

They're web scale
Post reply on HN