There is a counter example of his reasoning in the Python world. There is a HTTP client library "urllib" in the standard library, but nowadays everyone rather pulls in the external dependency "requests" because the urllib API is terrible.
It is mature, well tested, good documented code though.
Kill Your Dependencies
141–150 of 229 posts
Re: Kill Your Dependencies
#142Earlier quoted context omitted.
This sadly also happens in some Linux repositories that add too many dependencies to a few key packages. On NixOS, last time I tried, installing mutt ended up bringing python as well.
In Nix, it's very easy to make a minimal mutt variant by removing the python input.
Yes, but these things shouldn't be happening. Default builds of packages should be kept more minimal.
I understand it's hard with Nix philosophy, and things are improving with different package outputs. For the record, python was pulled indirectly via the gnupg dependency I think.
Re: Kill Your Dependencies
#143if you're running multiple rails processes on a server like this, couldn't you somehow do the initialization in one process, then fork off the new processes? wouldn't that prevent the base libraries from being copied in memory?
Yes, http://unicorn.bogomips.org/ popularized this for ruby / rack / rails with its forking model and preload_app option. http://puma.io/ does the same thing, but additionally runs multiple threads in each process.
The garbage collector in Ruby 1.8 / 1.9 negated the benefits of copy-on-write forking, but that's fixed since Ruby 2.0
Re: Kill Your Dependencies
#144This is a huge mistake if applied without care. Building things from scratch necessarily will introduce more bugs, more maintenance costs and leave you with a codebase that suffers from a lack of maturity.
Building things from scratch necessarily will introduce more bugs, more maintenance costs and leave you with a codebase that suffers from a lack of maturity.
Unfortunately, in some programming language ecosystems where having many small and transitive dependencies on modules from an non-curated repository is common, none of those three things is necessarily true.
Code reuse is not a trivial problem, and you always have to weigh the benefits against the costs and risks to decide whether it’s worth it. If we’re depending on GitHub repositories with a dozen files and three subdirectories just to provide some simple functionality that any junior programmer could implement directly in five lines of code, we’ve probably lost the plot. On the other hand, if we have a full in-house implementation of encryption algorithms we use to throw sensitive customer data around between the browser and our servers, we’ve also probably lost the plot.
Re: Kill Your Dependencies
#145Avoid shims. There are lots of libraries that just put one interface on top of another interface. They don't do much actual work. Pulling in shims, especially if they pull in lots of other stuff you're not using, should be avoided. If the dependency does real work you'd otherwise have to code, then use it.
cough Mongoose. Been moving away from it on my projects. While it does provide a nice interface, it just creates more work down the road.
Re: Kill Your Dependencies
#146Earlier quoted context omitted.
Balance is good. Unfortunately ideologues have taken over software as they have taken over politics.
Are there really people out there whose decisionmaking process goes like "well we don't really need this library, but I'm gonna depend on it anyway to further the ideology of our movement"
I know plenty of people who want to port everything to JavaScript. When you ask "Why?" they don't know. That's an ideology.
Re: Kill Your Dependencies
#147Earlier quoted context omitted.
Charming quote. The same argument could be made about making your own food "from scratch." But it's not a solid refutation of the essay. Let's say that if one end of the spectrum is unlimited dependencies and complete indifference to the complexity and size of the project, and the other end of the spectrum is raising your own goats, there must be an ideal somewhere in the middle.
I completely agree, but the extremism in the essay is ridiculous. If I'm writing some software, I don't want to have to roll my own version of every little thing when there are battle tested libraries out there that already do it and benefit from many people using it. If my use case is vastly different, sure, but if it's the same use case as everyone else then I see little benefit.
Have you read a modest proposal by J Swift. Satire at its best! https://en.wikipedia.org/wiki/A_Modest_Proposal
Re: Kill Your Dependencies
#148Earlier quoted context omitted.
Not always. Dependencies were a huge problem at Google, even in C++ (perhaps especially in C++), because they mean that the linker has to do a lot of extra work. And unlike compiling, linking can't be parallelized, since it has to produce one binary. At one point the webserver for Google Search grew big enough that the linker started running out of RAM on build machines, and then we had a big problem. There's still n…
Wow. Was that with or without link-time optimization?
It was debug builds, which put an additional strain on the linker in that they keep around all the debug symbols. Regular builds were slow but manageable, so it's not like we couldn't release or develop, but it meant tracking down any sort of crash or serious bug became very difficult until we got the dependencies under control.
I forget the exact compiler settings - wasn't my department - but I think it included link-time optimization, and also FDO.
Re: Kill Your Dependencies
#149I disagree, and this quote I've seen floating around the internet sort of sums the idea up to me (albeit with a music analogy): > I thought using loops was cheating, so I programmed my own using samples. I then thought using samples was cheating, so I recorded real drums. I then thought that programming it was cheating, so I learned to play drums for real. I then thought using bought drums was cheating, so I learned…
I don't think that's what he's advocating: he's just saying that every dependency you have is another thing you have to worry about, so why not try to limit them as much as possible? Obviously it's impractical sometimes, and that's OK, as long as you understand the consequences.
Re: Kill Your Dependencies
#150Earlier quoted context omitted.
Not always. Dependencies were a huge problem at Google, even in C++ (perhaps especially in C++), because they mean that the linker has to do a lot of extra work. And unlike compiling, linking can't be parallelized, since it has to produce one binary. At one point the webserver for Google Search grew big enough that the linker started running out of RAM on build machines, and then we had a big problem. There's still n…
Was it running out of memory because of templates? For instance, parts of boost like boost serialization generates an obscene amount of symbols due to the way they do metaprogramming.
Templates were a problem but not a huge one. They aren't used extensively in the webserver, and in any case they bloat compile-time moreso than link-time.
The bigger problem was that we'd adopted a dependency strategy of "lots of little libraries" instead of "one big library with lots of source files". This offloads a lot of the work from the compiler to the linker. There are various advantages of this strategy - it speeds up incremental rebuilds, it encourages you to explicitly track all your dependencies, it simplifies IWYU, it's easier to parallelize - but linker RAM usage is not one of these advantages.