Tripping over the potholes in too many libraries
rachelbythebay.com
Tripping over the potholes in too many libraries
1–10 of 57 posts
Re: Tripping over the potholes in too many libraries
#2There are so many problems working in tandem here:
- Leaky abstractions which are treated/documented as if they're perfect. Read a file, do a thing, write a file. Except when running in parallel, things are never that simple.
- Lots of important projects ("base of Arch Linux" level important) still don't have public repos, an official site (or even page) with references, bug tracker, or any reasonable way of contacting the maintainers. How do these projects get updated? I guess they simply don't.
- Even the best feedback mechanisms are still awful. Figuring out whether something is effectively abandoned before spending 30+ minutes writing up a bug report could take longer than writing the bug report itself, encouraging writing the absolute minimum (which ends up being less than minimum half the time). Every project is also its own unique snowflake. They want different types and amounts of information, some won't touch an issue unless it's reproducible in the very latest commit (requiring hours of setup and lots of domain knowledge to compile the damn thing), and they all have a completely different process.
- Most tools are still skewed towards one maintainer per project. Take any random GitHub abandonware as an example. You can fork it with one click, but so have 20 other people. You can put up a web site for your fork, but search engines will rank the original web site (or just the GitHub repo) highest for years. You can rename it, but now you've just burned bridges with all the other forks and the original repo, in ways which would be difficult to repair. Now try to migrate all the issues, mailing list, forum, IRC channel or other communications. It's basically hopeless unless the original maintainer is completely on board.
- Most open source software solves only the author's immediate problem. But not a single user of the software has exactly the same problem to solve as the original author. So pretty much by definition no library will solve the problem you are dealing with.
Maybe one way out of this mess for any particular language might be if the standard library was meant to grow to take over popular code, possibly with a tiered approach. You could nominate a library, then it would go through reviews, extensive testing, the whole shebang, before being accepted into a low level "this might someday be part of the standard library in some form" tier. This ideally would make it possible for example to replace core libraries in steps, steadily pushing the old one out of core and pulling a better alternative in.
Re: Tripping over the potholes in too many libraries
#3This comes up all the time and I never understand this attitude.
Yes: dependencies are bad. Not having dependencies and writing everything yourself: also bad.
Honestly, you have to rely on heuristics in your deps. How active is the project? How simple is the thing it's doing (simple enough to probably not have major bugs, but not so simple it's faster to code it yourself) etc.
You get so much velocity from depending on external libs, it's straightforwardly bad not to take advantage of it. Are the junior devs at your company really going to write a better config handling library than the random one from github? Probably not. Maybe they'll both be bad code but the open source one has a better shot of someone else noticing it.
Re: Tripping over the potholes in too many libraries
#4I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well.
Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages (left-pad, is-even, math functions,...)
My proposal is to create a new package manager for the latter (or adapt existing ones) which handles micro-packages, typically scoped to 1 function, 1 file or a small set of files. The big difference with regular NPM is that these micropackages do not get downloaded into node_modules, but are downloaded inside the src directory and committed to source control. This means that when you add a micropackage to your project, its source code is pulled into the regular src folder and is subject to the same code review as the other code your team writes. When the micropackage is updated the changes are visible during code review as well (instead of just being an opaque version bump in package.json)
This process makes small dependencies more visible and has review builtin. It also solves the problem of having to rewrite the same logic in every project in a slightly different way.
Edit: I'm not currently working on this due to time constraints but if anyone wants to talk about this, hit me up.
Re: Tripping over the potholes in too many libraries
#5> I show up with a problem ("hey, this thing keeps getting corrupted because X and Y") and suddenly it's because I'm "from" G or FB or something and I "want unreasonable things" from their stuff. So, my request is invalid, thank you drive though.
No, the answer is they're trying to deliver impact for the customer and Rachel is instead asking them to invest time in a solution that doesn't bring them any closer to that. I'd imagine most of them would be fine with fixing the root cause, but for some fucking reason everyone from Google or FB feels the need to reinvent every wheel in order to peacock and show that their IQ is the highest.
With that said, at my "normal people" non-bourgeoisie company the 3P libraries are all converted to the internal build system. If there was a fuckup of this magnitude, someone would just create a branch and bump the version number with the fix. Problem solved.
Re: Tripping over the potholes in too many libraries
#6> I show up with a problem ("hey, this thing keeps getting corrupted because X and Y") and suddenly it's because I'm "from" G or FB or something and I "want unreasonable things" from their stuff. So, my request is invalid, thank you drive though. No, the answer is they're trying to deliver impact for the customer and Rachel is instead asking them to invest time in a solution that doesn't bring them any closer to that…
This is a pretty bad take. Rachel has said implicitly that there was no version with a fix available and explicitly that working with the maintainers to get a fix is often unreasonably difficult. This matches my own experience fairly well. So your ‘normal people’ solution won’t work.
You may also want to consider that there are good reasons for engineers and engineering management to be default suspicious of 3rdparty dependencies at large companies such as those you’ve listed. These reasons have nothing to do with peacocking or demonstrating high IQ (replicating things available elsewhere is not a way to demonstrate your intelligence, it turns out). They have much more to do with the high bar for security consciousness, unique need to deal with scale or low performance tolerance, or extreme organizational risk associated with being a company with a target on your back for every major hacker, researcher, regulator, journalist, or self-proclaimed watchdog — not to mention operating under several consent decrees already.
Re: Tripping over the potholes in too many libraries
#7I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…
CCAN (C Code Archive Network) http://ccodearchive.net/ follows a similar philosophy. CCAN contains various small, self contained snippet of C code are meant to be be downloaded in a local directory ("vendored") and used directly or with small modifications.
The most straightforward way to use CCAN is to
1) `git clone` it in a subdir;
2) create a `config.h`;
3) include whatever `lib.h` you want in your project.
Updating these micro dependencies means doing `git pull`.
All libraries come with tests and a somewhat standardized API design.
What is missing (compared to your aims) is a declarative way to lock a dependency to a specific version.
Re: Tripping over the potholes in too many libraries
#8I think a lot of comments here are missing the fact that one could reimplement the config file handling library and atomic file update library in less time than it took to track down the package responsible for the bug and root cause it.
I’ve found such issues are the common case for libraries written in languages that encourage tons of tiny dependencies.
Re: Tripping over the potholes in too many libraries
#9From personal experience: it doesn't take a FAANG type of scale to reveal those "potholes". For most open-source projects (and especially things like pip, npm and other infrastructure and build tools) a simple, almost classical enterprise outbound proxy with authentication and MitM-style HTTPS-reencryption is more than enough kill almost every assumption they have in their code. In my case, that proxy tended to deliver incomplete files at random, which makes you discover that there is no checksumming happening. Then you build a local package proxy to help with that and then you find out that a particular library doesn't re-evaluate proxy environment variables on redirect, so that a redirect from external site to internal does not work. I can't remember filing that many issues at open-source projects as that time and it's been a draining game of whack-o-mole. At the same time you are reading tutorials called "How I did something in three hours" and you consider whether after-hours drinking might have been a better career choice.
Sorry. Enough ranting. That problem is everywhere, not just in libraries. I'm just not sure the problems lies in the users of those libraries, it's not too easy to use those libraries, it's too easy to publish them with a generic sounding name, but solving only a particular use case.
Re: Tripping over the potholes in too many libraries
#10This comes up all the time and I never understand this attitude. Yes: dependencies are bad. Not having dependencies and writing everything yourself: also bad. Honestly, you have to rely on heuristics in your deps. How active is the project? How simple is the thing it's doing (simple enough to probably not have major bugs, but not so simple it's faster to code it yourself) etc. You get so much velocity from depending…
And by re-implementing a library, you're just importing dependencies into your code, with the difference that now you're responsible for bug fixes and security patches, without the opportunity to inherit performance improvements contributed by others.