Live data from Hacker News

Tripping over the potholes in too many libraries

rachelbythebay.com

21–30 of 57 posts

Re: Tripping over the potholes in too many libraries

#21

Earlier quoted context omitted.

You also have a much easier route to fixing issues, since there will be far less politics involved. Probably you will also have a deeper understanding of the code since you wrote it.

If you find the fix yourself you don't need to deal with politics if you don't want to. Submitting pull requests is optional after all.

If you don't submit a pull request and get your fix merged, essentially you have given up on the benefits of an open source library maintained by other people. You will have to maintain your fix. At that point, you might as well have written the thing yourself, especially for trivial stuff.

Re: Tripping over the potholes in too many libraries

#22
post #9

From personal experience: it doesn't take a FAANG type of scale to reveal those "potholes". For most open-source projects (and especially things like pip, npm and other infrastructure and build tools) a simple, almost classical enterprise outbound proxy with authentication and MitM-style HTTPS-reencryption is more than enough kill almost every assumption they have in their code. In my case, that proxy tended to deliv…

Could you explain a bit more? I think most pkg repos are just static directories. If you do https reencryption, and https certs do not match EVERYTHING will break.

Yes, it will. Normally, the IT department would roll-out their MitM CA certificates globally into Windows trusted certificates store or something like it. However, if you are trying to do CI/CD on a Linux server, you have to manually provide those CA certificates to the tools you are using. Many tools are developed in a kind of "HTTPS everywhere" bubble, so that they think HTTPS just works and don't provide any kind of remedy for broken, outdated or plain internal certificates. Then you start working around it and it feels like it never ends....

Re: Tripping over the potholes in too many libraries

#23
post #4

I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…

I think the answer is to not import the dependency at all.

If you absolutely can't write a single function, then copy/pasting it from a set of "known good" functions would be better than importing a dependency.

Of course, just writing the function would be better. It would probably take less time than discovering the package, working out the interface for it, discovering that it doesn't actually work for your use case, discovering another, better package, etc.

Re: Tripping over the potholes in too many libraries

#24
This is so bizarre, as if fixing a config write to be atomic would somehow fix the root issue of this program was not designed with a config store in mind that can tolerate multiple instances of itself. Because of course it doesn't. Because having multiple instances in parallel was a novel requirement you introduced that the original author obviously didn't care for.

This isn't a learning experience about flock at all.

Re: Tripping over the potholes in too many libraries

#25
post #3

This comes up all the time and I never understand this attitude. Yes: dependencies are bad. Not having dependencies and writing everything yourself: also bad. Honestly, you have to rely on heuristics in your deps. How active is the project? How simple is the thing it's doing (simple enough to probably not have major bugs, but not so simple it's faster to code it yourself) etc. You get so much velocity from depending…

But once they've done it your junior devs will know how to write a config handling library. And they'll have hit a few bugs so they'll know about file locking, and concurrency problems, and overwriting, etc. It will be a great learning project.

If you teach them to just import a dependency, then that's all they know. They'll never be able to write a concurrent library, they'll just know how to import one.

Having lots of velocity is bad if there are too many potholes in the road. Slow is smooth, smooth is quick.

Re: Tripping over the potholes in too many libraries

#26
I guess the conundrum is: for 80% of people, maybe an 80% library is good enough; and 80% of library vendors are happy to write libraries that are 80% good enough - and 80% of customers have learnt to put up with stuff that occasionally doesn't work?

To me, the best solution is a language ecosystem that distinguishes itself by having properly written standard libraries to start with - whether built in, dynamically linked or optionally included in your project one way or another. For example, key=value config file parsing and saving should absolutely not need a third-party library. That should be as much a selling point of your language ecosystem than having a package manager that can pull directly from github.

One of the few upsides to "enterprise Java" is that there's a lot of "enterprise level" libraries around for it; even a lot of the open source ones are maintained by organisations like the Apache foundation rather than some random person on the internet. At one place I used to work, we had a whitelist of approved libraries and vendors, and even if you could in theory just import a library from the web, you had to make a case to your manager and get someone to sign it off if you wanted to use something not on the list. (As a side-effect, they'd also check the licence conditions, as we were writing closed-source commercial stuff. As a young intern, I got my first lesson in what (L)GPL meant when I requested to include something in one of our projects.)

Re: Tripping over the potholes in too many libraries

#27
post #14

> My guess was that people get into these situations where it seems like a library is going to be a solid "100% solution", and yet it lets you down and maybe reaches the 80% mark. That's probably true. I think a difficulty in avoiding this is that often the person reaching for this library, if they tried to write it themselves, would only hit the 60% mark. And I don't think there's anything wrong with that. There are…

I'm always amazed by this attitude. That somehow the authors of random libraries on npm are amazing coders whose code quality can't be touched by mere mortals like us.

Firstly, if you've always imported a dependency to get around a problem, then yes it's going to be hard to solve that problem yourself. But it's also a learning experience. Keep doing it and eventually it won't be hard.

Secondly, the library probably isn't a perfect match for your use case, and probably contains a lot of flexibility to match a wide range of use cases. Maybe as much as 80% of the code isn't actually any use to your project. It'll be more complex code, because it's dealing with a wider set of use cases. The thing you'd write to solve your particular use case would be smaller and cleaner almost by definition. You'd end up with less code, that you understand well. This is always a better position to be in.

Re: Tripping over the potholes in too many libraries

#28
post #4

I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…

> possible solution for a small portion of this problem

In CS research, this is known as "software components", and people have been worrying about it since at least the 1970s with very slow progress. Here's Stoustroup writing about it in the 90s, for example: https://www.semanticscholar.org/paper/Language-technical-asp...

"Reuse happens only when a variety of social conditions are favorable. However, social conditions, development processes, and design methods alone cannot guarantee success. In the end, working code must be produced"

The dream used to be that software could be like electronics: standard elements like resistors would have a small number of well-characterised parameters, and you can just pick them from a catalog. Large components would come from vendors with large datasheets that would also describe their performance, and would be guaranteed a certain period of availability such as ten years.

The end result is not quite like that. A lot of components have made it into standard libraries (what python calls "batteries included"). Everything else is available on repositories. However, because everything has to be free (and sometimes Free), that also selects for the absolute minimum maintenance effort and quality. Freedom has many huge advantages, as does not having to get purchase order or "BOM" approval every time you add a dependency to your project, but it does leave people in the "if it breaks you get to keep both pieces" situation.

Re: Tripping over the potholes in too many libraries

#29
post #4

I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…

Horrible idea, if a package get corrupted by an adversary not only would you have to clean the npm package but also hope everyone who committed the code into a repository gets the news and fixes it. At least with NPM there is the chance that a fixed version gets automatically pulled on the next build.

In addition to that, most package managers rely on metadata, like description files, the bloat introduced by these micro-packages is unfathomable.

Then there is the obvious answer to all this, use a proper language, learn it and don't pull in stuff like "left-pad" from external sources. It's a one liner. If you can't come up with that yourself get the fuck out of this industry. Seriously.

Re: Tripping over the potholes in too many libraries

#30
Rachel seems to trip over the concept of Division of Labor[1] to a point. There is arguably more net gain from not having everyone reinvet OpenSSL just to communicate.

Or maybe she's more against being indiscriminate about re-use.

But there just aren't many companies big enough to engineer their own complete solutions.

[1] https://en.m.wikipedia.org/wiki/Division_of_labour

Post reply on HN