Live data from Hacker News

Tripping over the potholes in too many libraries

rachelbythebay.com

41–50 of 57 posts

Re: Tripping over the potholes in too many libraries

#41
I've run into an application, used for monitoring, that had that exact type of bug, albeit not with a dot file.

A customer of my old business had built a little monitoring system for their compute nodes mounting a parallel file system. Their integrated test had every compute node open a particular fixed path and file name (of course the same on every node across the system) for read and write.

This "monitoring" script meant that they could have up to 2000 or so simultaneous IOs going to the same file, with no read/write locking. The tool read/wrote some number of bytes to get a performance read.

The end result was 1) lots of contention of course at the metadata layer, 2) often times spurious and incorrect reports of the parallel file system being offline (it wasn't).

We tried helping them on this, but they insisted they were doing this correctly (they weren't).

This is less about libraries with potholes per se, and more about critical applications (to a degree similar to libraries providing critical functions that need to be correct in semantics, implementation, and in error generation) that are broken due to a misdesign/mis-implementation somewhere.

With regard to her commentary on CPAN, one of the more annoying things I've dealt with in many libraries is their choices of error return. Some will throw exceptions. Some will return real errors you can process in-line. I am not a fan of throwing exceptions, and when I build larger Perl apps, I tend to insert some boilerplate exception handlers specifically due to the burns I've encountered in the past when modules do dumb things.

Re: Tripping over the potholes in too many libraries

#42
post #9

From personal experience: it doesn't take a FAANG type of scale to reveal those "potholes". For most open-source projects (and especially things like pip, npm and other infrastructure and build tools) a simple, almost classical enterprise outbound proxy with authentication and MitM-style HTTPS-reencryption is more than enough kill almost every assumption they have in their code. In my case, that proxy tended to deliv…

An unsatisfying solution is to surrender and try to be as vanilla as possible in everything. With a few carefully thought out exceptions as needed by your business.

Re: Tripping over the potholes in too many libraries

#43
post #4

I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…

I think the answer is to not import the dependency at all. If you absolutely can't write a single function, then copy/pasting it from a set of "known good" functions would be better than importing a dependency. Of course, just writing the function would be better. It would probably take less time than discovering the package, working out the interface for it, discovering that it doesn't actually work for your use cas…

I agree, but I would take it a step further.

If you think it’s easier to find and import a library to test if a number is even than to use the modulus operator in an if clause, should you really be responsible for writing any code at all?

I feel like JavaScript in particular has developed this ecosystem of inexperienced programmers that “don’t know what they don’t know”. I don’t want to discourage anyone, but I feel like other languages (particularly more mature ones) don’t have that problem to the same degree.

For instance, I feel like the cluster module in Node is far easier for people to do things they shouldn’t with than any of the equivalents in C# or Java or even PHP (so it’s not just compiled vs scripting languages).

Re: Tripping over the potholes in too many libraries

#44
post #3

This comes up all the time and I never understand this attitude. Yes: dependencies are bad. Not having dependencies and writing everything yourself: also bad. Honestly, you have to rely on heuristics in your deps. How active is the project? How simple is the thing it's doing (simple enough to probably not have major bugs, but not so simple it's faster to code it yourself) etc. You get so much velocity from depending…

But once they've done it your junior devs will know how to write a config handling library. And they'll have hit a few bugs so they'll know about file locking, and concurrency problems, and overwriting, etc. It will be a great learning project. If you teach them to just import a dependency, then that's all they know. They'll never be able to write a concurrent library, they'll just know how to import one. Having lots…

> But once they've done it your junior devs will know how to write a config handling library

This is true, but there are always so many things that need building. It's kind of like needing an end table, and either buying one from Target or building it yourself in your garage. End tables aren't that hard to build! The end table at Target is a piece of crap! You'll learn a lot trying to build your own end table! These things are all true, but the question that you have to ask before all that is: do I want to spend my limited time building an end table?

> If you teach them to just import a dependency, then that's all they know. They'll never be able to write a concurrent library, they'll just know how to import one.

This is probably true of some developers, but I don't think I've run into a developer that doesn't love building things themselves. (Possibly selection bias). The tendency coders have is to code. They're going to write something from scratch (and learn a lot) whether you tell them to or not.

> Having lots of velocity is bad if there are too many potholes in the road. Slow is smooth, smooth is quick.

I agree with this principle. The way I think of it though is that it's not a road you're on: the asphalt stretches out in all directions, and you don't know which direction you'll be going next. It's very often the case that you spend a bunch of time smoothing over a single pothole to perfection, and crash into a another one right next to it. It is probably correct to shittily half-fill all the potholes around you just to avoid catastrophe and come back later to the ones you run over again and again.

Re: Tripping over the potholes in too many libraries

#45
post #19
post #15

I think the generally accepted fix here (despite Rachel’s aversion) is to submit a PR to the file writing library that fixes the corruption issue (likely using atomic rename), then get the tool to bump the version of their dep or vendor in the fixed version. I’ll admit, though, that the balkanization of code adds overhead from the abstraction. I just don’t think it’s a bad thing, because it’s all very new and things…

It is likely that fix would break somebody else's code which unwittingly depends on the bug. Then, burden of educating the users would fall on maintainers. Who, most likely, aren't having any of that. (Author even linked a article about this.)

How could you depend on a race condition bug?

Re: Tripping over the potholes in too many libraries

#46
post #4

I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…

I think the answer is to not import the dependency at all. If you absolutely can't write a single function, then copy/pasting it from a set of "known good" functions would be better than importing a dependency. Of course, just writing the function would be better. It would probably take less time than discovering the package, working out the interface for it, discovering that it doesn't actually work for your use cas…

I think the catch is that "microlibraries" can still contain reasonably large amounts of functionality that's non-trivial to replicate.

I'd be happy enough to import single-function dependencies for "urlencode" or "is_valid_email" rather than manually grow out all the edge cases (again).

Re: Tripping over the potholes in too many libraries

#47
post #4

I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…

I think the answer is to not import the dependency at all. If you absolutely can't write a single function, then copy/pasting it from a set of "known good" functions would be better than importing a dependency. Of course, just writing the function would be better. It would probably take less time than discovering the package, working out the interface for it, discovering that it doesn't actually work for your use cas…

Many useful libraries cannot be easily re-created. Consider datetime handling and X.509 parsing.

Re: Tripping over the potholes in too many libraries

#48
post #4

I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…

I really like this idea.

When reviewing changes of the imported code, one will need the original change sets with comments. Therefore the system will need a format for representing these diffs. I don't think git is the right answer.

After review, the tool could upload a "passed review" signature to the central repository. Folks can see which version has been reviewed and approved by which organizations. This would let small organizations benefit from the review work done by large organizations. For example, a startup founder can feel more confident using a library version that has passed review by several of the FAANG companies.

Re: Tripping over the potholes in too many libraries

#49
post #19

Earlier quoted context omitted.

It is likely that fix would break somebody else's code which unwittingly depends on the bug. Then, burden of educating the users would fall on maintainers. Who, most likely, aren't having any of that. (Author even linked a article about this.)

How could you depend on a race condition bug?

It's not so hard to imagine a scenario where system happens to depend on both data concurrently written into a config file by 2 processes. If the file is properly written and renamed then data from one process gets lost, and/or if there is locking, one process will suddenly stall.

Re: Tripping over the potholes in too many libraries

#50
post #44

Earlier quoted context omitted.

But once they've done it your junior devs will know how to write a config handling library. And they'll have hit a few bugs so they'll know about file locking, and concurrency problems, and overwriting, etc. It will be a great learning project. If you teach them to just import a dependency, then that's all they know. They'll never be able to write a concurrent library, they'll just know how to import one. Having lots…

> But once they've done it your junior devs will know how to write a config handling library This is true, but there are always so many things that need building. It's kind of like needing an end table, and either buying one from Target or building it yourself in your garage. End tables aren't that hard to build! The end table at Target is a piece of crap! You'll learn a lot trying to build your own end table! These…

> It's kind of like needing an end table, and either buying one from Target or building it yourself in your garage.

Except that, in this analogy, you're a carpenter. Building furniture is your profession, and hopefully something you enjoy doing. Learning how to build end tables quickly is a useful skill to have in your profession. Practising that skill by building an end table for yourself is not a waste of time.

> It is probably correct to shittily half-fill all the potholes around you just to avoid catastrophe and come back later to the ones you run over again and again.

Indeed, but that's another skill :) Knowing when to accumulate tech debt because you're not sure if the feature will be needed is important. And again, it's a skill one acquires with practice.

Post reply on HN