Leaky Abstractions

These are the kinds of articles I love to see on HN!

I think it's so interesting to see how different OS' have approached these long lived operations over the years. Arguably, *nix OS' have been able to move past these problems since it appears that package management is much more of a first party concern. "Oh, you want to upgrade this utility? Well, I rely on these other things so you should update them too."

I'm a webdev so a lot of this is foreign to me w/ how it actually all functions but it's cool to read about these problems and speculate how/why they came to be and why they may only exist on certain platforms.

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files".

I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

This kind of thing doesn't surprise me at all - a surprising number of developers miss those types of things, and one of the things a lot of people miss seems to be checking for situations where you end up with excessive amount of calls doing little work, for some reason.

E.g. at one point (long time ago), MySQL's C client library would call read() for 4 bytes to read a length indicator, and then read exactly the number of bytes indicated by the protocol, which led to a lot of time spent context-switching vs. reading into a larger user-space buffer and copying out of that.

On Linux simple, plain "strace" remains one of my favourite first steps to find performance problems in part because these things are near immediately apparent if you strace a process because the really pathological cases often ends up dominating the output so much that you'll spot them right away.

Another "favourite" issue that shows up often when you use strace like this is e.g. excessive include paths - running strace on MRI Ruby with rubygems enables and lots of gems pulled in is a good way of seeing that in action - like this zip problem it's an example that seems totally reasonable when the number of gems is small, and that first becomes apparent when you test with lots of gems, and look at what's actually happening under the hood.

A similar example of testing with too small datasets and/or on a large enough machine to not spot what becomes immediately obvious if you trace execution was how a type 1 font reference library made available by Adobe (no idea if they wrote it or if it was from another source originally) which would exhibit another typical pathological behaviour and call malloc() hundreds of times on loading a font to allocate 4 bytes at the time instead of allocating larger buffers. (strace won't catch malloc(), but ltrace does)) To avoid messing too much with the code, we replaced the calls to malloc() with calls to a simple arena allocator and load speed shot through the roof and memory usage dropped massively.

Too few people trace execution of their code. I know that because if most developers did, issues like the above would get caught much sooner, and would be rarer in published code.

I find it's kinda hell moving files from both my android AND iPhone ... it feels like a 20 year old process in terms of it working in fits and starts and random failures.