Live data from Hacker News

Leaky Abstractions

textslashplain.com

21–30 of 119 posts

Re: Leaky Abstractions

#21
post #20
post #19

Interesting, didn't even know you can cut files from a zip in that Windows zip file viewer. I would have thought it's some read-only filesystem like viewing a mounted CD or so. In that context I wonder if you could cut from rewritable CD-RWs as well back in the day (can't remember) - that seems like another abstraction that's similarly slow in reality.

I never tried cutting from CD-RW, but AFAIR each burning would append a non-trivial header (like 20MBs or so) so that would be a pretty expensive thing to do :) AFAIR when you "copied" into CD-RW the files would show up semi-transparent (pending) and you'd have to click a button to process with burning. Probably same for cutting I guess.

Been years, but the ISO file system says 16 sectors past the start of the last track is where the file system index starts. (the first 15 sectors are reserved for booting) Then however many sectors you need for the index - which depends on how many files you have, and how long the filenames are. Note that both CDR and CDRW don't allow you to write sectors in the middle of a track, so you need to know where every file will be on disk before you can write anything - you can't build the index on the fly. (I remember this part because I was writing backups and so I didn't know how big the file would be until I was done and thus couldn't write the index first - I eventually got around the rule by creating a new track with just the index)

Re: Leaky Abstractions

#23

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

copy + delete one at a time makes a lot of sense if you're working on a filesystem without a way to move without copying (I don't think you can actually move a file in fat32), because copy all could require more space than is available.

The same could be true here where you're moving from a zip file to probably the same filesystem the zip files is in; if removing a file from the zip file is actually an in-place move data then truncate. The problem, of course, is that removing a file from the zip file is tremendously expensive. Reading the file with one syscall per byte doesn't help (especially post-Spectre workarounds that make syscalls more expensive).

Re: Leaky Abstractions

#24

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

It's so if the move fails midway you won't need to start from scratch.

Re: Leaky Abstractions

#25
post #22

Unless I'm mistaken, this seems to be the original programmer on youtube: Dave's Garage - Secret History of Windows ZIPFolders https://youtu.be/aQUtUQ_L8Yk

It is. I have seen a few of his videos before. They are generally interesting and worth checking out.

Re: Leaky Abstractions

#26
post #23

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

copy + delete one at a time makes a lot of sense if you're working on a filesystem without a way to move without copying (I don't think you can actually move a file in fat32), because copy all could require more space than is available. The same could be true here where you're moving from a zip file to probably the same filesystem the zip files is in; if removing a file from the zip file is actually an in-place move…

This is the real answer IMO. For example, if you have a 2TB drive with 50GB available space left and are trying to move 1TB of data, and the move is requiring copying, but all individual files on their own are less than 50GB in size, then I’d be pretty upset if my computer was unable to move the files just because it was wanting to not delete anything until the end.

But ideally I’d want the system to delete at the end if possible, and to otherwise delete as needed, instead of either doing only all at end or only after every single file.

Re: Leaky Abstractions

#27
"Unfortunately, the code hasn’t really been updated in a while. A long while. The timestamp in the module claims it was last updated on Valentine’s Day 1998"

When I saw this timestamp sometime ago on my PC I thought it was a joke?! C'mon 1998 like WTF!

Re: Leaky Abstractions

#28
post #20
post #19

Interesting, didn't even know you can cut files from a zip in that Windows zip file viewer. I would have thought it's some read-only filesystem like viewing a mounted CD or so. In that context I wonder if you could cut from rewritable CD-RWs as well back in the day (can't remember) - that seems like another abstraction that's similarly slow in reality.

I never tried cutting from CD-RW, but AFAIR each burning would append a non-trivial header (like 20MBs or so) so that would be a pretty expensive thing to do :) AFAIR when you "copied" into CD-RW the files would show up semi-transparent (pending) and you'd have to click a button to process with burning. Probably same for cutting I guess.

Speaking of old Windows and CDs, Windows had some crazy trick to turn CD-R (not CD-RW) into rewriteable medium. I'm guessing they simulated a regular file system on top of an append-only representation. I never dug into the details back in the day, because I never used this feature. Unfortunately, IIRC, this trickery was enabled by default when copying files to CDs - which was a problem, because nothing else could read it. This caused me an unending stream of calls from friends and relatives, who all tried to burn some family/vacation photos onto a CD-R, in order to view them on their TVs, but the CD readers for TVs couldn't parse the format.

Re: Leaky Abstractions

#29
post #6

This kind of thing doesn't surprise me at all - a surprising number of developers miss those types of things, and one of the things a lot of people miss seems to be checking for situations where you end up with excessive amount of calls doing little work, for some reason. E.g. at one point (long time ago), MySQL's C client library would call read() for 4 bytes to read a length indicator, and then read exactly the num…

> Another "favourite" issue that shows up often when you use strace like this is e.g. excessive include paths - running strace on MRI Ruby with rubygems enables and lots of gems pulled in is a good way of seeing that in action - like this zip problem it's an example that seems totally reasonable when the number of gems is small, and that first becomes apparent when you test with lots of gems, and look at what's actually happening under the hood.

When you add e.g. Python's standard library as a ZIP to the Python search path, the thing will open/read/read/read/close that file approximately a gazillion times on startup. That's OK on Linux/Unix, where that is fairly cheap. Guess which OS doesn't like that pattern at all?

Re: Leaky Abstractions

#30

Another similar and good article: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

Another common leaky abstraction: floating-point numbers as an abstraction of the real numbers. It works nearly all the time, until you have to really know about numerical precision, or where a NaN came from.

https://www.johndcook.com/blog/2009/04/06/numbers-are-a-leak...

Post reply on HN