Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Leaky Abstractions

textslashplain.com

Post reply on HN
Print view View original

11–20 of 119 posts

Re: Leaky Abstractions

#11

Post by lmilcin »

bentcorner wrote: post #4

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on.

As to slow operations... that is more likely because of synchronous implementation.

The popular, naive implementation is, as above, to repeat same simple operation over and over again: read from source, write to destination, read from source, write to destination.

A better implementation (what I would do) would be to pipeline operations. Basic pipeline would have three components, each streaming data to and/or from buffer

1: Read from source to pipeline buffer

2: Read from pipeline buffer to write to destination, write information about files to delete to another pipeline buffer

3: Read files to delete from pipeline buffer and execute deletions.

Using *nix shell you could do something like that in single line

1: tar -c files to output

2: pipe output to tar -xv, write files to destination producing list of written files, pipe written files to output

3: read piped list of written files and remove them from input dir

Now, this is not perfect because we are wasting performance on creating tar file when we immediately discard it, but you get the picture.

Re: Leaky Abstractions

#12

Post by tetha »

lmilcin wrote: post #11
bentcorner wrote: post #4

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. As to slow operations... that is more likely because of synchronous implementation. The popular, naive implementation is, as above, to repeat same simple operation over and over again: r…

> It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on.

Litte bit of a rant, but I see this SO MUCH in database layers in applications. Implement an operation for one row, slap a for loop around it, it works kinda quickly on the small test data set... and then prod has an intimate conversation with a brick wall. It has been 0 days at work since that happened.

> As to slow operations... that is more likely because of synchronous implementation.

Interestingly, I think the answer is a solid maybe and depends on the storage and how you issue your i-o operations. A flash storage will increase performance if you increase parallel operations, up to a point. However - and this code apparently was written 20 years ago - on spinning drives, parallel io-operations slow you down if the OS does not merge those operations. So it's entirely not obvious.

Re: Leaky Abstractions

#13

Post by slver »

bentcorner wrote: post #4

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

In theory all software is coded using a "many by default" approach. So every time batching matters, we take those batching opportunities automatically due to the way software is coded.

In practice we only batch when it starts hurting. It doesn't hurt to delete files one by one on a normal file system. It's made for that. So the API wasn't "many by default" and that's how it works for zip files as well.

Re: Leaky Abstractions

#15

Post by lmilcin »

tetha wrote: post #12
lmilcin wrote: post #11

Earlier quoted context omitted.

It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. As to slow operations... that is more likely because of synchronous implementation. The popular, naive implementation is, as above, to repeat same simple operation over and over again: r…

> It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. Litte bit of a rant, but I see this SO MUCH in database layers in applications. Implement an operation for one row, slap a for loop around it, it works kinda quickly on the small test…

On single drive you run a variation where you read X MB of data to a buffer, then write X MB of data out, then execute deletes, and so on. This lets avoid some of the problems with small files. Not all, because small files will unlikely to be consecutive, the head will have to jump a lot, and then you still need to do a lot of small writes to filesystem (to remove the files).

There are obviously improvements you could do. For some filesystem you can just remove entire folders rather than remove the files individually just to remove parent folder.

Re: Leaky Abstractions

#16

Post by TeMPOraL »

lmilcin wrote: post #11
bentcorner wrote: post #4

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. As to slow operations... that is more likely because of synchronous implementation. The popular, naive implementation is, as above, to repeat same simple operation over and over again: r…

> The popular, naive implementation is, as above, to repeat same simple operation over and over again: read from source, write to destination, read from source, write to destination.

Reminds me of the kind of patterns functional programming languages introduce, where you process data by describing operations on individual items and assembling them into "a stream". I'm always wary of those - without a good implementation and some heavy magic at the language level, they tend to become the kind of context-switching performance disaster you describe.

Re: Leaky Abstractions

#17

Post by jakub_g »

genpfault wrote: post #5
bentcorner wrote: post #4

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

> doing similar operations to a connected android phone MTP is terrible[1]. [1]: https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Perfor...

Oh boy. I was thinking USB 2.0 is main reason for AndroidWindows copying of photos to suck so much, but the rabbit hole is much deeper.

It's sad that with cloud being the solution for everything those days, this will probably never be improved within next decade.

Re: Leaky Abstractions

#18

Post by lmilcin »

TeMPOraL wrote: post #16
lmilcin wrote: post #11

Earlier quoted context omitted.

It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. As to slow operations... that is more likely because of synchronous implementation. The popular, naive implementation is, as above, to repeat same simple operation over and over again: r…

> The popular, naive implementation is, as above, to repeat same simple operation over and over again: read from source, write to destination, read from source, write to destination. Reminds me of the kind of patterns functional programming languages introduce, where you process data by describing operations on individual items and assembling them into "a stream". I'm always wary of those - without a good implementat…

Yes. Functional world is not impervious to leaky abstractions.

I am personally of the opinion that, to be a good developer, you have to have mental model of what happens beneath. If you are programming in a high level language it is easy to try forget about the fact that your program runs on real hardware.

I know, because I work mostly on Java projects and trying to talk to Java developers about real hardware is useless.

I have an interview question where I ask "what prevents one process from dereferencing a pointer written out by another process on the same machine" and I get all sorts of funny answers and only 5-10% candidates even have beginning of understanding what is going on. Most don't know what virtual memory is or are surprised that two processes can resolve different values under same pointer.

Re: Leaky Abstractions

#19

Post by Sol- »

Interesting, didn't even know you can cut files from a zip in that Windows zip file viewer. I would have thought it's some read-only filesystem like viewing a mounted CD or so.

In that context I wonder if you could cut from rewritable CD-RWs as well back in the day (can't remember) - that seems like another abstraction that's similarly slow in reality.

Re: Leaky Abstractions

#20

Post by jakub_g »

Sol- wrote: post #19

Interesting, didn't even know you can cut files from a zip in that Windows zip file viewer. I would have thought it's some read-only filesystem like viewing a mounted CD or so. In that context I wonder if you could cut from rewritable CD-RWs as well back in the day (can't remember) - that seems like another abstraction that's similarly slow in reality.

I never tried cutting from CD-RW, but AFAIR each burning would append a non-trivial header (like 20MBs or so) so that would be a pretty expensive thing to do :)

AFAIR when you "copied" into CD-RW the files would show up semi-transparent (pending) and you'd have to click a button to process with burning. Probably same for cutting I guess.

Post reply on HN