Linux file write patterns: So you want to write to a file fast • Hacker News Board

Re: Linux file write patterns: So you want to write to a file fast

#21

Post by Niten » Thu, May 01, 2014, 3:50 PM UTC

asveikau wrote: post #6

Assuming all I/O failures are EINTR is really, really odd, as if to say disks never fill up or fail and sockets never disconnect. He does say: > in a real program you’d have to do real error handling instead of assertions, of course But somebody somewhere is reading this and thinking this is a "semantically correct pattern" (as it is introduced) and may just copy-paste it into their program. Especially when contrasti…

> But somebody somewhere is reading this and thinking this is a "semantically correct pattern" (as it is introduced) and may just copy-paste it into their program.

I dare say that would be their fault for blindly copying and pasting without taking the time to understand the context. (He even gives an explicit disclaimer!) Robust error handling would just be more noise to filter through for people actually reading the article, and I don't think it's the author's responsibility to childproof things for people who aren't.

Re: Linux file write patterns: So you want to write to a file fast

#22

Post by dekhn » Thu, May 01, 2014, 3:52 PM UTC

Phlarp wrote: post #18

dekhn wrote: post #15

Earlier quoted context omitted.

M B . Bytes. nobody quotes disk speeds in bits (or if they do I typically ignore them). I've frequently observed sustained 500MB/sec writes and reads on my cheap ($250) 250GB SSDs. One of my favorite instances was running out of RAM while assembling a gigapan in Hugin. I added a swap file on my SSD and continued- it ran over night with nearly 500MB/sec reads and writes more or less continuously, but the job finished…

I weep for the memory sectors that got re-written continuously for an entire night.

I used to, but given not a single one of my SSD drives (I have 4 deployed in my house) has so much as balked once in a year of continuous deployment, I am cautiously optimistic.

Re: Linux file write patterns: So you want to write to a file fast

#23

Post by mtdewcmu » Thu, May 01, 2014, 3:54 PM UTC

It's a little bit reassuring that there weren't any clear winners and losers. In a perfect world, the OS and hardware would figure out what your intent is and carry it out the fastest way possible, right? Ideally, you'd write the code the most convenient way and it would run the most performant way. Maybe the future is now.

Re: Linux file write patterns: So you want to write to a file fast

#24

Post by ef47d35620c1 » Thu, May 01, 2014, 4:14 PM UTC

dekhn wrote: post #22

Phlarp wrote: post #18

Earlier quoted context omitted.

I weep for the memory sectors that got re-written continuously for an entire night.

I used to, but given not a single one of my SSD drives (I have 4 deployed in my house) has so much as balked once in a year of continuous deployment, I am cautiously optimistic.

I've had very good reliability from my SSD drives as well. Some have been running almost continuously since 2009.

Re: Linux file write patterns: So you want to write to a file fast

#25

Post by masklinn » Thu, May 01, 2014, 4:42 PM UTC

maxhou wrote: post #19

masklinn wrote: post #17

Earlier quoted context omitted.

> Assuming all I/O failures are EINTR is really, really odd, as if to say disks never fill up or fail and sockets never disconnect. The point is to retry on EINTR and to abort completely in case of other IO failures. assert(errno == EINTR); continue; is equivalent to if (errno == EINTR) continue; abort(); > But somebody somewhere is reading this and thinking this is a "semantically correct pattern" (as it is introduc…

ENOSPC ?

Is something the author very specifically noted he does not care about in his examples a bit later:

> I don’t care about a “disk full” that I could catch and act on

Re: Linux file write patterns: So you want to write to a file fast

#26

Post by masklinn » Thu, May 01, 2014, 4:43 PM UTC

Phlarp wrote: post #18

dekhn wrote: post #15

Earlier quoted context omitted.

M B . Bytes. nobody quotes disk speeds in bits (or if they do I typically ignore them). I've frequently observed sustained 500MB/sec writes and reads on my cheap ($250) 250GB SSDs. One of my favorite instances was running out of RAM while assembling a gigapan in Hugin. I added a swap file on my SSD and continued- it ran over night with nearly 500MB/sec reads and writes more or less continuously, but the job finished…

I weep for the memory sectors that got re-written continuously for an entire night.

SSD controllers do write-leveling, the blocks a filesystem writes to is virtual and remapped (think VMEM)

Re: Linux file write patterns: So you want to write to a file fast

#27

Post by rwmj » Thu, May 01, 2014, 4:47 PM UTC

lukesandberg wrote: post #10

rwmj wrote: post #2

A bit surprising (considering he started off talking about coredumps) that he doesn't mention sparse files. Core dumps can be very sparse, and you might save time and definitely will save space by not writing out the all-zeroes parts.

to do that, wouldn't you have to look at every byte just to detect the runs of 0s, that would mean that you have to pull the whole file through the memory hierarchy of your system (rather than just passing chunks from syscall to syscall) wouldn't that alone slow you down significantly?

It depends. If the data is coming from a pipe (like core_pattern) then yes you have to check for runs of zeroes. If it's coming from a filesystem, then there are various system calls that let you skip them (specifically SEEK_HOLE and SEEK_DATA flags of lseek(2)).

Also if the data is being copied into userspace anyway, then it's quite fast to check that memory is zero. There's no C "primitive" for this, but all C compilers can turn a simple loop into relatively efficient assembler[1].

If you're using an API that never copies the data into userspace and you have to read from a pipe, then yes sparse detection will be much more expensive.

In either case it should save disk space for core files which are highly sparse.

[1] https://stackoverflow.com/a/1494021

Re: Linux file write patterns: So you want to write to a file fast

#28

Post by asveikau » Thu, May 01, 2014, 4:58 PM UTC

masklinn wrote: post #17

asveikau wrote: post #6

Assuming all I/O failures are EINTR is really, really odd, as if to say disks never fill up or fail and sockets never disconnect. He does say: > in a real program you’d have to do real error handling instead of assertions, of course But somebody somewhere is reading this and thinking this is a "semantically correct pattern" (as it is introduced) and may just copy-paste it into their program. Especially when contrasti…

> Assuming all I/O failures are EINTR is really, really odd, as if to say disks never fill up or fail and sockets never disconnect. The point is to retry on EINTR and to abort completely in case of other IO failures. assert(errno == EINTR); continue; is equivalent to if (errno == EINTR) continue; abort(); > But somebody somewhere is reading this and thinking this is a "semantically correct pattern" (as it is introduc…

You are wrong. assert is a no-op when NDEBUG is defined. Some compilers will set that for you in an optimized build.

Using an assert in place of real error checking or otherwise relying on its side effects is consequently a huge wtf in C.

Re: Linux file write patterns: So you want to write to a file fast

#29

Post by Phlarp » Thu, May 01, 2014, 5:11 PM UTC

masklinn wrote: post #26

Phlarp wrote: post #18

Earlier quoted context omitted.

I weep for the memory sectors that got re-written continuously for an entire night.

SSD controllers do write-leveling, the blocks a filesystem writes to is virtual and remapped (think VMEM)

wouldn't an ~8GB page/swap file being continuously rewritten on a 250GB drive still consume a non-negligible number of write cycles over several days / weeks at most?

Re: Linux file write patterns: So you want to write to a file fast

#30

Post by tlb » Thu, May 01, 2014, 5:20 PM UTC

The code in the second example is wrong. If a write partially succeeds, instead of writing the remaining part it writes again from the beginning of the buffer. The resulting file will be incorrect. That doesn't normally happen on disk writes, but it does when writing to a pipe.