Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux file write patterns: So you want to write to a file fast

blog.plenz.com

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Linux file write patterns: So you want to write to a file fast

#41

Post by asveikau »

Niten wrote: post #21
asveikau wrote: post #6

Assuming all I/O failures are EINTR is really, really odd, as if to say disks never fill up or fail and sockets never disconnect. He does say: > in a real program you’d have to do real error handling instead of assertions, of course But somebody somewhere is reading this and thinking this is a "semantically correct pattern" (as it is introduced) and may just copy-paste it into their program. Especially when contrasti…

> But somebody somewhere is reading this and thinking this is a "semantically correct pattern" (as it is introduced) and may just copy-paste it into their program. I dare say that would be their fault for blindly copying and pasting without taking the time to understand the context. (He even gives an explicit disclaimer!) Robust error handling would just be more noise to filter through for people actually reading the…

I'd agree, except that I've seen too many examples where people, particularly those who still have things to learn, cite blog snippets as authoritative. IMO we have something of a duty to those people to get it right. In this case it's not a lot of effort to get it right. Relying on the side-effects of an assert() is not getting it right.

The fact that I got a reply based on a misunderstanding of how asserts work tells me it's a point that needs to be made.

Re: Linux file write patterns: So you want to write to a file fast

#42

Post by _quasimodo »

awda wrote: post #31
Phlarp wrote: post #29

Earlier quoted context omitted.

wouldn't an ~8GB page/swap file being continuously rewritten on a 250GB drive still consume a non-negligible number of write cycles over several days / weeks at most?

It depends on how much free space you have on the SSD. But yes, especially because the swap file isn't ssd-aware, you get a high degree of write amplification which wears the disk more than necessary. That being said, newish SSDs can take a beating, even under these kinds of workloads.

I think swap files are written page-wise, so as long as the start of the page file is aligned, all the writes should be aligned. (assuming memory page size and SSD page size are the same)

Re: Linux file write patterns: So you want to write to a file fast

#44

Post by StillBored »

dekhn wrote: post #16
tytso wrote: post #11

It's 2014; why was the author using ext3 instead of ext4? Ext4 does have fallocate support. Also, if you use fallocate(2) instead of posix_fallocate(3), you don't have to worry about glibc trying to emulate fallocate() for those file systems which don't use it. Finally, it's a little surprising the author didn't try using O_DIRECT writes.

Most people who use O_DIRECT writes stop quickly, thinking it's "slow". What's actually happening is you're seeing what the system is actually capable of in terms of write bandwidth, without any of the 'clever' optimizations like write caching.

I don't think this is accurate. We have a kernel bypass for disk operations. We use our own memory buffers, and bypass the filesystem, block, and SCSI midlayers. Our stuff is basically what O_DIRECT should be.

There are cases where we are 50% faster than O_DIRECT without any "caching". Furthermore, in high bandwidth applications (>4GB/sec) without O_DIRECT its easy to become CPU limited in the blk/midlayer so again we win.

Now that said, I haven't tried the latest blk-mq, scsi-mq, etc patches which are tuned for higher IOP rates. These patches were driven by people plugging in high performance flash arrays and discovering huge performance issues in the kernel. Still, I expect if you plug in a couple high end flash arrays the kernel is going to be the limit rather than the IO subsystem on a modern xeon.

Re: Linux file write patterns: So you want to write to a file fast

#45

Post by zobzu »

dekhn wrote: post #16
tytso wrote: post #11

It's 2014; why was the author using ext3 instead of ext4? Ext4 does have fallocate support. Also, if you use fallocate(2) instead of posix_fallocate(3), you don't have to worry about glibc trying to emulate fallocate() for those file systems which don't use it. Finally, it's a little surprising the author didn't try using O_DIRECT writes.

Most people who use O_DIRECT writes stop quickly, thinking it's "slow". What's actually happening is you're seeing what the system is actually capable of in terms of write bandwidth, without any of the 'clever' optimizations like write caching.

but thats also why his tests are unreliable in this case

Re: Linux file write patterns: So you want to write to a file fast

#46

Post by awda »

_quasimodo wrote: post #42
awda wrote: post #31

Earlier quoted context omitted.

It depends on how much free space you have on the SSD. But yes, especially because the swap file isn't ssd-aware, you get a high degree of write amplification which wears the disk more than necessary. That being said, newish SSDs can take a beating, even under these kinds of workloads.

I think swap files are written page-wise, so as long as the start of the page file is aligned, all the writes should be aligned. (assuming memory page size and SSD page size are the same)

Does writes being aligned on page boundaries have anything to do with them being 13 base 2 orders of magnitude smaller than flash erase blocks? An unaligned page means you erase (mostly) one or (very rarely) two flash blocks. An aligned page means you erase one flash block. The biggest amplification is the 1 32MB amplification...

Re: Linux file write patterns: So you want to write to a file fast

#47

Post by nkurz »

asveikau wrote: post #41
Niten wrote: post #21

Earlier quoted context omitted.

> But somebody somewhere is reading this and thinking this is a "semantically correct pattern" (as it is introduced) and may just copy-paste it into their program. I dare say that would be their fault for blindly copying and pasting without taking the time to understand the context. (He even gives an explicit disclaimer!) Robust error handling would just be more noise to filter through for people actually reading the…

I'd agree, except that I've seen too many examples where people, particularly those who still have things to learn, cite blog snippets as authoritative. IMO we have something of a duty to those people to get it right. In this case it's not a lot of effort to get it right. Relying on the side-effects of an assert() is not getting it right. The fact that I got a reply based on a misunderstanding of how asserts work tel…

The fact that I got a reply based on a misunderstanding of how asserts work tells me it's a point that needs to be made.

I'm just a bystander, but I think you may be jumping to unfounded conclusions here. Based on previous comment history, I presume that 'masklinn' understands perfectly well how assert() works. Yes, if you define NDEBUG your error handling will go away. So don't define NDEBUG unless you want your error handling to go away!

By contrast, your assertion that Some compilers will set that for you in an optimized build strikes me as unlikely. Some program specific build systems do this, and if you use one of them you should be aware that your assert() functions may drop out. But I don't think I've ever used a compiler that drops the assert() based on optimization level.

I don't particularly disagree with your conclusion, just your argument. I think 'awda' gets closer to the truth: the default assert() from with its negative reliance on NDEBUG is tricky and probably best avoided -- not just for error handling but altogether. Personally, I use two distinct macros: ERROR_ASSERT() and DEBUG_ASSERT(). ERROR_ASSERT() cannot be disabled, and DEBUG_ASSERT() only runs if DEBUG was defined at compile time.

Re: Linux file write patterns: So you want to write to a file fast

#48

Post by asveikau »

nkurz wrote: post #47
asveikau wrote: post #41

Earlier quoted context omitted.

I'd agree, except that I've seen too many examples where people, particularly those who still have things to learn, cite blog snippets as authoritative. IMO we have something of a duty to those people to get it right. In this case it's not a lot of effort to get it right. Relying on the side-effects of an assert() is not getting it right. The fact that I got a reply based on a misunderstanding of how asserts work tel…

The fact that I got a reply based on a misunderstanding of how asserts work tells me it's a point that needs to be made. I'm just a bystander, but I think you may be jumping to unfounded conclusions here. Based on previous comment history, I presume that 'masklinn' understands perfectly well how assert() works. Yes, if you define NDEBUG your error handling will go away. So don't define NDEBUG unless you want your err…

> your assertion that Some compilers will set that for you in an optimized build strikes me as unlikely

Uhh, I didn't make it up. I remember now what I was thinking of: the defaults for Visual Studio (not the compiler, the IDE) are to have -DNDEBUG in release mode. So lots of Windows projects end up having it without the authors explicitly asking.

(I thought I also once used a machine, maybe some obscure Unix, where cc would add it if you specified -O. I don't remember the details of that, or if I might be confusing it with what VS did.)

FWIW I don't think it's weird that assert has this quirk, I think some people in this discussion just disagree about what an assert is. If you think of it as an extra debug check that might not be evaluated and should not have side effects, and are fine with that conceptually, no problems.

Re: Linux file write patterns: So you want to write to a file fast

#50

Post by mtdewcmu »

rwmj wrote: post #27
lukesandberg wrote: post #10

Earlier quoted context omitted.

to do that, wouldn't you have to look at every byte just to detect the runs of 0s, that would mean that you have to pull the whole file through the memory hierarchy of your system (rather than just passing chunks from syscall to syscall) wouldn't that alone slow you down significantly?

It depends. If the data is coming from a pipe (like core_pattern) then yes you have to check for runs of zeroes. If it's coming from a filesystem, then there are various system calls that let you skip them (specifically SEEK_HOLE and SEEK_DATA flags of lseek(2)). Also if the data is being copied into userspace anyway, then it's quite fast to check that memory is zero. There's no C "primitive" for this, but all C comp…

The easiest way to handle things like sparse files correctly is to invoke a program like GNU dd that already has this feature built in. GNU cp handles it, too, but it doesn't accept input from stdin.
Post reply on HN