Unix’s file durability problem • Hacker News Board

Re: Unix’s file durability problem

#11

Post by ktRolster » Sat, Apr 16, 2016, 5:31 PM UTC

rocky1138 wrote: post #2

Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?

The problem isn't just the mainline source code, you also have drivers which of course vary for each device.

Then even completely beyond control of the kernel, devices have their own RAM caches, which means that even if the device reports that something was written, it might not have been. So there's no way to absolutely be sure.

Re: Unix’s file durability problem

#12

Post by ChuckMcM » Sat, Apr 16, 2016, 5:31 PM UTC

One of the engineers at Google took the time to figure this out, and updated the page out code in the Linux kernel they were using so that the "correct" steps were known if you had to know your data was on disk. It was discussed on LKML as I recall but considered "not generally useful" and I doubt it made it into the main sources.

One of the interesting things about writes is that they disrupt reads more significantly than you might expect. Greg Lindahl characterized the impact at Blekko when we were crawling so that we could optimize writes from the crawl to not disrupt latency on the search engine side. Later we completely separated those functions for similar reasons. I believe every disk I've evaluated over the years is slowest on the random read/write 50% test.

Re: Unix’s file durability problem

#13

Post by cowsandmilk » Sat, Apr 16, 2016, 5:31 PM UTC

As long as your hardware doesn't lie as well??

http://brad.livejournal.com/2094221.html

http://brad.livejournal.com/2116715.html

Re: Unix’s file durability problem

#14

Post by wanderr » Sat, Apr 16, 2016, 5:35 PM UTC

userbinator wrote: post #9

I'll admit that one reason I'm unusually grumpy about this is that I feel rather unhappy not knowing what I need to do to safeguard data that I care about. ...backups? This issue is not unsolvable at a technical level, but it probably is at a political level. Someone would have to determine and write up what is good enough now (on sane setups), and then Unix kernel people would have to say 'enough, we are not accepti…

Backups do not solve the problem of not knowing whether data you just wrote to disk will still be there in the event of a power outage or system crash. You know, before a backup has a chance to run. Sure, those events should be extremely rare but that doesn't mean that we can or should just ignore it, at a large enough scale even extremely improbable events are guaranteed to happen.

Re: Unix’s file durability problem

#15

Post by planckscnst » Sat, Apr 16, 2016, 5:36 PM UTC

faragon wrote: post #6

Rule of thumb for OS disk I/O: write as soon as possible, i.e. write as soon as possible without hurting performance using buffers on memory for slow component amortization, spreading operations during longer periods.

The article wasn't about the decision of when to write to disk, but how to actually do it. Say you have a point in your program where you have made the decision that it is necessary to write to disk. How do you actually do that for sure? It's not write() or even necessarily fsync().

The author is finding this frustrating because there are several things to do that can seem arbitrary, random, and counterintuitive. His trust in the OS was damaged and now he or she is understandably grumpy about it.

Re: Unix’s file durability problem

#16

Post by em3rgent0rdr » Sat, Apr 16, 2016, 5:37 PM UTC

I've been told to type sync into terminal whenever I want writes to complete.

Re: Unix’s file durability problem

#17

Post by lyle_nel » Sat, Apr 16, 2016, 5:37 PM UTC

rocky1138 wrote: post #2

Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?

The Linux Documentation Project has this to say about flushing of buffer cache: "In traditional UNIX systems, there is a program called update running in the background which does a sync every 30 seconds, so it is usually not necessary to use sync. Linux has an additional daemon, bdflush, which does a more imperfect sync more frequently to avoid the sudden freeze due to heavy disk I/O that sync sometimes causes."

Re: Unix’s file durability problem

#18

Post by ape4 » Sat, Apr 16, 2016, 5:39 PM UTC

I think the man page http://linux.die.net/man/2/fsync answers the questions the article asks about fsync().

fsync() = YES metadata, YES data, NO dir entry

fdatasync() = NO metadata, YES data, NO dir entry

fsync() of dir = NO metadata, NO data, YES dir entry

Re: Unix’s file durability problem

#19

Post by striking » Sat, Apr 16, 2016, 5:41 PM UTC

rogerbinns wrote: post #8

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

I wonder what the implications of making an SQLite filesystem would be.

Re: Unix’s file durability problem

#20

Post by johnbender » Sat, Apr 16, 2016, 5:43 PM UTC

userbinator wrote: post #9

I'll admit that one reason I'm unusually grumpy about this is that I feel rather unhappy not knowing what I need to do to safeguard data that I care about. ...backups? This issue is not unsolvable at a technical level, but it probably is at a political level. Someone would have to determine and write up what is good enough now (on sane setups), and then Unix kernel people would have to say 'enough, we are not accepti…

Backups don't help if writes don't make it to disk in the order and manner expected by the application programmer. There's an emerging consensus that there are crash protocol bugs lurking everywhere due to I/O scheduler reordering. For example this bug in gzip:

http://bugs.gnu.org/22768