Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix’s file durability problem

utcc.utoronto.ca

Post reply on HN
Print view View original

21–30 of 161 posts

Re: Unix’s file durability problem

#21

Post by planckscnst »

lyle_nel wrote: post #17
rocky1138 wrote: post #2

Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?

The Linux Documentation Project has this to say about flushing of buffer cache: "In traditional UNIX systems, there is a program called update running in the background which does a sync every 30 seconds, so it is usually not necessary to use sync. Linux has an additional daemon, bdflush, which does a more imperfect sync more frequently to avoid the sudden freeze due to heavy disk I/O that sync sometimes causes."

Wow, I'd love to know what they mean by "imperfect". In Linux, specifically, the sync_dirty_buffers syncs all the dirty write buffers (nope - see edit) and isn't imperfect or incomplete in any particular way.

Edit: Nevermind, I'm remebering now there is a dirty_background_ratio separate from a dirty_ratio. It will sync some of the dirty buffers as long as it's above dirty_background_ratio and under dirty_ratio. And it was actually bdflush that determined all of this; the sync_dirty_buffer function just took a specific buffer object and synced that.

Edit 2: And I'm finding out that information seems to be outdated. Here's the old documentation about bdflush [1]. That was either moved or removed in the latest version, but I did find this [2], which suggests that there was an old bdflush vs something else now. Looks like I missed something important and have some research to do.

[1] http://lxr.free-electrons.com/source/Documentation/sysctl/vm...

[2] http://lxr.free-electrons.com/source/Documentation/vm/active...

Re: Unix’s file durability problem

#22

Post by jclulow »

striking wrote: post #19
rogerbinns wrote: post #8

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

I wonder what the implications of making an SQLite filesystem would be.

If you then expose that file system through a POSIX file system API, you have all of the same issues of underspecified or unclear behaviour that the article mentions.

Re: Unix’s file durability problem

#23

Post by johnbender »

rogerbinns wrote: post #8

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

Do you know if they address I/O reordering within the scheduler? For example transaction implementations often require that writes (distinct file system calls) hit the disk in a particular order to guarantee a sane state for the database. Prime example is the GNU bug for gzip:

http://bugs.gnu.org/22768

The the writes to the `foo.gz` file have to hit the disk before the unlink but the I/O scheduler can reorder these potentially and a badly timed crash could result in data loss. Note that journaling doesn't fix this issue because the transactions are distinct too.

Re: Unix’s file durability problem

#25

Post by Pyxl101 »

userbinator wrote: post #9

I'll admit that one reason I'm unusually grumpy about this is that I feel rather unhappy not knowing what I need to do to safeguard data that I care about. ...backups? This issue is not unsolvable at a technical level, but it probably is at a political level. Someone would have to determine and write up what is good enough now (on sane setups), and then Unix kernel people would have to say 'enough, we are not accepti…

Not a good answer for application servers. You can lose data in between the application issuing the write and the disk persisting it, and in between persistence and backup.

The absence of data corruption doesn't necessarily mean the presence of all data either. Imagine a fleet of application servers that write several times per second and there are hundreds of them. How would you know if one write had gotten lost in a power failure?

The best available answer among folks who need one is to pass all writes through a distributed system that can commit the data durably: data replicated in real time as part of the process of committing a write, the metadata tracked with a quorum or lock managed by quorum.

Re: Unix’s file durability problem

#26

Post by jclulow »

em3rgent0rdr wrote: post #16

I've been told to type sync into terminal whenever I want writes to complete.

The sync(1M) command provided on many UNIX systems is often a thin wrapper around the sync(2) system call. The sync(2) entry point does not, as specified, guarantee that all inflight data _has_ been written; it's more of a guideline than an actual rule. The user must subsequently wait for the data to actually make it to disk, often for an arbitrarily long amount of time.

Re: Unix’s file durability problem

#28

Post by rdtsc »

planckscnst wrote: post #15
faragon wrote: post #6

Rule of thumb for OS disk I/O: write as soon as possible, i.e. write as soon as possible without hurting performance using buffers on memory for slow component amortization, spreading operations during longer periods.

The article wasn't about the decision of when to write to disk, but how to actually do it. Say you have a point in your program where you have made the decision that it is necessary to write to disk. How do you actually do that for sure? It's not write() or even necessarily fsync(). The author is finding this frustrating because there are several things to do that can seem arbitrary, random, and counterintuitive. His…

> How do you actually do that for sure? It's not write() or even necessarily fsync().

Right. For decision of "when" you can configure a few kernel parameters, namely dirty writes thresholds. That assumes you just do write()s from your process and kernel decides to flush that data out. That can be configured to be a function of time and or amount of unwritten data.

I had to do it a few times. Once it was a realtime-ish system which was recording data to disk, and noticed recording thread would time-out. (Timeouts were in the 10s of seconds and there were noticed by a watchdog system). Thought 10 seconds should be enough for that process. But it turns out because of how priorities were setup and how fast writes went, dirty page flushing was not keeping up with writes. Periodically it would hit the top threshold and at that point any process doing disk writes would be blocked.

Was able to get it under control by essentially doing what gp suggested, writing a little bit at a time, but more often. The total throughput probably went went down but performance was smoothed out quite a bit.

Re: Unix’s file durability problem

#29

Post by userbinator »

wanderr wrote: post #14
userbinator wrote: post #9

I'll admit that one reason I'm unusually grumpy about this is that I feel rather unhappy not knowing what I need to do to safeguard data that I care about. ...backups? This issue is not unsolvable at a technical level, but it probably is at a political level. Someone would have to determine and write up what is good enough now (on sane setups), and then Unix kernel people would have to say 'enough, we are not accepti…

Backups do not solve the problem of not knowing whether data you just wrote to disk will still be there in the event of a power outage or system crash. You know, before a backup has a chance to run. Sure, those events should be extremely rare but that doesn't mean that we can or should just ignore it, at a large enough scale even extremely improbable events are guaranteed to happen.

Backups do not solve the problem of not knowing whether data you just wrote to disk will still be there in the event of a power outage or system crash.

It's not a "problem", it's just reality. You can't predict when exactly the crash will occur relative to the disk write or backups.

Re: Unix’s file durability problem

#30

Post by andolanra »

striking wrote: post #19
rogerbinns wrote: post #8

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

I wonder what the implications of making an SQLite filesystem would be.

I'm not actually sure whether this actually helps with the problem described in the post, but such a thing does exist:

https://github.com/guardianproject/libsqlfs

Post reply on HN