Unix’s file durability problem • Hacker News Board

Re: Unix’s file durability problem

#51

Post by EdiX » Sat, Apr 16, 2016, 6:59 PM UTC

johnbender wrote: post #20

userbinator wrote: post #9

I'll admit that one reason I'm unusually grumpy about this is that I feel rather unhappy not knowing what I need to do to safeguard data that I care about. ...backups? This issue is not unsolvable at a technical level, but it probably is at a political level. Someone would have to determine and write up what is good enough now (on sane setups), and then Unix kernel people would have to say 'enough, we are not accepti…

Backups don't help if writes don't make it to disk in the order and manner expected by the application programmer. There's an emerging consensus that there are crash protocol bugs lurking everywhere due to I/O scheduler reordering. For example this bug in gzip: http://bugs.gnu.org/22768

This bug report is truly surreal. A filesystem could easily just write on disk directly skipping write buffers, the whole reason they don't is "because performance".

In fact the "file system mathematically guaranteed to not lose data" is too slow to be used in practice and will implement fsync/fdatasync in the future to regain performance (and in the process it will stop being "mathematicall guaranteed not to lose data").

Clearly the solution is to add fsync/fdatasync calls to every single program so as to negate the performance gains of file system write buffers entirely.

Clearly, the next step after that is for filesystem to start ignoring fsync/fdatasync entirely, because otherwise they would be too slow.

Re: Unix’s file durability problem

#52

Post by rwmj » Sat, Apr 16, 2016, 7:04 PM UTC

twa927 wrote: post #44

rogerbinns wrote: post #8

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

> A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen(). So if I need to create a file with a timestamp, you are suggesting creating a SQLite database, a table, and inserting my timestamp there? It's nonsensical to "fix" a Unix low-level syscall issue by using SQLite.

I'm pretty sure any OS or API developed continuously over 25-40 years (depending how you count) would suffer from the same problems. Technology changes. Early assumptions prove to be wrong. Any OS developed from scratch now would suffer the same problem 25 years from now.

It's worthwhile to use libraries which embody the knowledge, trials and errors of years, such as SQLite.

Re: Unix’s file durability problem

#53

Post by rogerbinns » Sat, Apr 16, 2016, 7:05 PM UTC

tobias3 wrote: post #48

rogerbinns wrote: post #8

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

SQLite just (two month ago) got a new synchronous level EXTRA which fsyncs the directory after deleting the journal. The default is FULL ( https://www.sqlite.org/pragma.html#pragma_synchronous ). So you still have to think about those problems.

> So you still have to think about those problems.

That is the case, but the advantage of the SQLite approach is that they have already done so, and hence their code is far more likely to be "correct" than any new code you write. The many billions of instances of SQLite already deployed also help with maturity and finding corner case issues.

And as a bonus it also lets you do rollbacks, transactions etc which aren't easy if trying to implement them yourself.

Re: Unix’s file durability problem

#54

Post by Animats » Sat, Apr 16, 2016, 7:09 PM UTC

I've suggested an approach to this before. There should be several types of files.

- "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. This should be the default when a file is opened via "creat()"

- "Temp" files always go away completely after a crash. This should be the default for designated temp directories.

- "Log" files can only be appended. Writers cannot seek. The file system guarantees that after a crash, the end of the file is at the end of some write; the file may not tail off into junk. This should be the default for files opened for append.

- "Managed" files are for structured databases. They have an additional API. "writemanaged()" has a callback parameter. It returns when the data has been queued, but in addition, the writer gets a callback when the write has committed to disk. The file system must guarantee that a write for which the callback has been made will survive a crash. This provides fine-grained information of when data has been committed to disk, which is what a database needs. It improves performance by not blocking waiting for disk commit to take place. The database can have several I/O operations going at once in different parts of the file without blocking.

Re: Unix’s file durability problem

#55

Post by rwmj » Sat, Apr 16, 2016, 7:12 PM UTC

leni536 wrote: post #31

What about O_DIRECT | O_SYNC options in linux's open()?

O_SYNC has abysmal performance. O_DIRECT has very underspecified semantics, but particularly it demands that you only read and write whole "blocks" (where "block" depends on the filesystem type, underlying block device and phase of the moon).

Re: Unix’s file durability problem

#56

Post by jwatte » Sat, Apr 16, 2016, 7:23 PM UTC

fdatasync() followed by fsync() "should" do it although on very recently created/mounted directories you'll want to use sync() as well. Luckily sync() is synchronous on Linux (sanity!) even though traditional Unix doesn't require it to be.

So perhaps the article can be shortened to "sync() is not guaranteed to be synchronous on non-Linux Unix."

Re: Unix’s file durability problem

#57

Post by twa927 » Sat, Apr 16, 2016, 7:24 PM UTC

rwmj wrote: post #52

twa927 wrote: post #44

Earlier quoted context omitted.

> A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen(). So if I need to create a file with a timestamp, you are suggesting creating a SQLite database, a table, and inserting my timestamp there? It's nonsensical to "fix" a Unix low-level syscall issue by using SQLite.

I'm pretty sure any OS or API developed continuously over 25-40 years (depending how you count) would suffer from the same problems. Technology changes. Early assumptions prove to be wrong. Any OS developed from scratch now would suffer the same problem 25 years from now. It's worthwhile to use libraries which embody the knowledge, trials and errors of years, such as SQLite.

SQLite is an SQL database engine. Using it for storing a single value is like attacking a fly with a nuclear weapon.

Unless you are talking about a function in the SQLite library that implements the durability calls properly and can be called from other programs.

Re: Unix’s file durability problem

#58

Post by rwmj » Sat, Apr 16, 2016, 7:33 PM UTC

twa927 wrote: post #57

rwmj wrote: post #52

Earlier quoted context omitted.

I'm pretty sure any OS or API developed continuously over 25-40 years (depending how you count) would suffer from the same problems. Technology changes. Early assumptions prove to be wrong. Any OS developed from scratch now would suffer the same problem 25 years from now. It's worthwhile to use libraries which embody the knowledge, trials and errors of years, such as SQLite.

SQLite is an SQL database engine. Using it for storing a single value is like attacking a fly with a nuclear weapon. Unless you are talking about a function in the SQLite library that implements the durability calls properly and can be called from other programs.

You should use SQLite whenever you are tempted to write some data and you have any expectations to be able to read back that data. That applies even for a single value although most programs will store more than just a single value.

Re: Unix’s file durability problem

#59

Post by jstimpfle » Sat, Apr 16, 2016, 7:34 PM UTC

Animats wrote: post #54

I've suggested an approach to this before. There should be several types of files. - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. This should be the default when a file is opened via "creat()" - "Temp" files a…

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version.

Incidentally, this is the way it currently is. From rename(2): If "newpath already exists, it will be atomically replaced".

Just don't forget that "replacing" means to create a _new_ file, writing it, syncing it, and then replacing the old version with rename(2) (and fsyncing the directory).

> - "Temp" files always go away completely after a crash. This should be the default for designated temp directories.

You can have that: Just don't link the inode of the file. On Linux, you can use O_TMPFILE. Alternatively, just create the file, and keep it open but immediately unlink it (it's a little hack but nothing dramatic).

You can also write to /tmp. Normally that gets cleaned at reboot, which might be enough.

> - "Log" files can only be appended.

In open(2), look for O_APPEND.

> Writers cannot seek.

That's not how you write a kernel API. If you don't want to seek, then don't.

> The file system guarantees that after a crash, the end of the file is at the end of some write

How do you intend to implement that? Disks don't support arbitrarily large atomic commits. Making such a guarantee in software is necessarily inefficient. The kernel will not choose your tradeoffs for you. You can do it from userland. Decide yourself what is the least bad way to do it.

> "Managed" files are for structured databases. [..] The writer gets a callback when the write has committed to disk. The file system must guarantee that a write for which the callback has been made will survive a crash.

Fsync does just that, except it isn't asynchronous. Maybe there's something behind aio(7)?

Re: Unix’s file durability problem

#60

Post by Ericson2314 » Sat, Apr 16, 2016, 7:34 PM UTC

jstimpfle wrote: post #38

Ericson2314 wrote: post #24

Yet another example about how the filesystem, and its POSIX realization, is in fact a horrible abstraction.

I don't think there is anything wrong with the POSIX file system API. Two things: - I think it's mainly the modern file systems like Btrfs and I think partly also ext4, which introduced a shift in paradigm, which broke old applications (or at least broke their performance, for example dpkg). - We're talking about a hierarchical file system, meaning it's easy for humans to find data, but terrible for machines because…

A friend of mine had the realization that all optimization boils down to making lower layers understand higher layers' concerns, or higher level layers understand on lower levels' concerns. Here are some examples off the top of my head:

POSIX fails as a high level interface:

- No (at all powerful) notion of transactions. Mutation without transactions is extraordinarily primitive.

- No multiple FS roots to indicate boundary across which data will never be synchronized. (This is also good for how to spread data across multiple devices, a low-level concern.)

- Overall pushes people to maintain their own structure within files rather than use FS's trees.

POSIX fails as a low level interface:

- No way to hint or control caching along the memory hierarchy.

- Block size, locality out of control.

- Can't statically disallow various non-free actions which are costly (ability to expand file), can only hope that not using them does incur penalty.

Perhaps ZFS an BTRFS made other approaches more viable/obvious, but these weakness are inherent to the API itself and the high level stuff was noticeably missing from the get-go.