Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix’s file durability problem

utcc.utoronto.ca

Post reply on HN
Print view View original

151–160 of 161 posts

Re: Unix’s file durability problem

#151

Post by jstimpfle »

caf wrote: post #130
jstimpfle wrote: post #59

Earlier quoted context omitted.

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2) : If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing"…

rename(2) atomic replacement only means atomic with respect to processes running on the system at the same time: anyone looking up the newpath will get either the old file or the new file (and won't, for example, see a spurious ENOENT). It doesn't mean atomic with respect to a system crash. The easiest way to get asynchronous fsync(2) is to do the fsync(2) in a different thread.

> The easiest way to get asynchronous fsync(2) is to do the fsync(2) in a different thread.

However you would need to create a new thread for each simultaneous asynchronous fsync to get the benefits of asynchronicity.

Re: Unix’s file durability problem

#153

Post by pgaddict »

FWIW this uncertainty in fsync behavior is essentially the cause of the recent PostgreSQL bug on ext4 (fixed in the last round of minor releases).

Basically, on XFS and all other tested filesystems is seems to work just fine, on EXT4 we may loose the last rename() effects.

So this makes it difficult to get the durability right, even when the project is as careful about it as PostgreSQL.

Re: Unix’s file durability problem

#154

Post by pbsd »

rikkus wrote: post #145

I use something akin to djb's Maildir delivery procedure when I want to be as close to sure as I can be that a file 'has been saved'. 1. Create a temp file on the same mount 2. Write data to file, checking return of each write(), then a final fsync(), and close() 3. link() to filename we actually want I still haven't figured out how to be this safe on Windows, because AFAICT there's no atomic way to link() or 'move'…

`ReplaceFile` is supposed to be atomic, at least according to the documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/hh8...

Re: Unix’s file durability problem

#156

Post by colin_mccabe »

UNIX has "a standard way to deal with the file durability problem." If you fsync the file, then fsync the containing directory, that should be durable on all properly behaved setups.

Of course this comes with a lot of footnotes. There was a bug on some older Linux kernels where they didn't tell the hard drive to flush the data after an fsync. This is why a lot of people still incorrectly believe that enabling hard disk write cache is unsafe. But this was a bug, and the bug was fixed. There are also reports of hard drives that don't honor flush requests. There's not much UNIX or any other OS can do about this-- if the hardware lies, you are in trouble.

You could also imagine a much richer durability API. Empirically, databases need such a richer API rather than simple fsync. The POSIX async I/O standard was supposed to standardize all this, but Linux's glibc just implements it as a thread pool making blocking system calls. If you want real async I/O on Linux, you need to use an OS-specific API.

Re: Unix’s file durability problem

#157

Post by anarazel »

haberman wrote: post #128
avar wrote: post #120

Earlier quoted context omitted.

> That makes no sense. Have each process write to its own log. This is > making your life hard for no reason. This is literally how pretty much every daemon process works on Unix systems. You don't log in and tail /var/log/apache/access.log.{1..N} where N is the number of threads/processes. You just tail /var/log/apache/access.log which all of them write to by virtue of PIPE_BUF semantics. Having to tail N logs where…

> You just tail /var/log/apache/access.log which all of them write to by virtue of PIPE_BUF semantics. I can see how this would be convenient. That's fine and I have nothing against it. It's just not a problem I consider interesting, because there is no inherent need for all 100 writers to write to the same file. You could just as easily have each write to a separate file and have a separate process that merges them…

>> I really don't see how any of this is relevant to the original point of having "log" files. > Maybe you're not aware: every major database keeps a commit log, and writes it to a file called a log file (sometimes "write-ahead log file").

> You do realize that SQLite can't just corrupt your database because you ran two SQLites concurrently, right?

Sure, but, so? I don't know sqlite's WAL logging code in detail, but I do know PostgreSQL's fairly intimately. I don't see how such an interface[1] would be relevant for WAL logging. Such logs usually have checksums and pointers to previous records in their format. For those to be correct each writing process needs to know about previous records (or at least their starting point). Thus you need locking and coordination in userspace anyway - kernel level append mechanics aren't that interesting.

In addition to that, if you care about performance, you'll want to pre-allocate the WAL files and possibly re-use them after a checkpoint. For many filesystems overwriting files is a lot more efficient than allocating new blocks. It avoids the need for fs-internal metadata journaling, avoids fragmentation etc.. With pre-allocated files you then can use fdatasync() instead of fsync(), which can be considerable performance benefit in our experience.

There are things that'd make it easier and more efficient to write correct and efficient journaling, but imo not what you were talking about. Querying and actually getting guarantees about which size of writes are atomic, for example; otherwise you need to use rather expensive workarounds like WAL logging full page contents after checkpoints, or double-write buffers.

Proper asynchronous fsync(), fdatasync() would also be rather useful.

[1] > I'm basically talking about compare and swap, except instead of compare and swap its compare and append

Re: Unix’s file durability problem

#158

Post by tzs »

sdk77 wrote: post #143
brigade wrote: post #140

Earlier quoted context omitted.

Why? Wouldn't a large enough capacitor last longer than batteries? UPS batteries I've seen are only rated for 3 years, and half of my recent laptop batteries have physically swelled after 3 years of use.

It would, but the size would be impractical and it's too costly. Quick estimate, let's assume we need 10W for 10 seconds, that's 100 Joule of energy. The energy stored in a capacitor is 0.5 C V^2. Say we use a 10V capacitor then C = 2 Farad. They exist, but they are very large (look them up on amazon for instance). You'll probably need more like twice the capacity though because it's impossible to extract all energy…

Instead of 10 V and 2 F, how about going for 2.5 V and 50 F? A capacitor with those specs is only 40 mm long and 18 mm diameter [1]. That shouldn't be too hard to fit in a typical server or desktop. That's under $4 in quantity.

[1] http://www.mouser.com/ds/2/257/Maxwell_HCSeries_DS_1013793-9...

Re: Unix’s file durability problem

#159

Post by rasz_pl »

cm2187 wrote: post #135
rasz_pl wrote: post #133

Earlier quoted context omitted.

1 is not possible on anything with more than one >100MHz chip

You mean there is no capacitor powerful enough?

Its one thing to add $0.5 supercap and a diode to SSD (and still the likes of OCZ didnt bother for cost reasons). Its entirely another to slap bank of boost/buck converters, 100F cap and spider web of control circuity all over the place.

Adding $20-50 in bom the size of two D batteries just in case something _very rare_ happens is not economically viable. There are better alternatives, like banks of batteries ala google boxes. Turning server into oversized laptop gives you more than couple of seconds buffer.

Re: Unix’s file durability problem

#160

Post by sdk77 »

tzs wrote: post #158
sdk77 wrote: post #143

Earlier quoted context omitted.

It would, but the size would be impractical and it's too costly. Quick estimate, let's assume we need 10W for 10 seconds, that's 100 Joule of energy. The energy stored in a capacitor is 0.5 C V^2. Say we use a 10V capacitor then C = 2 Farad. They exist, but they are very large (look them up on amazon for instance). You'll probably need more like twice the capacity though because it's impossible to extract all energy…

Instead of 10 V and 2 F, how about going for 2.5 V and 50 F? A capacitor with those specs is only 40 mm long and 18 mm diameter [1]. That shouldn't be too hard to fit in a typical server or desktop. That's under $4 in quantity. [1] http://www.mouser.com/ds/2/257/Maxwell_HCSeries_DS_1013793-9...

Theoretically it's possible, but in practice the lower input voltage makes it harder to convert it to +5/+12V. It becomes increasingly lossy and expensive - to convert 0.5V to 12V at 10W is not trivial to begin with. Voltage across a capacitor drops continuously while discharging (unlike a battery). So with a 2.5V capacitor, being able to use it between 1.25V and 2.5V is already pushing it. On average, discharge current at 10W is around 8A. The internal resistance of the capacitor (ESR) better be very low (it probably isn't) at this low voltage - even if it's 0.1 Ohm, at 8A we already lost 0.8V from out meager 2.5V, and now the useful energy is just 1.7-1.25 = 0.5C0.45V^2 = 5J, just enough for 500ms at 10W.
Post reply on HN