Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix’s file durability problem

utcc.utoronto.ca

Post reply on HN
Print view View original

81–90 of 161 posts

Re: Unix’s file durability problem

#81

Post by rocky1138 »

rsync wrote: post #78
em3rgent0rdr wrote: post #16

I've been told to type sync into terminal whenever I want writes to complete.

"I've been told to type sync into terminal whenever I want writes to complete." If you'd really like a guarantee, you can always unmount the filesystem, or alternatively, mount it read-only: mount -ur /mnt/blah That will guarantee, at least at the filesystem/OS level, that the writes are committed.

My understanding that, after that point, the hardware drivers could lie and say they've written it and we'd never know, correct?

And even after that, the hardware could return a "written OK" value but not actually do the job, right?

So the point is that without complete transparency from end-to-end, there's no way to tell.

Is that correct?

Re: Unix’s file durability problem

#82

Post by schmichael »

Disks can buffer, disks can have firmware bugs, disks can fail both catastrophically and subtly. Even the fanciest battery-backed RAID controllers have firmware bugs and dead batteries. Those are just ways your storage hardware can fail you even if the kernel and libraries are bug free and you follow the right mystic incantations to sync data.

While there's no excuse for bad documentation or poor APIs, you can never consider data written to a single local disk "safe". It never is.

It's a shame making a best effort at safety is nontrivial, but it does force developers to write more defensive and crash-safe code which is all that can save you in the end.

Re: Unix’s file durability problem

#83

Post by frutiger »

Sanddancer wrote: post #62
dap wrote: post #43

Earlier quoted context omitted.

How would it be different than the filesystem API? There was a great lightning talk a few years ago that I can't seem to find where the author described an API for storing blobs in a hierarchical namespace. Of course, halfway through, it became clear that it's just the POSIX API: you can "open" handles to objects, "rename" them, remove them, and so on. You'd end a transaction with "fsync()". (Okay, that one's a littl…

The problem is that fsync() runs outside of the control of the program. There's no way for an application to start a transaction, perform steps x, y, and z, and then end a transaction, rolling back to before step x if there are any failures. For example, suppose you're rotating a an audit log file at the same moment your backup program is running. Your backup program reads the directory, and at that same moment, your…

Realistically, backups need to be based on atomic snapshots, like `zfs`.

Re: Unix’s file durability problem

#84

Post by tremon »

rwmj wrote: post #70
robotresearcher wrote: post #69

Earlier quoted context omitted.

SQLite stores data on disk in a file.

Er, yes, that's the point.

I think his point was that SQLite still has to work around the durability problem described in the article. As such, SQLite isn't a solution, it's a middleware API. I'm not trying to knock it, but solving a problem by adding a new layer isn't good design.

Also, middleware shouldn't be needed in open source because you can simply patch the underlying API. That the middleware is the best solution in this case says a lot about the problem.

Re: Unix’s file durability problem

#85

Post by robotresearcher »

rwmj wrote: post #70
robotresearcher wrote: post #69

Earlier quoted context omitted.

SQLite stores data on disk in a file.

Er, yes, that's the point.

So the reliability and storage semantics bottom out at the filesystem after all. It's not as simple as 'I'll use SQLite: problem solved'. Not for the authors of SQLite, for sure.

Re: Unix’s file durability problem

#87

Post by theoh »

jstimpfle wrote: post #38
Ericson2314 wrote: post #24

Yet another example about how the filesystem, and its POSIX realization, is in fact a horrible abstraction.

I don't think there is anything wrong with the POSIX file system API. Two things: - I think it's mainly the modern file systems like Btrfs and I think partly also ext4, which introduced a shift in paradigm, which broke old applications (or at least broke their performance, for example dpkg). - We're talking about a hierarchical file system, meaning it's easy for humans to find data, but terrible for machines because…

I think this is a bad option from a concurrency point of view, but why not one of the relational database alternatives for maintaining a hierarchy? http://stackoverflow.com/questions/4048151/what-are-the-opti...

Re: Unix’s file durability problem

#88

Post by tremon »

rocky1138 wrote: post #81
rsync wrote: post #78

Earlier quoted context omitted.

"I've been told to type sync into terminal whenever I want writes to complete." If you'd really like a guarantee, you can always unmount the filesystem, or alternatively, mount it read-only: mount -ur /mnt/blah That will guarantee, at least at the filesystem/OS level, that the writes are committed.

My understanding that, after that point, the hardware drivers could lie and say they've written it and we'd never know, correct? And even after that, the hardware could return a "written OK" value but not actually do the job, right? So the point is that without complete transparency from end-to-end, there's no way to tell. Is that correct?

Yes, and there's also the problem that magnetization of material isn't permanent so your data may disappear after N days even if it were written correctly (typically, N > 5000). Should your "complete transparency" model also include expected longevity of the written bits?

Re: Unix’s file durability problem

#89

Post by ryao »

kentonv wrote: post #79
Animats wrote: post #67

Earlier quoted context omitted.

Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.) O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly. How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating t…

> How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. Note that Linux ext4 does not do this. On a power outage, you can get bogus trailing zeros on a file which you were appending, because the file size was updated before the data was written. I asked Ted T'so about this and he said it was working as intended. https://plus.google.com/+K…

ZFS does do this. Changes are atomic, so they either happen or do not. There is no in-between.

Re: Unix’s file durability problem

#90

Post by rwmj »

tremon wrote: post #84
rwmj wrote: post #70

Earlier quoted context omitted.

Er, yes, that's the point.

I think his point was that SQLite still has to work around the durability problem described in the article. As such, SQLite isn't a solution , it's a middleware API. I'm not trying to knock it, but solving a problem by adding a new layer isn't good design. Also, middleware shouldn't be needed in open source because you can simply patch the underlying API. That the middleware is the best solution in this case says a l…

SQLite does solve the problem because it knows exactly what fsync()/fdatasync()/whatever calls are necessary to persist a file, and it also knows that for the many operating systems that SQLite is implemented on. It's also tested regularly.

I can't believe the level of ignorance in this thread.

Post reply on HN