Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Working with Files Is Hard (2019)

danluu.com

Post reply on HN
Print view View original

61–70 of 123 posts

Re: Working with Files Is Hard (2019)

#61

Post by IgorPartola »

Retr0id wrote: post #9
continuational wrote: post #4

> Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually…

> why the file API so hard to use that even experts make mistakes? I think the short answer is that the APIs are bad. The POSIX fs APIs and associated semantics are so deeply entrenched in the software ecosystem (both at the OS level, and at the application level) that it's hard to move away from them.

Not only that, but the POSIX file API also assumes that NFS is a thing but NFS breaks half the important guarantees of a file system. I don’t know if it’s a baby and bath water situation, but NFS just seems like a whole bunch of problems. It’s like having eval in a programming language.

Re: Working with Files Is Hard (2019)

#62

Post by huntaub »

IgorPartola wrote: post #61
Retr0id wrote: post #9

Earlier quoted context omitted.

> why the file API so hard to use that even experts make mistakes? I think the short answer is that the APIs are bad. The POSIX fs APIs and associated semantics are so deeply entrenched in the software ecosystem (both at the OS level, and at the application level) that it's hard to move away from them.

Not only that, but the POSIX file API also assumes that NFS is a thing but NFS breaks half the important guarantees of a file system. I don’t know if it’s a baby and bath water situation, but NFS just seems like a whole bunch of problems. It’s like having eval in a programming language.

What aspects of NFS do you think break half of the important guarantees of a file system?

Re: Working with Files Is Hard (2019)

#63

Post by mystified5016 »

userbinator wrote: post #28

I don't get it. The only times I've had problems with filesystem corruption in the past few decades was with a hardware problem, and said hardware was quickly replaced. FAT family has been perfectly fine while I've encountered corruption on every other FS including NTFS, exFAT, and the ext* family. Meanwhile you can read plenty of stories of others having the exact opposite experience. If you keep losing data to powe…

> If you keep losing data to power losses or crashes, perhaps fix the cause of that? It doesn't make sense to try to work around it.

Ponder this notion for a moment: there are problems within one's control and problems outside of one's control.

For example, we can't control the weather. If it snows three feet overnight you simply have to deal with the fact that you're not getting to work today.

Since we can't simply stop hardware from failing, we have to deal with the fact that hardware fails. Your seventeen redundant UPSes might experience a one in a trillion cascade failure. It might take the utility ten minutes longer to restore your power than you have onsite generation.

This is not a class of problem we can control or prevent. We fix these problems by building systems which withstand failures. You can't just will electrons out of the wall socket, but you can build a better disk or FS that corrupts less data when the electrons stop.

Re: Working with Files Is Hard (2019)

#65

Post by josephg »

ryao wrote: post #39
josephg wrote: post #32

Earlier quoted context omitted.

It doesn’t have to be one or the other. Developers could decide by passing flags to open. But even then, doing atomic writes of multi gigabyte files doesn’t sound that hard to implement efficiently. Just write to disk first and update the metadata atomically at the end. Or whenever you choose to as a programmer. The downside is that, when overwriting, you’ll need enough free space to store both the old and new versio…

Packages consist of multiple files. An atomic file write would not allow packages to be either installed or not installed by APT.

Atomicity could encompass a whole bunch of writes at once.

Databases implemented atomic transactions in the 70s. Let’s stop pretending like this is an unsolvable CS problem. Its not.

Re: Working with Files Is Hard (2019)

#66

Post by Joker_vD »

BobbyTables2 wrote: post #50
Joker_vD wrote: post #20

Earlier quoted context omitted.

> Most applications can get by with file-level atomicity--make whole file read/writes atomic with a copy-on-write model, and you can eliminate whole classes of filesystem bugs pretty quickly. int fd = open(".config", O_RDWR | O_CREAT | O_SYNC_ON_CLOSE, 0o666); // effects of calls to write(2)/etc. are invisible through any other file description // until the close(2) is called on all descriptors to this file descripti…

Surely this can’t always be true? What happens when a lot of data is written and exceeds the dirty threshold?

It gets written on the disk but into different inodes, I imagine.

Re: Working with Files Is Hard (2019)

#67

Post by josephg »

timewizard wrote: post #41
josephg wrote: post #32

Earlier quoted context omitted.

It doesn’t have to be one or the other. Developers could decide by passing flags to open. But even then, doing atomic writes of multi gigabyte files doesn’t sound that hard to implement efficiently. Just write to disk first and update the metadata atomically at the end. Or whenever you choose to as a programmer. The downside is that, when overwriting, you’ll need enough free space to store both the old and new versio…

> Developers could decide by passing flags to open. Provided the underlying VFS has implemented them. They may not. Hence the point in the article that some developers only choose to support 'ext4' and nothing else. > you’ll need enough free space to store both the old and new versions of your data. The sacrifice is increased write wear on solid state devices. > It would allow all sorts of useful programs to be writt…

Yes, a feature like this would need cooperation with the filesystem. But that’s just an implementation problem. That’s like saying we can’t add flexbox to browsers because all the browsers would need to add it. So?

As for wear on SSDs, I don’t think it would increase wear. You’re writing the same number of sectors on the drive. A 2gb write would still write 2gb (+ negligible metadata overhead). Why would the drive wear out faster in this scheme?

And I think it would work way better with multiple processes than the existing system. Right now the semantics when multiple processes edit the same file at once are somewhat undefined. With this approach, files would have database like semantics where any reader would either see the state before a write or the state after. It’s much cleaner - since it would become impossible for skewed reads or writes to corrupt a shared file.

Would you argue against the existence of database transactions? Of course not. Nobody does. They’re a great idea, and they’re way easier to reason about and use correctly compared to the POSIX filesystem api. I’m saying we should have the same integrity guarantees on the filesystem. I think if we had those guarantees already, you’d agree too.

Re: Working with Files Is Hard (2019)

#69

Post by eviks »

continuational wrote: post #4

> Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually…

> why the file API so hard to use that even experts make mistakes?

Because it was poorly designed, and there is a high resistance to change, so those design mistakes from decades ago continue to bite

Re: Working with Files Is Hard (2019)

#70

Post by senderista »

kccqzy wrote: post #17
continuational wrote: post #4

> Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually…

By the way, LMDB's main developer Howard Chu responded to the paper. He said, > They report on a single "vulnerability" in LMDB, in which LMDB depends on the atomicity of a single sector 106-byte write for its transaction commit semantics. Their claim is that not all storage devices may guarantee the atomicity of such a write. While I myself filed an ITS on this very topic a year ago, http://www.openldap.org/its/inde…

This is called “Atomic Write Unit Power Failure” (AWUPF).
Post reply on HN