Working with Files Is Hard (2019)

> Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually atomic, incorrectly assuming that operations that can be re-ordered will execute in program order, etc.

> When they did this, they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug. This isn't a knock on the developers of this software or the software -- the programmers who work on things like Leveldb, LBDM, etc., know more about filesystems than the vast majority programmers and the software has more rigorous tests than most software. But they still can't use files safely every time! A natural follow-up to this is the question: why the file API so hard to use that even experts make mistakes?

Re: Working with Files Is Hard (2019)

#5

Post by Retr0id » Thu, Jan 23, 2025, 5:46 PM UTC

> they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug.

This is why whenever I need to persist any kind of state to disk, SQLite is the first tool I reach for. Filesystem APIs are scary, but SQLite is well-behaved.

Of course, it doesn't always make sense to do that, like the dropbox use case.

Re: Working with Files Is Hard (2019)

#6

Post by ziddoap » Thu, Jan 23, 2025, 5:53 PM UTC

Retr0id wrote: post #5

> they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug. This is why whenever I need to persist any kind of state to disk, SQLite is the first tool I reach for. Filesystem APIs are scary, but SQLite is well-behaved. Of course, it doesn't always make sense to do that, like the dropbox use case.

>SQLite is the first tool I reach for.

Hopefully in whichever particular mode is referenced!

Re: Working with Files Is Hard (2019)

#7

Post by Retr0id » Thu, Jan 23, 2025, 5:55 PM UTC

ziddoap wrote: post #6

Retr0id wrote: post #5

> they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug. This is why whenever I need to persist any kind of state to disk, SQLite is the first tool I reach for. Filesystem APIs are scary, but SQLite is well-behaved. Of course, it doesn't always make sense to do that, like the dropbox use case.

> SQLite is the first tool I reach for. Hopefully in whichever particular mode is referenced!

WAL mode, yes!

Re: Working with Files Is Hard (2019)

#8

Post by praptak » Thu, Jan 23, 2025, 5:57 PM UTC

Ext4 actually special-handles the rename trick so that it works even if it should not:

"If auto_da_alloc is enabled, ext4 will detect the replace-via-rename and replace-via-truncate patterns and [basically save your ass]"[0]

[0]https://docs.kernel.org/admin-guide/ext4.html

Re: Working with Files Is Hard (2019)

#9

Post by Retr0id » Thu, Jan 23, 2025, 6:00 PM UTC

continuational wrote: post #4

> Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually…

> why the file API so hard to use that even experts make mistakes?

I think the short answer is that the APIs are bad. The POSIX fs APIs and associated semantics are so deeply entrenched in the software ecosystem (both at the OS level, and at the application level) that it's hard to move away from them.

Re: Working with Files Is Hard (2019)

#10

Post by __loam » Thu, Jan 23, 2025, 6:12 PM UTC

Retr0id wrote: post #9

continuational wrote: post #4

> Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually…

> why the file API so hard to use that even experts make mistakes? I think the short answer is that the APIs are bad. The POSIX fs APIs and associated semantics are so deeply entrenched in the software ecosystem (both at the OS level, and at the application level) that it's hard to move away from them.

POSIX is also so old and essential that it's hard to imagine an alternative.