Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Working with Files Is Hard (2019)

danluu.com

Post reply on HN
Print view View original

51–60 of 123 posts

Re: Working with Files Is Hard (2019)

#51

Post by tjalfi »

ncruces wrote: post #49
dkarl wrote: post #12

Earlier quoted context omitted.

> why the file API so hard to use that even experts make mistakes? Sounds like Worse Is Better™: operating systems that tried to present safer abstractions were at a disadvantage compared to operating systems that shipped whatever was easiest to implement. (I'm not an expert in the history, just observing the surface similarity and hoping someone with more knowledge can substantiate it.)

POSIX file locking is clearly modeled around whatever was simplest to implement, although it makes no sense at all.

Jeremy Allison tracked down why POSIX standardized this behavior[0].

The reason is historical and reflects a flaw in the POSIX standards process, in my opinion, one that hopefully won't be repeated in the future. I finally tracked down why this insane behavior was standardized by the POSIX committee by talking to long-time BSD hacker and POSIX standards committee member Kirk McKusick (he of the BSD daemon artwork). As he recalls, AT&T brought the current behavior to the standards committee as a proposal for byte-range locking, as this was how their current code implementation worked. The committee asked other ISVs if this was how locking should be done. The ISVs who cared about byte range locking were the large database vendors such as Oracle, Sybase and Informix (at the time). All of these companies did their own byte range locking within their own applications, none of them depended on or needed the underlying operating system to provide locking services for them. So their unanimous answer was "we don't care". In the absence of any strong negative feedback on a proposal, the committee added it "as-is", and took as the desired behavior the specifics of the first implementation, the brain-dead one from AT&T.

[0] https://www.samba.org/samba/news/articles/low_point/tale_two...

Re: Working with Files Is Hard (2019)

#52

Post by wmf »

ablob wrote: post #19
nodamage wrote: post #16

Earlier quoted context omitted.

Before becoming too overconfident in SQLite note that Rebello et al. ( https://ramalagappan.github.io/pdfs/papers/cuttlefs.pdf ) tested SQLite (along with Redis, LMDB, LevelDB, and PostgreSQL) using a proxy file system to simulate fsync errors and found that none of them handled all failure conditions safely. In practice I believe I've seen SQLite databases corrupted due to what I suspect are two main causes: 1. The…

I believe it is impossible to prevent dataloss if the device powers off during a write. The point about corruption still stands and appears to be used correctly from what I skimmed in the paper. Nice reference.

If the file system uses strict COW it should survive that situation.

Re: Working with Files Is Hard (2019)

#53

Post by wmf »

pjdesno wrote: post #15
wruza wrote: post #3

No mention on ntfs and windows keywords in the article, for those interested.

Although the conference this was presented at is platform-agnostic, the author is an expert on Linux, and the motivation for the talk is Linux-specific. (Dropbox dropping support for non-ext4 file systems) The post supports its points with extensive references to prior research - research which hasn't been done in the Microsoft environment. For various reasons (NDAs, etc.) it's likely that no such research will ever…

Universities can get Windows source code under NDA and do research on it but nobody really cares about such work.

Re: Working with Files Is Hard (2019)

#54

Post by userbinator »

userbinator wrote: post #42
nodamage wrote: post #35

Earlier quoted context omitted.

> If you keep losing data to power losses or crashes, perhaps fix the cause of that? I keep telling my users to make sure to plug their phones in before the battery dies, but for some reason they keep forgetting...

Then that's entirely their fault. They deserve all the corruption they get.

Seems like I hit a nerve. Apparently teaching users responsibility is a bad thing?

No wonder things are "hard". Because otherwise many in this godforsaken industry wouldn't need to be employed.

Re: Working with Files Is Hard (2019)

#55

Post by emmelaich »

__loam wrote: post #10
Retr0id wrote: post #9

Earlier quoted context omitted.

> why the file API so hard to use that even experts make mistakes? I think the short answer is that the APIs are bad. The POSIX fs APIs and associated semantics are so deeply entrenched in the software ecosystem (both at the OS level, and at the application level) that it's hard to move away from them.

POSIX is also so old and essential that it's hard to imagine an alternative.

Some of the problems transcend POSIX. Someone I know maintains a non-relational db on IBM mainframes. When diving into a data issue, he was gob-smacked to find out that sync'd writes did not necessarily make it to the disk. They were cached in the drive memory and (I think) the disk controller memory. If all failed, data was lost.

Re: Working with Files Is Hard (2019)

#56

Post by ncruces »

tjalfi wrote: post #51
ncruces wrote: post #49

Earlier quoted context omitted.

POSIX file locking is clearly modeled around whatever was simplest to implement, although it makes no sense at all.

Jeremy Allison tracked down why POSIX standardized this behavior[0]. The reason is historical and reflects a flaw in the POSIX standards process, in my opinion, one that hopefully won't be repeated in the future. I finally tracked down why this insane behavior was standardized by the POSIX committee by talking to long-time BSD hacker and POSIX standards committee member Kirk McKusick (he of the BSD daemon artwork). A…

The most egregious part of it for me is that if I open and close a file I might be canceling some other library's lock that I'm completely unaware of.

I resisted using them in my SQLite VFS, until I partially relented for WAL locks.

I wish more platforms embraced OFD locks. macOS has them, but hidden. illumos fakes them with BSD locks (which is worse, actually). The BSDs don't add them. So it's just Linux, and Windows with sane locking. In some ways Windows is actually better (supports timeouts).

Re: Working with Files Is Hard (2019)

#57

Post by hackit2 »

emmelaich wrote: post #55
__loam wrote: post #10

Earlier quoted context omitted.

POSIX is also so old and essential that it's hard to imagine an alternative.

Some of the problems transcend POSIX. Someone I know maintains a non-relational db on IBM mainframes. When diving into a data issue, he was gob-smacked to find out that sync'd writes did not necessarily make it to the disk. They were cached in the drive memory and (I think) the disk controller memory. If all failed, data was lost.

Just wait till he has to deal with raid controllers.

Re: Working with Files Is Hard (2019)

#58

Post by PaulHoule »

userbinator wrote: post #28

I don't get it. The only times I've had problems with filesystem corruption in the past few decades was with a hardware problem, and said hardware was quickly replaced. FAT family has been perfectly fine while I've encountered corruption on every other FS including NTFS, exFAT, and the ext* family. Meanwhile you can read plenty of stories of others having the exact opposite experience. If you keep losing data to powe…

There was that time (2009 or so?) I wrote 2 million files to a single directory on NTFS and that filesystem was never the same again. It didn't seem to be a hardware problem. I used to be really careful to not put a crazy number of files in a directory on Linux and Windows storing them in subdirs like

  b7/b74a/b74a56
where the digits are derived from a hash of the file name but lately I've had some NTFS volumes with a 1M file directory that seem to be OK.

Hardware problems also manifest in mysterious ways. On both Windows and MacOS I had computers that seemed to be OK until I did an OS update which caused enough IO that a failing HDD was pushed over the edge and the update failed; in one case I was able to roll back the update but not apply the update, in another case the machine was trashed. Careful investigation (like taking the disk out and inspecting it on another computer) revealed a hard drive error although there was no clear indication of this in the UI and the average person would blame to software update

Re: Working with Files Is Hard (2019)

#59

Post by huntaub »

Retr0id wrote: post #9
continuational wrote: post #4

> Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually…

> why the file API so hard to use that even experts make mistakes? I think the short answer is that the APIs are bad. The POSIX fs APIs and associated semantics are so deeply entrenched in the software ecosystem (both at the OS level, and at the application level) that it's hard to move away from them.

I take a different view on this. IMO the tricks that existing file systems play to get more performance (specifically around ordering and atomicity) make it extra hard for developers to reason about. Obviously, you can't do anything about fsync dropping error codes, but some of these failure modes just aren't possible over file systems like NFS due to protocol semantics.

Re: Working with Files Is Hard (2019)

#60

Post by huntaub »

ryao wrote: post #39
josephg wrote: post #32

Earlier quoted context omitted.

It doesn’t have to be one or the other. Developers could decide by passing flags to open. But even then, doing atomic writes of multi gigabyte files doesn’t sound that hard to implement efficiently. Just write to disk first and update the metadata atomically at the end. Or whenever you choose to as a programmer. The downside is that, when overwriting, you’ll need enough free space to store both the old and new versio…

Packages consist of multiple files. An atomic file write would not allow packages to be either installed or not installed by APT.

This is kind of an interesting thought that more mirrors how Docker uses OverlayFS to track changes to the entire file system. No need for new file APIs.
Post reply on HN