Live data from Hacker News

Working with Files Is Hard (2019)

danluu.com

41–50 of 123 posts

Re: Working with Files Is Hard (2019)

#41
post #32

Earlier quoted context omitted.

> make whole file read/writes atomic with a copy-on-write model, I have many files that are several GB. Are you sure this is a good idea? What if my application only requires best effort? > eliminate whole classes of filesystem bugs pretty quickly. Block level deduplication is notoriously difficult. > where the only kind of write allowed is to atomically append a chunk of data to the file Which sounds good until you…

It doesn’t have to be one or the other. Developers could decide by passing flags to open. But even then, doing atomic writes of multi gigabyte files doesn’t sound that hard to implement efficiently. Just write to disk first and update the metadata atomically at the end. Or whenever you choose to as a programmer. The downside is that, when overwriting, you’ll need enough free space to store both the old and new versio…

> Developers could decide by passing flags to open.

Provided the underlying VFS has implemented them. They may not. Hence the point in the article that some developers only choose to support 'ext4' and nothing else.

> you’ll need enough free space to store both the old and new versions of your data.

The sacrifice is increased write wear on solid state devices.

> It would allow all sorts of useful programs to be written easily

Sure. As long as you don't need multiple processes to access the same file simultaneously. I think the article misses this point, too, in that, every FS on a multi user system is effectively a "distributed system." It's not distributed for _redundancy_ but it doesn't eliminate the attendant challenges.

Re: Working with Files Is Hard (2019)

#42

I don't get it. The only times I've had problems with filesystem corruption in the past few decades was with a hardware problem, and said hardware was quickly replaced. FAT family has been perfectly fine while I've encountered corruption on every other FS including NTFS, exFAT, and the ext* family. Meanwhile you can read plenty of stories of others having the exact opposite experience. If you keep losing data to powe…

> If you keep losing data to power losses or crashes, perhaps fix the cause of that? I keep telling my users to make sure to plug their phones in before the battery dies, but for some reason they keep forgetting...

Then that's entirely their fault. They deserve all the corruption they get.

Re: Working with Files Is Hard (2019)

#44
post #38
post #34

Earlier quoted context omitted.

consumer Optane were not "power loss protected", that is every different than not honoring a requested a synchronous write. The crash-consistency problem is very different than the durability of real synchronous writes problem. There are some storage devices which will lie about synch writes, sometimes hoping that a backup battery will allow them to complete those write. System crashes are inevitable, use things like…

You're missing the point. GP was mentioning the common assumption that all systems in the last 30 years are sector-atomic under power loss condition. Either the sector is fully written or fully not written. Optane was a rare counter example, where sector can become partially written, thus not sector-atomic.

It is not rare for flash storage devices to lose data on power loss, even data that is FLUSH'd. See https://news.ycombinator.com/item?id=38371307

There are known cases where power loss during a write can corrupt previously written data (data at rest). This is not some rare occurrence. This is why enterprise flash storage devices have power loss protection.

See also: https://serverfault.com/questions/923971/is-there-a-way-to-p...

Re: Working with Files Is Hard (2019)

#45
post #30
post #22

Earlier quoted context omitted.

It's not hard to design a less bug-prone API that would enable you to do everything the POSIX file API permits and admits equally-high-performance implementations. But making that new API a replacement for the POSIX API would require rewriting essentially all of the software that somebody cares about to use your new, better API instead of the POSIX API. This is probably only feasible in practice for small embedded sy…

You could do a phased transition, where both the legacy posix api and the new api are available. This has already happened with a lot of the old C standard library. Old, unsafe functions like strcpy were gradually replaced by safer alternatives like strncpy. Database developers don’t want the complexity or poor performance of posix. It’s wild to me that we still don’t have any alternative to fsync in Linux that can a…

NVMe has no barrier that doesn't flush the pipeline/ringbuffer of IO requests submitted to it :(

Re: Working with Files Is Hard (2019)

#46
post #31

> On Linux ZFS, it appears that there's a code path designed to do the right thing, but CPU usage spikes and the system may hang or become unusable. ZFS fsync will not fail, although it could end up waiting forever when a pool faults due to hardware failures: https://papers.freebsd.org/2024/asiabsdcon/norris_openzfs-fs...

ZFS on Linux unfortunately has a long standing bug which makes it unusable under load: https://github.com/openzfs/zfs/issues/9130. 5.5 years old, nobody knows the root cause. Symptoms: under load (such as what one or two large concurrent rsyncs may generate over a fast network - that's how I encountered it) the pool begins to crap out and shows integrity errors and in some cases loses data (for some users - it never lost data for me). So if you do any high rate copies you _must_ hash-compare source and destination. This needs to be done after all the writes are completed to the zpool, because concurrent high rate reads seem to exacerbate the issue. Once the data is at rest, things seem to be fine. Low levels of load are also fine.

Re: Working with Files Is Hard (2019)

#47
post #44
post #38

Earlier quoted context omitted.

You're missing the point. GP was mentioning the common assumption that all systems in the last 30 years are sector-atomic under power loss condition. Either the sector is fully written or fully not written. Optane was a rare counter example, where sector can become partially written, thus not sector-atomic.

It is not rare for flash storage devices to lose data on power loss, even data that is FLUSH'd. See https://news.ycombinator.com/item?id=38371307 There are known cases where power loss during a write can corrupt previously written data (data at rest). This is not some rare occurrence. This is why enterprise flash storage devices have power loss protection. See also: https://serverfault.com/questions/923971/is-there-a…

I wish someone would sell an SSD that was at most a firmware update away between regular NVMe drive and ZNS NVMe drive. The latter just doesn't leave much room for the firmware to be clever and just swallow data.

Maybe also add a pSLC formatting mode for a namespace so one can be explicit about that capability...

It just has to be a drive that's useable as a generic gaming SSD so people can just buy it and have casual fun with it, like they did with Nvidia GTX GPUs and CUDA.

Re: Working with Files Is Hard (2019)

#49
post #12

> Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope write to files safely, like databases and version control systems: Leveldb, LMDB, GDBM, HSQLDB, Sqlite, PostgreSQL, Git, Mercurial, HDFS, Zookeeper. They then wrote a static analysis tool that can find incorrect usage of the file API, things like incorrectly assuming that operations that aren't atomic are actually…

> why the file API so hard to use that even experts make mistakes? Sounds like Worse Is Better™: operating systems that tried to present safer abstractions were at a disadvantage compared to operating systems that shipped whatever was easiest to implement. (I'm not an expert in the history, just observing the surface similarity and hoping someone with more knowledge can substantiate it.)

POSIX file locking is clearly modeled around whatever was simplest to implement, although it makes no sense at all.

Re: Working with Files Is Hard (2019)

#50

Earlier quoted context omitted.

Not really, there's been lots of APIs that have improved on the POSIX model. The kind of model I prefer is something based on atomicity. Most applications can get by with file-level atomicity--make whole file read/writes atomic with a copy-on-write model, and you can eliminate whole classes of filesystem bugs pretty quickly. (Note that something like writeFileAtomic is already a common primitive in many high-level fi…

> Most applications can get by with file-level atomicity--make whole file read/writes atomic with a copy-on-write model, and you can eliminate whole classes of filesystem bugs pretty quickly. int fd = open(".config", O_RDWR | O_CREAT | O_SYNC_ON_CLOSE, 0o666); // effects of calls to write(2)/etc. are invisible through any other file description // until the close(2) is called on all descriptors to this file descripti…

Surely this can’t always be true?

What happens when a lot of data is written and exceeds the dirty threshold?

Post reply on HN