Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix’s file durability problem

utcc.utoronto.ca

Post reply on HN
Print view View original

101–110 of 161 posts

Re: Unix’s file durability problem

#101

Post by NotSammyHagar »

hendzen wrote: post #37
derefr wrote: post #34

Earlier quoted context omitted.

If I were designing a userland from the ground up, I'd probably give processes a transactional MVCC object store , and make guarantees about that; and then implement a "POSIX compatibility layer" file system API in terms of that, but explicitly say that none of the same guarantees from the object-store layer apply. Some days I really do wish we weren't so inured to the particular 50-year-old systems-programming abstr…

Microsoft actually tried something similar with WinFS [0] in the 00's, but the project failed. [0] - https://en.wikipedia.org/wiki/WinFS

I was a developer on WinFS. It turns out to be hard to have a performant system that also has transactional commits between something changing a regular file, and a change in a database. It was also stupid to write OS components in .net, when it would just crash if it ran out of memory, among many other problems. Many people working on it wrote reports about how this was infeasible or impractical, yet they just said keep going, it will be all right.

I think we could have built something that worked well enough, but they grew cowardly after the general failure of the entire longhorn project. Don't try to change the compiler, the language, the os, the office application to use same "in motion" components, plus the database.

It's probably little known outside of Microsoft, but they tried to build many different higher end object filesystems, including JAWS (Jim Alchin Windows Filesystem), and others I can't even remember.

Re: Unix’s file durability problem

#102

Post by nickpsecurity »

xenophonf wrote: post #98
nickpsecurity wrote: post #66

Earlier quoted context omitted.

The concept was proven before w/ RMS in OpenVMS: https://en.wikipedia.org/wiki/Files-11 It could work. Just best to have hybrids with different types of files, including those bypassing RDBMS function, so one can select proper reliability vs performance tradeoffs. SQLite might have cross-platform, FS-type API's too that I don't know about. Not sure if it's already there or be an extra development.

I really miss OpenVMS. It's a damn shame it doesn't run on x86-64. I might try to set up one of the Alpha AXP emulators that are floating around out there, if only to feed my nostalgia.

This company has licensed it from HP to update it and port it to Xeon:

http://www.vmssoftware.com/index.html

They're working on it as we speak. :)

Re: Unix’s file durability problem

#103

Post by jstimpfle »

ryao wrote: post #89
kentonv wrote: post #79

Earlier quoted context omitted.

> How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. Note that Linux ext4 does not do this. On a power outage, you can get bogus trailing zeros on a file which you were appending, because the file size was updated before the data was written. I asked Ted T'so about this and he said it was working as intended. https://plus.google.com/+K…

ZFS does do this. Changes are atomic, so they either happen or do not. There is no in-between.

I don't know about ZFS, but the usual write(2) API does not support this: It might for example return early with a short write because some interrupt occurred. Can happen for all "slow devices" (see signal(7)). And I think that's a good thing and am sure many programs expect this.

Re: Unix’s file durability problem

#104

Post by GauntletWizard »

schmichael wrote: post #82

Disks can buffer, disks can have firmware bugs, disks can fail both catastrophically and subtly. Even the fanciest battery-backed RAID controllers have firmware bugs and dead batteries. Those are just ways your storage hardware can fail you even if the kernel and libraries are bug free and you follow the right mystic incantations to sync data. While there's no excuse for bad documentation or poor APIs, you can never…

Yeah, from a SRE perspective, the last N writes are always purely probabilistic. The real quest is to have enough redundancy that that curve is fairly close to 1, and enough failure warning that your system will fix itself before it droops. That means Battery Backup, SMART detection, ECC memory, etc.

Re: Unix’s file durability problem

#105

Post by haberman »

avar wrote: post #99
haberman wrote: post #94

Earlier quoted context omitted.

> "Log" files can only be appended. Writers cannot seek. I think the semantic you want for log files is: write atomically at offset X, but only if the size of the file is currently X. If the write fails due to a file size mismatch, you are racing with someone else. You can either read the other writer's data before you try writing again or just fail immediately if races shouldn't happen.

I don't mean to sound dismissive, but do you have any idea of how O_APPEND interacts with PIPE_BUF? The problem you're describing is basically solved on modern POSIX OS's. You're just proposing moving the semantics of what could be better done in kernel-space by resizing PIPE_BUF to userspace. The way this works now is that you open a logfile with O_APPEND, any write(2) you make to the file will be atomic up to PIPE_…

> I don't mean to sound dismissive, but do you have any idea of how O_APPEND interacts with PIPE_BUF?

Nope. I was replying to one proposed semantic and arguing that the ideal semantic is slightly different. I haven't looked deeply into how such a semantic could or could not be mapped onto current APIs.

> at the point where you're making >4K writes you're usually better off having some custom log system anyway and not rely on the kernel serializing things for you

I disagree. The kernel has access to the I/O queue, and I don't. The kernel can fundamentally do this more efficiently and robustly than user space can.

So it sounds like my options are to limit my write size, serialize writers in user space (impossible generally unless you can physically prohibit processes from running that don't follow the serialization protocol) or flock(), which "sucks." Sounds like there is room for an api that does what is desired but doesn't suck.

EDIT: I just read about O_APPEND and it doesn't sound like what I'm talking about at all. It always appends to the end. It sounds like if two writers race, both chunks of new data are appended. That's not as useful as what I described. I'm basically talking about compare and swap, except instead of compare and swap its compare and append. If something changed in the meantime you don't want to just blindly append your new block that is unaware of the concurrently appended log data.

Re: Unix’s file durability problem

#106

Post by rdtsc »

planckscnst wrote: post #45
rdtsc wrote: post #28

Earlier quoted context omitted.

> How do you actually do that for sure? It's not write() or even necessarily fsync(). Right. For decision of "when" you can configure a few kernel parameters, namely dirty writes thresholds. That assumes you just do write()s from your process and kernel decides to flush that data out. That can be configured to be a function of time and or amount of unwritten data. I had to do it a few times. Once it was a realtime-is…

That's still missing the point. Say you're writing a program that stores a log of transactions. People are trading resources and you log those transactions so you can find out who owns which resources. User A just gave 100 units to user B; you need to store this information and tell both users that the transaction is complete. How do you do that? Your program will do something like "write(t_log_fd, t_entry, t_bytes);…

I guess I have misunderstood, your question about "The article wasn't about the decision of when to write to disk".

I thought you are asking how to control in general when your system is writing to disk.

Well if you don't use fsync then it decides at some point like I mentioned above. If want to talk about transactions then do a write and fsync, does that not work someetimes fro you? If you are worried about the new file appearing in the directory then fsync the directory as well.

If you want more guarantees, you'll have to dig deeper and find out about your specific device, does it have battery power and does it tell lies about it writing data and so on.

Re: Unix’s file durability problem

#107

Post by yuubi »

userbinator wrote: post #29
wanderr wrote: post #14

Earlier quoted context omitted.

Backups do not solve the problem of not knowing whether data you just wrote to disk will still be there in the event of a power outage or system crash. You know, before a backup has a chance to run. Sure, those events should be extremely rare but that doesn't mean that we can or should just ignore it, at a large enough scale even extremely improbable events are guaranteed to happen.

Backups do not solve the problem of not knowing whether data you just wrote to disk will still be there in the event of a power outage or system crash. It's not a "problem", it's just reality. You can't predict when exactly the crash will occur relative to the disk write or backups.

In many applications, crashes are less of a problem than falsely believing a write happened persistently. Consider a mail server: if it can wait until the message is really saved before returning the final success response, then a crash means a retry later (and maybe a duplicated message if the crash happened in the small window between the write committing and the response going out) rather than a lost message.

Re: Unix’s file durability problem

#108

Post by ymse »

rwmj wrote: post #55
leni536 wrote: post #31

What about O_DIRECT | O_SYNC options in linux's open()?

O_SYNC has abysmal performance. O_DIRECT has very underspecified semantics, but particularly it demands that you only read and write whole "blocks" (where "block" depends on the filesystem type, underlying block device and phase of the moon).

See also Linux Torvalds' opinion on O_DIRECT:

The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances.

https://lkml.org/lkml/2002/5/11/58

Re: Unix’s file durability problem

#109

Post by asveikau »

Animats wrote: post #67
jstimpfle wrote: post #59

Earlier quoted context omitted.

> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2) : If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing"…

Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.) O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly. How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating t…

> NTFS doesn't do append correctly.

Care to elaborate on that? I haven't seen issues with the two mechanisms I know of in NT:

1. CreateFile with FILE_APPEND_DATA

2. WriteFile with the OVERLAPPED offset set to ~0

Both of these get you writes that are as far as I know safe to have multiple processes writing to a common log file. Not sure about flushes to disk but you could probably get away with FlushFileBuffers for this.

Re: Unix’s file durability problem

#110

Post by Sanddancer »

frutiger wrote: post #83
Sanddancer wrote: post #62

Earlier quoted context omitted.

The problem is that fsync() runs outside of the control of the program. There's no way for an application to start a transaction, perform steps x, y, and z, and then end a transaction, rolling back to before step x if there are any failures. For example, suppose you're rotating a an audit log file at the same moment your backup program is running. Your backup program reads the directory, and at that same moment, your…

Realistically, backups need to be based on atomic snapshots, like `zfs`.

Yep. I'll admit my example was a bit convoluted, but I was trying to show a way in which common tasks can race and create undesirable situations. ZFS snapshots would definitely make things quite a bit more predictable.
Post reply on HN