Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Removing fsync from our local storage engine

fractalbits.com

Post reply on HN
Print view View original

71–80 of 88 posts

Re: Removing fsync from our local storage engine

#71

Post by klodolph »

asdfasgasdgasdg wrote: post #20
klodolph wrote: post #14

Earlier quoted context omitted.

:-/ it’s a statistical guarantee in the first place. A successful commit in a durable storage engine just needs to achieve some finite level of durability, like “10^-7 probability of loss per year”. The durability is a property of the whole system, and it is possible to achieve durability without fsync, you just may have a hard time explaining what the durability is, how you calculated it, and what the evidence or ju…

10^-7 (loss/record) * 10^8 (record/year) yields 10 data losses per year. If you're even a medium sized business you need a much better than 10^-7 probability of losses.

The half-remembered storage system I pulled those numbers from had records ~100G in size, so a 10^-7 loss is 1 loss event per year, per exabyte of data. A loss event is just “at least one bit in the record cannot be read within a certain deadline”.

Durability is a knob. If you have enough data, or turn the knob too far in the direction of durability, you will simply bankrupt yourself or maybe drown your service in latency. It makes sense that you would have storage services that provide different levels of durability.

Re: Removing fsync from our local storage engine

#73

Post by klodolph »

asdfasgasdgasdg wrote: post #42
Dylan16807 wrote: post #41

Earlier quoted context omitted.

That's only true if your typical loss event loses one record. If you have a one in a million chance of an array failure taking out 10% of your production database, and otherwise have zero possibility of data loss, you also get 10^-7 losses per record. And I wouldn't assume they meant that number to be per record in the first place.

I don't think anyone in history has ever achieved a true 10^-7 annual probability of any data loss incident. So they must have been making some kind of per record or per operation claim.

I like to think that the true AFR for data is bounded by something like 10^-3, because maybe that’s close to the rate at which civilizations collapse. You have to use a kind of subtle definition to support 10^-7 or 10^-9 or 10^-11. Or maybe instead of “subtle definition”, you can call it a “whimsical, imaginary definition”. Depends on how cynical you are.

The way I would go is by saying that you multiply the number of objects by AFR, and that’s close to the actual losses on most years. You can then exclude WW3 and the late holocene extinction event from your consideration. Or simple bankruptcy, for that matter. If your employer is gone, you don’t care about its data any more.

Re: Removing fsync from our local storage engine

#75

Post by atombender »

zzsheng wrote: post #2

Author here. This is not a general argument against fsync; the design depends on SSD-only deployment, preallocated files, O_DIRECT, single-key atomicity, and device write guarantees.

I'm surprised none of the design decisions considered an indirection between the folder tree structure and the actual files.

For example, if you map folders like /foo and /foo/bar to numeric IDs, then each file can simply refer their parent folder. Renaming a folder, or moving a folder to a new parent, does not need to update any files.

You can take this a step further and have a three-level split: Tree, file-tree join table, and files. The tree describes the hierarchical structure of folders (which changes more rarely than files do), while the file-tree join table is essentially [folder_id, file_id]. When a file is moved, only the join table (which is much smaller than the files and super sortable and compressible) must be updated.

I take the point that updating multiple discrete pieces of information puts more demand on the transactional layer, which has to ensure atomicity and consistency. But I'm surprised it wasn't even mentioned as an alternative that was evaluated and rejected. The article starts out with the premise that a flat key/value approach is the only choice on the table.

Re: Removing fsync from our local storage engine

#78

Post by jmalicki »

nh2 wrote: post #49
thomas_fa wrote: post #25

Earlier quoted context omitted.

Yes, as we mentioned in the post, it is targeted for the virtualized NVME disk and we don't have control for actually issing FUA command. We are also changing to open data files with O_DATA_SYNC to make them work with normal on-prem deployment environments.

Even then, I also share the confusion of the poster you're replying to. I don't see how a virtualised NVMe disk is different from a physical one. Especially if you don't have control over the underlying hardware (so you don't know if it has power-loss-protection PLP SSDs), you should send the FUA. > O_DATA_SYNC You mean `O_DSYNC`? Why would you need `O_DSYNC` on-premise, but not on cloud VMs? (Or are you saying you'd…

Fsync on PLP drives isn't strictly a NOP - you still take a latency hit from the round trip of the command to the NVMe device, where it is implemented as a NOP.

Re: Removing fsync from our local storage engine

#79

Post by zzsheng »

zzsheng wrote: post #59
bawolff wrote: post #13

Am i understanding correctly that you are just targeting consistency and not durability?

actually both crash consistentcy and durability. after we ack, we make sure data will be lost due to crash, restart or power loss.

sorry, typo. data will *not* be lost.

Re: Removing fsync from our local storage engine

#80

Post by zbentley »

nh2 wrote: post #7

> fsync doesn’t just sync the file’s data, it syncs every piece of metadata the file depends on: ... directory entry Famously not, as the man page says. It is also said later in the article: > POSIX strictly requires a parent-directory fsync to make a newly created file’s existence durable. So I'm not sure why the dirent sync is claimed earlier.

I’m curious: what happens when a file and its metadata are fsync’d, but the dirent is not and the system loses power? If the file is brand new, does it show up as an orphan upon reboot, or is it just gone? Can you somehow access it by inode even if it’s not findable? Is this filesystem specific?

Is there something else weird that can happen if the file is not new, not unlinked, but changes are made that would alter the directory entry in it in some way?

Post reply on HN