Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Removing fsync from our local storage engine

fractalbits.com

Post reply on HN
Print view View original

11–20 of 88 posts

Re: Removing fsync from our local storage engine

#11

Post by 100ms »

zzsheng wrote: post #2

Author here. This is not a general argument against fsync; the design depends on SSD-only deployment, preallocated files, O_DIRECT, single-key atomicity, and device write guarantees.

Your approach looks interesting but I was curious when you talk about path-based splitting for ART, do you literally mean always on "/"? I know S3 directory buckets always use /, but the classical S3 model had no natural separator character and I was wondering if supporting those styles of prefix or custom delimiter queries suffered any impediment in your approach.

Bookmarked your whole blog for later consumption, interesting stuff!

Re: Removing fsync from our local storage engine

#14

Post by klodolph »

matja wrote: post #10

Even with O_DIRECT and aligned blocks, I still don't understand how the storage engine can return a "successful commit" to the client without a sync at some point, because a sync (IIRC) is the only way to guarantee an ATA/NVMe FUA command is sent, and the device write cache/buffer is committed.

:-/ it’s a statistical guarantee in the first place. A successful commit in a durable storage engine just needs to achieve some finite level of durability, like “10^-7 probability of loss per year”. The durability is a property of the whole system, and it is possible to achieve durability without fsync, you just may have a hard time explaining what the durability is, how you calculated it, and what the evidence or justifications are for the numbers you give.

Even if you just look at hardware failure rates, you get unrecoverable I/O errors (data corruption) at about one in 10^15 bits, disk failures at a rate of about 1% per year, etc. People usually like to have better guarantees than those numbers give you with just a plain fsync anyway; so you are probably forced to do an analysis of the whole system if you want to provide good durability guarantees and be able to explain where the guarantees come from.

Re: Removing fsync from our local storage engine

#15

Post by tptacek »

dale_glass wrote: post #8
dboreham wrote: post #4

Almost full-circle back to when Oracle took over the entire volume and implemented its own filesystem.

I wonder why this is not more common. LVM is easy to set up, and it's already common to allocate volumes for things like disk images for VMs, so why not databases?

If you preallocate and O_DIRECT, haven't you basically soaked up most of the benefit of skipping the filesystem?

Re: Removing fsync from our local storage engine

#16

Post by binaryturtle »

matja wrote: post #10

Even with O_DIRECT and aligned blocks, I still don't understand how the storage engine can return a "successful commit" to the client without a sync at some point, because a sync (IIRC) is the only way to guarantee an ATA/NVMe FUA command is sent, and the device write cache/buffer is committed.

To truly guarantee things you probably also would need an uncached read afterwards (to verify the data comes back properly from the device). Now that would kill any sort of performance, of course.

Re: Removing fsync from our local storage engine

#17

Post by [deleted] »

100ms wrote: post #11
zzsheng wrote: post #2

Author here. This is not a general argument against fsync; the design depends on SSD-only deployment, preallocated files, O_DIRECT, single-key atomicity, and device write guarantees.

Your approach looks interesting but I was curious when you talk about path-based splitting for ART, do you literally mean always on "/"? I know S3 directory buckets always use /, but the classical S3 model had no natural separator character and I was wondering if supporting those styles of prefix or custom delimiter queries suffered any impediment in your approach. Bookmarked your whole blog for later consumption, in…

[deleted]

Re: Removing fsync from our local storage engine

#20

Post by asdfasgasdgasdg »

klodolph wrote: post #14
matja wrote: post #10

Even with O_DIRECT and aligned blocks, I still don't understand how the storage engine can return a "successful commit" to the client without a sync at some point, because a sync (IIRC) is the only way to guarantee an ATA/NVMe FUA command is sent, and the device write cache/buffer is committed.

:-/ it’s a statistical guarantee in the first place. A successful commit in a durable storage engine just needs to achieve some finite level of durability, like “10^-7 probability of loss per year”. The durability is a property of the whole system, and it is possible to achieve durability without fsync, you just may have a hard time explaining what the durability is, how you calculated it, and what the evidence or ju…

10^-7 (loss/record) * 10^8 (record/year) yields 10 data losses per year. If you're even a medium sized business you need a much better than 10^-7 probability of losses.
Post reply on HN