Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Removing fsync from our local storage engine

fractalbits.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Removing fsync from our local storage engine

#51

Post by convolvatron »

mightyham wrote: post #48
convolvatron wrote: post #46

Earlier quoted context omitted.

if there is a hole in the log then the end of the log is before the hole. you do have to have checksums on log chunks, and better a kind of rolling hash, but you're really just talking about he number of entires that we would have liked to commit but didn't

Yeah this is a good point, and maybe a hole wasn't the right way to explain myself. The point is that the way a WAL is supposed to work is that the main data store always lags behind the WAL, so that if a partial operation (always idempotent) occurs on shutdown it is replayed on start up and fixed. In the case I describe, because of a lack of fsync it's possible for the WAL to lag the main data store, so partial oper…

that's a much more interesting problem. fundamentally we're in a bad position by having two different formats, one optimized for writing and one for reading, that admit inconsistency between them. Postgres mitigates this slightly by having page level updates to the read indices also be present in the log (physiological), but that's always seemed like a huge waste to me.

if we give ourselves two definitions of persisted - logically(wal or write) and physically (index or read), it seems like we can maintain the invariant that P edit: of course one of the root problems here is the drive lying, so how can we understand that some log block has actually commit so that we can update P

Re: Removing fsync from our local storage engine

#52

Post by thomas_fa »

nh2 wrote: post #49
thomas_fa wrote: post #25

Earlier quoted context omitted.

Yes, as we mentioned in the post, it is targeted for the virtualized NVME disk and we don't have control for actually issing FUA command. We are also changing to open data files with O_DATA_SYNC to make them work with normal on-prem deployment environments.

Even then, I also share the confusion of the poster you're replying to. I don't see how a virtualised NVMe disk is different from a physical one. Especially if you don't have control over the underlying hardware (so you don't know if it has power-loss-protection PLP SSDs), you should send the FUA. > O_DATA_SYNC You mean `O_DSYNC`? Why would you need `O_DSYNC` on-premise, but not on cloud VMs? (Or are you saying you'd…

Thanks for the feedback, since I have relied in other thread related to O_DSYNC which a lot of folks have already suggested, and I will not repeat it here.

For the benchmark results, and they were mainly due to metadata management. We have implemented our own KV store, see internal here [1], which is more efficient than ext4 namespace management, even after doing very aggressive fs tuning for that [2] (plus 65536 sharding for each leveled dir).

[1] https://fractalbits.com/blog/metadata-engine-for-our-object-...

[2] https://github.com/fractalbits-labs/fractalbits/commit/12109...

Re: Removing fsync from our local storage engine

#53

Post by thomas_fa »

bradfa wrote: post #39

There’s lies, damn lies, and lies that disks tell the operating system. Don’t believe any of them! If you need to know it’s been persisted to non-volatile storage then you need to own the full stack of every piece of software between the OS and the actual physical memory. Every managed flash drive is going to have layers and layers of complexity and caching and things you simply can’t easily control or really underst…

Well said and there are some bitter lessons in the storage industry.

In my last company we need to disable the disk write cache during each reboot, and we also heard a lot industry stories related to underneath firmware implementation from oxide computer podcasts [1]. Yes, to provide truly reliable service, we need to evaluate underneath hardware settings case-by-case.

[1] https://onthemetal.transistor.fm/

Re: Removing fsync from our local storage engine

#54

Post by uroni »

In my similar project (s3 compatible single-node storage) https://github.com/uroni/hs5 I do use proper fsync for data and metadata durability. But it can be turned of via switch. It is a pet peeve of mine that the defaults should always be to fsync. I do have a section on this in my README of the project.

I also do have an optional WAL. Maybe I should add an additional mode that disables fsync only for the WAL. I don't think it would be a good idea. My WAL does use checksums and sequence numbers etc. to prevent committing wrong data.

Re: Removing fsync from our local storage engine

#55

Post by sethev »

jandrewrogers wrote: post #37
sethev wrote: post #26

This seems sketchy. O_DIRECT skips the operating system's page cache, it does not guarantee that the SSD driver sent the data to the SSD or issued a flush to the drive itself. The data could still be in the driver's memory or the in non-durable memory in the drive itself when this engine says "ok, we're good". EDIT: sketchy from an answering "what exactly are the guarantees?" perspective

The model here is that the storage device is directly reading and writing the userspace buffer via DMA. It is one of the reasons use of O_DIRECT creates additional constraints on buffer alignment and size. Some storage devices guarantee durability of non-persisted writes, which is explicitly part of their model. Consequently, the entire durable write path is the storage device completing a DMA read of their buffer. T…

Right - I mean, what you're describing makes sense, but it doesn't sound like what they're describing. Their benchmarks are running on an EC2 instance and the post's author is here saying that they run on virtualized hardware. Plus they run on top of a file system. None of that screams "direct DMA from our buffers" to me.

I'm not saying it's impossible, but typically people who want to lean on hardware guarantees for extra performance control more of the stack.

Re: Removing fsync from our local storage engine

#56

Post by zzsheng »

mightyham wrote: post #32

Unless I am mistaken, it seems like there is a glaring flaw in this scheme, which is that without fsync you cannot guarantee the previous WAL blocks have been persisted before the current one, so a power loss event could leave a hole in the log and cause erroneous recovery. I believe that SSDs reorder writes internally so even having atomic batched O_DIRECT is not a strong enough guarantee for durability. I'll admit…

thanks for feedback. actually it was pointed out in blog that we do not use append only log to avoid fsync due to size change. what we use is preallocate fixed size log file and we do write journal data and space reclaim by 4KB unit, also with direct-io.

Re: Removing fsync from our local storage engine

#60

Post by zzsheng »

loeg wrote: post #38

This design ACKs writes that aren't yet durably persisted (to the journal or data areas). That might be ok, but it might not. It's certainly unusual not to at least persist the journal update.

nop. we will not ack any write which is not in data or journal. please check the put details in the blog.
Post reply on HN