This is really great work. Kudos to the team for such an elegant solution.
Thanks for the kind words! You check more of our work in https://github.com/fractalbits-labs/fractalbits.
Removing fsync from our local storage engine
31–40 of 88 posts
Re: Removing fsync from our local storage engine
#32Unless I am mistaken, it seems like there is a glaring flaw in this scheme, which is that without fsync you cannot guarantee the previous WAL blocks have been persisted before the current one, so a power loss event could leave a hole in the log and cause erroneous recovery. I believe that SSDs reorder writes internally so even having atomic batched O_DIRECT is not a strong enough guarantee for durability. I'll admit that I could be misunderstanding something about the system that alleviates this concern.
Re: Removing fsync from our local storage engine
#33So basically, you are writing data without guarantees it's actually written? "YOLO mode" but for data written to a device?
Would you be so kind to explain what happens in a power-loss scenario?
Re: Removing fsync from our local storage engine
#34So basically, you are writing data without guarantees it's actually written? "YOLO mode" but for data written to a device? Would you be so kind to explain what happens in a power-loss scenario?
[deleted]
Re: Removing fsync from our local storage engine
#35Unless I am mistaken, it seems like there is a glaring flaw in this scheme, which is that without fsync you cannot guarantee the previous WAL blocks have been persisted before the current one, so a power loss event could leave a hole in the log and cause erroneous recovery. I believe that SSDs reorder writes internally so even having atomic batched O_DIRECT is not a strong enough guarantee for durability. I'll admit…
I also asked what happens when a power loss happens.
Re: Removing fsync from our local storage engine
#36> fsync doesn’t just sync the file’s data, it syncs every piece of metadata the file depends on: ... directory entry Famously not, as the man page says. It is also said later in the article: > POSIX strictly requires a parent-directory fsync to make a newly created file’s existence durable. So I'm not sure why the dirent sync is claimed earlier.
Thanks for pointing it out the mistakes. We should make it clearer, when fsync an opened file descriptor, it would only sync its own metadata. To make it truly persistent, we need to issue another fsync for the directory fd, which would make it more expensive.
You don't need to do that for every write though. Only when the database file is created.
Re: Removing fsync from our local storage engine
#37This seems sketchy. O_DIRECT skips the operating system's page cache, it does not guarantee that the SSD driver sent the data to the SSD or issued a flush to the drive itself. The data could still be in the driver's memory or the in non-durable memory in the drive itself when this engine says "ok, we're good". EDIT: sketchy from an answering "what exactly are the guarantees?" perspective
The model here is that the storage device is directly reading and writing the userspace buffer via DMA. It is one of the reasons use of O_DIRECT creates additional constraints on buffer alignment and size.
Some storage devices guarantee durability of non-persisted writes, which is explicitly part of their model. Consequently, the entire durable write path is the storage device completing a DMA read of their buffer.
The underlying assumptions will not hold true for every environment. However, it will hold true for many and you can check most (all?) of them at runtime.
Re: Removing fsync from our local storage engine
#38This design ACKs writes that aren't yet durably persisted (to the journal or data areas). That might be ok, but it might not. It's certainly unusual not to at least persist the journal update.
Re: Removing fsync from our local storage engine
#39There’s lies, damn lies, and lies that disks tell the operating system. Don’t believe any of them!
If you need to know it’s been persisted to non-volatile storage then you need to own the full stack of every piece of software between the OS and the actual physical memory.
Every managed flash drive is going to have layers and layers of complexity and caching and things you simply can’t easily control or really understand. Don’t trust it unless you know exactly how it works all the way down.
Re: Removing fsync from our local storage engine
#40Earlier quoted context omitted.
Thanks for pointing it out the mistakes. We should make it clearer, when fsync an opened file descriptor, it would only sync its own metadata. To make it truly persistent, we need to issue another fsync for the directory fd, which would make it more expensive.
You don't need to do that for every write though. Only when the database file is created.
Yes, especially for our object storage each putObject would need to create new entry for in the (data)name space which would need fsync for dir fd.