Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async I/O on Linux in databases

blog.canoozie.net

Post reply on HN
Print view View original

81–90 of 100 posts

Re: Async I/O on Linux in databases

#81

Post by gpderetta »

quietbritishjim wrote: post #65
loeg wrote: post #40

Earlier quoted context omitted.

> As I said, I don't know anything about fsync in io_uring. Maybe that has now control? io_uring fsync has byte range support: https://man7.org/linux/man-pages/man2/io_uring_enter.2.html#...

Sorry, that was a typo in my comment (now edited). "Now" was meant to be "more" i.e. "perhaps [io_uring] has more control [than sync APIs]?" Byte range is support is interesting but also present in the Linux sync API: https://man7.org/linux/man-pages/man2/sync_file_range.2.html I meant more like, perhaps it's possible to concurrently queue fsync for different writes in a way that isn't possible with the sync API. Fro…

You can insert synchronization OPs (i.e. barriers) in the queue to guarantee in-order execution.

Re: Async I/O on Linux in databases

#82

Post by loeg »

somat wrote: post #76
jandrewrogers wrote: post #71

Earlier quoted context omitted.

O_DIRECT is not a substitute for fsync(). It only guarantees that data gets to the storage device cache, which is not durable in most cases.

My understanding is that the storage device cache is opaque, that is, drives tend to lie, saying the write is done when it is in cache, and depend on having enough internal power capacity to flush on power loss.

Consumer devices sometimes lie (enterprise products less so), but there is a distinction between O_DIRECT and actual fsync at the protocol layer (e.g., in NVMe, fsync maps into a Flush command).

Re: Async I/O on Linux in databases

#83

Post by loeg »

cyanydeez wrote: post #48
loeg wrote: post #42

Earlier quoted context omitted.

No, we saw this scheme, it just doesn't work. Either of the async writes can fail after ack'ing the logical write to the client as successful (e.g., kernel crash or power failure) and then you have lost data.

You can always have data loss. The intent is that when the client is told the data is saved, it doesnt happen before the garuntee. I dont know if OP achieved this, but the client isnt told "we have your data" until both of the WALs are agreeing. If the system goes down those WALs are used to rebuild data in flight. The speed up allows for decoupling synchronous disk writes that are now parallel. You are not conceptua…

> I dont know if OP achieved this,

They did not.

> but the client isnt told "we have your data" until both of the WALs are agreeing.

Wrong. In the proposed scheme, the client writes are ack'd before the WAL writes are flushed. Their contents may or may not agree after subsequent power loss or kernel crash.

(It is generally considered unacceptable for network databases/filers to be lossier than the underlying media. Sometimes stronger guarantees are required/provided, but that is usually the minimum.)

Re: Async I/O on Linux in databases

#84

Post by loeg »

jasonthorsness wrote: post #50

Is the underlying NVME storage interface the kernel/drivers get to use cleaner/simpler than the Linux abstractions? Or does it get more complicated? Sometimes I wonder if certain high-performance applications would be better off running as special-purpose unikernels unburdened by interfaces designed for older generations of technology.

Also an option with io_uring: https://www.usenix.org/conference/fast24/presentation/joshi

(We use it at work it in a network object storage service in order to use the underlying NVMe T10-DIF[1], which isn't exposed nicely by conventional POSIX/Linux interfaces.)

Ultimately, having a full, ~normal Linux stack around makes system management / orchestration easier. And programs other than our specialized storage software can still access other partitions, etc.

[1]: https://en.wikipedia.org/wiki/Data_Integrity_Field

Re: Async I/O on Linux in databases

#85

Post by to11mtm »

ImPostingOnHN wrote: post #67
zozbot234 wrote: post #30

Earlier quoted context omitted.

So OP's real point is that fsync() sucks in the context of modern hardware where thousands of I/O reqs may be in flight at any given time. We need more fine-grained mechanisms to ensure that writes are committed to permanent storage, without introducing undue serialization.

Some applications, like Apache Kafka, don't immediately fsync every write. This lets the kernel batch writes and also linearize them, both adding speed. Until synced, the data exists only in the linux page cache. To deal with the risk of data loss, multiple such servers are used, with the hope that if one server dies before syncing, another server to which the data was replicated, performs an fsync without failure.

I feel like you can try to FAFO with that on a distributed log like Kafka (although also... eww, but also I wonder whether NATS does the same thing or not...)

I would think for something like a database, at most you'd want to have something like the io_uring_prep_fsync others mentioned with flags set to just not update the metadata.

To be clear, in my head I'm envisioning this case to be a WAL type scenario; in my head you can get away with just having a separate thread or threads pulling from WAL and writing to main DB files... but also I've never written a real database so maybe those thoughts are off base.

Re: Async I/O on Linux in databases

#86

Post by loeg »

jeffbee wrote: post #77

What's baffling to me about this post is that anyone would believe that io_uring was even capable of speeding up this workload by 10x. Unless your profile suggests that syscall entry is taking > 90% of your CPU time, that is impossible. The only thing io_uring can do for you is reduce your syscall count, so the upper bound of its utility is whatever you are currently spending on sysenter/exit.

You could also imagine it hiding write latency by allowing a very naive single-threaded application to do IOs concurrently, overlapped in time, instead of serialized. (But a threadpool would do much the same thing.)

Re: Async I/O on Linux in databases

#87

Post by gpderetta »

jtregunna wrote: post #61

Update: I updated the post based on the conversation below, I wholly missed an important callout about performance, and wasn't super clear that you do need to wait for the completion record to be written before responding to the client. That was implicitly mentioned by writing the completion record coming before responding, but I made it clearer to avoid confusion. Also the dual WAL approach is worse for latency, unl…

How can you know that the completion record is written to disk?

Re: Async I/O on Linux in databases

#88

Post by codys »

jtregunna wrote: post #61

Update: I updated the post based on the conversation below, I wholly missed an important callout about performance, and wasn't super clear that you do need to wait for the completion record to be written before responding to the client. That was implicitly mentioned by writing the completion record coming before responding, but I made it clearer to avoid confusion. Also the dual WAL approach is worse for latency, unl…

From the update added to the post:

> This is tracked through io_uring's completion queue - we only send a success response after receiving confirmation that the completion record has been persisted to stable storage.

Which completion queue event(s) are you examining here? I ask because the way this is worded makes it sound like you're waiting solely for the completion queue event for the _write_ to the "completion wal".

Doing that (waiting only on the "completion wal" write CQE)

1. doesn't ensure that the "intent wal" has been written (because it's a different io_uring and a different submission queue event used to do the "intent wal" write from the "completion wal" write), and

2. doesn't indicate the "intent wal" data or the "completion wal" data has made it to durable storage (one needs fsync for that, the completion queue events for writes don't make that promise. The CQE for an fsync opcode would indicate that data has made it to durable storage if the fsync has the right ordering wrt the writes and refers to the appropriate fd and data ranges. Alternatively, there are some flags that have the effect of implying an fsync following a write that could be used, but those aren't mentioned)

Re: Async I/O on Linux in databases

#89

Post by quietbritishjim »

ajross wrote: post #53
quietbritishjim wrote: post #29

The article claims that, when they switched to io_uring, > throughput increased by an order of magnitude almost immediately But right near the start is the real story: the sync version had > the classic fsync() call after every write to the log for durability They are not comparing performance of sync APIs vs io_uring. They're comparing using fsync vs not using fsync! They even go on to say that a problem with async…

> There's no getting around fsync if you want to be sure that your data is really on the storage medium. That's not correct; io_uring supports O_DIRECT write requests just fine. Obviously bypassing the cache isn't the same as just flushing it (which is what fsync does), so there are design impacts. But database engines are absolutely the target of io_uring's feature set and they're expected to be managing this comple…

Is that's true (notwithstanding objections from sibling comments) then that's just another spelling of fsync.

My point was really: you can't magically get the performance benefits of omitting fsync (or functional equivalent) while still getting the durability guarantees it gives.

Re: Async I/O on Linux in databases

#90

Post by gpderetta »

jeffbee wrote: post #77

What's baffling to me about this post is that anyone would believe that io_uring was even capable of speeding up this workload by 10x. Unless your profile suggests that syscall entry is taking > 90% of your CPU time, that is impossible. The only thing io_uring can do for you is reduce your syscall count, so the upper bound of its utility is whatever you are currently spending on sysenter/exit.

Io_uring could allow for better throughout by simply having multiple operations in flight and allow for better I/O scheduling.

But yes, this specific case seems to be a misunderstanding in what io_uring write completion means.

You would expect that they would have tested recovery by at least simulating system stops immediately after after Io completion notification.

Unless they are truly using asynchronous O_SYNC writes and are just bad at explaining it.

Post reply on HN