Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async I/O on Linux in databases

blog.canoozie.net

Post reply on HN
Print view View original

21–30 of 100 posts

Re: Async I/O on Linux in databases

#21

Post by LAC-Tech »

Great article, but I have a question:

The problem with naive async I/O in a database context at least, is that you lose the durability guarantee that makes databases useful. When a client receives a success response, their expectation is the data will survive a system crash. But with async I/O, by the time you send that response, the data might still be sitting in kernel buffers, not yet written to stable storage.

Shouldn't you just tie the successful response to a successful fsync?

Async or sync, I'm not sure what's different here.

Re: Async I/O on Linux in databases

#22

Post by leentee »

First, I think the article provides false claim, the solution doesn't guarantee durability. Second, I believe good synchronous code is better than bad asynchronous code, and it's way easier to write good synchronous code than asynchronous code, especially with io_uring. Modern NVMe are fast, even with synchronous IO, enough for most applications. Before thinking about asynchronous, make sure your application use synchronous IO well.

Re: Async I/O on Linux in databases

#23

Post by toolslive »

avinassh wrote: post #19
toolslive wrote: post #15

Earlier quoted context omitted.

it's possible to unify the WAL and the tree. There are some append only B-tree implementations. https://github.com/Incubaid/baardskeerder fe.

There are also CoW B Trees not entirely similar, but kinda same.

there's a whole class of persistent persistent (the repetition is intentional here) data structures. Some of them even combine performance with elegance.

Re: Async I/O on Linux in databases

#24

Post by zozbot234 »

jmpman wrote: post #6

“Write intent record (async) Perform operation in memory Write completion record (async) Return success to client During recovery, I only apply operations that have both intent and completion records. This ensures consistency while allowing much higher throughput. “ Does this mean that a client could receive a success for a request, which if the system crashed immediately afterwards, when replayed, wouldn’t necessari…

> Does this mean that a client could receive a success for a request, which if the system crashed immediately afterwards, when replayed, wouldn’t necessarily have that request recorded?

Yup. OP says "the intent record could just be sitting in a kernel buffer", but then the exact same issue applies to the completion record. So confirmation to the client cannot be issued until the completion record has been written to durable storage. Not really seeing the point of this blogpost.

Re: Async I/O on Linux in databases

#25

Post by jorangreef »

To be clear, this is different to what we do (and why we do it) in TigerBeetle.

For example, we never externalize commits without full fsync, to preserve durability [0].

Further, the motivation for why TigerBeetle has both a prepare WAL plus a header WAL is different, not performance (we get performance elsewhere, through batching) but correctness, cf. “Protocol-Aware Recovery for Consensus-Based Storage” [1].

Finally, TigerBeetle's recovery is more intricate, we do all this to survive TigerBeetle's storage fault model. You can read the actual code here [2] and Kyle Kingsbury's Jepsen report on TigerBeetle also provides an excellent overview [3].

[0] https://www.youtube.com/watch?v=tRgvaqpQPwE

[1] https://www.usenix.org/system/files/conference/fast18/fast18...

[2] https://github.com/tigerbeetle/tigerbeetle/blob/main/src/vsr...

[3] https://jepsen.io/analyses/tigerbeetle-0.16.11.pdf

Re: Async I/O on Linux in databases

#27

Post by cbzbc »

tlb wrote: post #3

The recovery process is to "only apply operations that have both intent and completion records." But then I don't see the point of logging the intent record separately. If no completion is logged, the intent is ignored. So you could log the two together. Presumably the intent record is large (containing the key-value data) while the completion record is tiny (containing just the index of the intent record). Is the po…

Presumably the intent record is large (containing the key-value data) while the completion record is tiny

I don't think this is necessarily the case, because the operations may have completed in a different order to how they are recorded in the intent log.

Re: Async I/O on Linux in databases

#28

Post by benjiro »

avinassh wrote: post #18

I don't get this scheme at all. The protocol violates durability, because once the client receives success from server, it should be durable. However, completion record is async, it is possible that it never completes and server crashes. During recovery, since the server applies only the operations which have both records, you will not recover a record which was successful to the client.

I think you missed the part in the middle:

-----------------

So the protocol ends up becoming:

Write intent record (async) Perform operation in memory Write completion record (async) Return success to client

-----------------

In other words, the client only knows its a success when both wal files have been written.

The goal is not to provide faster responses to the client, on the first intent record, but to ensure that the system is not stuck with I/O Waiting on fsync requests.

When you write a ton of data to database, you often see that its not the core writes but the I/O > fsync that eat a ton of your resources. Cutting back on that mess, results that you can push more performance out of a write heavy server.

Re: Async I/O on Linux in databases

#29

Post by quietbritishjim »

The article claims that, when they switched to io_uring,

> throughput increased by an order of magnitude almost immediately

But right near the start is the real story: the sync version had

> the classic fsync() call after every write to the log for durability

They are not comparing performance of sync APIs vs io_uring. They're comparing using fsync vs not using fsync! They even go on to say that a problem with async API is that

> you lose the durability guarantee that makes databases useful. ... the data might still be sitting in kernel buffers, not yet written to stable storage.

No! That's because you stopped using fsync. It's nothing to do with your code being async.

If you just removed the fsync from the sync code you'd quite possibly get a speedup of an order of magnitude too. Or if you put the fsync back in the async version (I don't know io_uring well enough to understand that but it appears to be possible with "io_uring_prep_fsync") then that would surely slide back. Would the io_uring version still be faster either way? Quite possibly, but because they made an apples-to-oranges comparison, we can't know from this article.

(As other commenters have pointed out, their two-phase commit strategy also fails to provide any guarantee. There's no getting around fsync if you want to be sure that your data is really on the storage medium.)

Re: Async I/O on Linux in databases

#30

Post by zozbot234 »

quietbritishjim wrote: post #29

The article claims that, when they switched to io_uring, > throughput increased by an order of magnitude almost immediately But right near the start is the real story: the sync version had > the classic fsync() call after every write to the log for durability They are not comparing performance of sync APIs vs io_uring. They're comparing using fsync vs not using fsync! They even go on to say that a problem with async…

So OP's real point is that fsync() sucks in the context of modern hardware where thousands of I/O reqs may be in flight at any given time. We need more fine-grained mechanisms to ensure that writes are committed to permanent storage, without introducing undue serialization.
Post reply on HN