I've watched the Tigerbeatle talk (youtube link in the article). This is very interesting even for those not in the space.
Async I/O on Linux in databases
41–50 of 100 posts
Re: Async I/O on Linux in databases
#42I don't get this scheme at all. The protocol violates durability, because once the client receives success from server, it should be durable. However, completion record is async, it is possible that it never completes and server crashes. During recovery, since the server applies only the operations which have both records, you will not recover a record which was successful to the client.
I think you missed the part in the middle: ----------------- So the protocol ends up becoming: Write intent record (async) Perform operation in memory Write completion record (async) Return success to client ----------------- In other words, the client only knows its a success when both wal files have been written. The goal is not to provide faster responses to the client, on the first intent record, but to ensure th…
No, we saw this scheme, it just doesn't work. Either of the async writes can fail after ack'ing the logical write to the client as successful (e.g., kernel crash or power failure) and then you have lost data.
Re: Async I/O on Linux in databases
#43The article claims that, when they switched to io_uring, > throughput increased by an order of magnitude almost immediately But right near the start is the real story: the sync version had > the classic fsync() call after every write to the log for durability They are not comparing performance of sync APIs vs io_uring. They're comparing using fsync vs not using fsync! They even go on to say that a problem with async…
Suggest watching the Tigerbeatle video link in the article. There they discuss bitrot, "fsync gate", how Postgres used fsync wrong for 30 years, etc. It is very interesting even as pure entertainment.
Re: Async I/O on Linux in databases
#44Earlier quoted context omitted.
So OP's real point is that fsync() sucks in the context of modern hardware where thousands of I/O reqs may be in flight at any given time. We need more fine-grained mechanisms to ensure that writes are committed to permanent storage, without introducing undue serialization.
Well, there already is slightly more fine gained control: in the sync version, you can perhaps call sync write() a few times before calling fsync() once i.e. basically batch up a few writes. That does have the disadvantage that you can't easily queue new writes while waiting for the previous ones. Perhaps you could use calls to write() in another thread while the first one is waiting for fsync() for the previous batc…
Postgres claims to have some kind of commit batching, but I couldn't figure out how to turn it on.
I wanted to scrub a table by processing each row, but without holding locks, so I wanted to commit every few hundred rows, but with only ACI and not D, since I could just run the process again. I don't think Postgres supports this feature. It also seemed to be calling fsync much more than once per transaction.
Re: Async I/O on Linux in databases
#45I feel like writing asynchronously to a WAL defeats its purpose.
Re: Async I/O on Linux in databases
#46Slightly off topic but anyone knows when/if Google is going to enable io_uring for Android?
Hopefully never. It almost seems to have been purpose-built for local privilege escalation exploits.
Re: Async I/O on Linux in databases
#47The article claims that, when they switched to io_uring, > throughput increased by an order of magnitude almost immediately But right near the start is the real story: the sync version had > the classic fsync() call after every write to the log for durability They are not comparing performance of sync APIs vs io_uring. They're comparing using fsync vs not using fsync! They even go on to say that a problem with async…
So OP's real point is that fsync() sucks in the context of modern hardware where thousands of I/O reqs may be in flight at any given time. We need more fine-grained mechanisms to ensure that writes are committed to permanent storage, without introducing undue serialization.
The Linux RWF_DSYNC flag sets the Full Unit Access (FUA) bit in write requests. This can be used instead of fdatasync(2) in some cases. It only syncs a specific write request instead of the entire disk write cache.
Re: Async I/O on Linux in databases
#48Earlier quoted context omitted.
I think you missed the part in the middle: ----------------- So the protocol ends up becoming: Write intent record (async) Perform operation in memory Write completion record (async) Return success to client ----------------- In other words, the client only knows its a success when both wal files have been written. The goal is not to provide faster responses to the client, on the first intent record, but to ensure th…
No, we saw this scheme, it just doesn't work. Either of the async writes can fail after ack'ing the logical write to the client as successful (e.g., kernel crash or power failure) and then you have lost data.
You can always have data loss. The intent is that when the client is told the data is saved, it doesnt happen before the garuntee.
I dont know if OP achieved this, but the client isnt told "we have your data" until both of the WALs are agreeing. If the system goes down those WALs are used to rebuild data in flight.
The speed up allows for decoupling synchronous disk writes that are now parallel.
You are not conceptualizing what data loss means in the ACID contract between DB and Client.
But you
Re: Async I/O on Linux in databases
#49Earlier quoted context omitted.
Well, there already is slightly more fine gained control: in the sync version, you can perhaps call sync write() a few times before calling fsync() once i.e. basically batch up a few writes. That does have the disadvantage that you can't easily queue new writes while waiting for the previous ones. Perhaps you could use calls to write() in another thread while the first one is waiting for fsync() for the previous batc…
Postgres claims to have some kind of commit batching, but I couldn't figure out how to turn it on. I wanted to scrub a table by processing each row, but without holding locks, so I wanted to commit every few hundred rows, but with only ACI and not D, since I could just run the process again. I don't think Postgres supports this feature. It also seemed to be calling fsync much more than once per transaction.
Maybe I don’t understand what you’re trying to do, but you can directly control how frequently commits occur.
BEGIN
INSERT … —- batch of N size
COMMIT AND CHAIN
INSERT …Re: Async I/O on Linux in databases
#50Is the underlying NVME storage interface the kernel/drivers get to use cleaner/simpler than the Linux abstractions? Or does it get more complicated? Sometimes I wonder if certain high-performance applications would be better off running as special-purpose unikernels unburdened by interfaces designed for older generations of technology.