Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async I/O on Linux in databases

blog.canoozie.net

Post reply on HN
Print view View original

71–80 of 100 posts

Re: Async I/O on Linux in databases

#71

Post by jandrewrogers »

ajross wrote: post #53
quietbritishjim wrote: post #29

The article claims that, when they switched to io_uring, > throughput increased by an order of magnitude almost immediately But right near the start is the real story: the sync version had > the classic fsync() call after every write to the log for durability They are not comparing performance of sync APIs vs io_uring. They're comparing using fsync vs not using fsync! They even go on to say that a problem with async…

> There's no getting around fsync if you want to be sure that your data is really on the storage medium. That's not correct; io_uring supports O_DIRECT write requests just fine. Obviously bypassing the cache isn't the same as just flushing it (which is what fsync does), so there are design impacts. But database engines are absolutely the target of io_uring's feature set and they're expected to be managing this comple…

O_DIRECT is not a substitute for fsync(). It only guarantees that data gets to the storage device cache, which is not durable in most cases.

Re: Async I/O on Linux in databases

#72

Post by sethev »

There's some faulty reasoning in this post. Without the code, it's hard to pin down exactly where things went wrong.

These are the steps described in the post:

   1. Write intent record (async)
   2. Perform operation in memory
   3. Write completion record (async)
   4. Wait for the completion record to be written to the WAL
   5. Return success to client
If 4 is done correctly then 3 is not needed - it can just wait for the intent to be durable before replying to the client. Perhaps there's a small benefit to speculatively executing the operation before the WAL is committed - but I'm skeptical and my guess is that 4 is not being done correctly. The author added an update to the article:

> This is tracked through io_uring's completion queue - we only send a success response after receiving confirmation that the completion record has been persisted to stable storage

This makes it sound like he's submitting write operations for the completion record and then misinterpreting the completion queue for those writes as "the record is now in durable storage".

Re: Async I/O on Linux in databases

#73

Post by morningsam »

immibis wrote: post #44
quietbritishjim wrote: post #36

Earlier quoted context omitted.

Well, there already is slightly more fine gained control: in the sync version, you can perhaps call sync write() a few times before calling fsync() once i.e. basically batch up a few writes. That does have the disadvantage that you can't easily queue new writes while waiting for the previous ones. Perhaps you could use calls to write() in another thread while the first one is waiting for fsync() for the previous batc…

Postgres claims to have some kind of commit batching, but I couldn't figure out how to turn it on. I wanted to scrub a table by processing each row, but without holding locks, so I wanted to commit every few hundred rows, but with only ACI and not D, since I could just run the process again. I don't think Postgres supports this feature. It also seemed to be calling fsync much more than once per transaction.

Looking through the options listed under "Non-Durable Settings", [1] I guess synchronous_commit = off fits the bill?

[1]: https://www.postgresql.org/docs/current/non-durability.html

Re: Async I/O on Linux in databases

#74

Post by gpderetta »

benjiro wrote: post #38
zozbot234 wrote: post #37

Earlier quoted context omitted.

> It has literally the same durability as a fsync write I don't think we can ensure this without knowing what fsync() maps to in the NVMe standard, and somehow replicating that. Just reading back is not enough, e.g. the hardware might be reading from a volatile cache that will be lost in a crash.

Unless your running cheap consumer NVME drives, that is not a issue on Enterprise SSD/NVMEs as they have their own capacitors to ensure data is always written. On cheaper NVME drives, your point is valid. But we also need to add, how much at risk are you. What is the chance of a system doing funky issues, that you just happened to send X amount of confirm requests to clients, with data that never got written. For spe…

Forget about consumer FD, unless you are explicitly doing O_DIRECT, why would you expect that a notification that your IO has completed would mean that it has reached the disk at all? The data might still be just in the kernel page buffer and not gotten close to the disk at all.

You mention you need to wait for the compilation record to be written. But how do you do that without fsync or O_DIRECT? A notification that the write is completed is not that.

Edit: maybe you are using RWF_SYNC in your write call. That could work.

Re: Async I/O on Linux in databases

#75

Post by cryptonector »

codys wrote: post #66
quietbritishjim wrote: post #29

The article claims that, when they switched to io_uring, > throughput increased by an order of magnitude almost immediately But right near the start is the real story: the sync version had > the classic fsync() call after every write to the log for durability They are not comparing performance of sync APIs vs io_uring. They're comparing using fsync vs not using fsync! They even go on to say that a problem with async…

> > you lose the durability guarantee that makes databases useful. ... the data might still be sitting in kernel buffers, not yet written to stable storage. > No! That's because you stopped using fsync. It's nothing to do with your code being async. From that section, it sounds like OP was tossing data into the io_uring submition queue and calling it "done" at that point (ie: not waiting for the io_uring completion q…

Correct, TFA needs to wait for the completion of _all_ writes to the WAL, which is what `fsync()` was doing. Waiting only for the completion of the "completion record" does not ensure that the "intent record" made it to the WAL. In the event of a power failure it is entirely possible that the intent record did not make it but the completion record did, and then on recovery you'll have to panic.

Re: Async I/O on Linux in databases

#76

Post by somat »

jandrewrogers wrote: post #71
ajross wrote: post #53

Earlier quoted context omitted.

> There's no getting around fsync if you want to be sure that your data is really on the storage medium. That's not correct; io_uring supports O_DIRECT write requests just fine. Obviously bypassing the cache isn't the same as just flushing it (which is what fsync does), so there are design impacts. But database engines are absolutely the target of io_uring's feature set and they're expected to be managing this comple…

O_DIRECT is not a substitute for fsync(). It only guarantees that data gets to the storage device cache, which is not durable in most cases.

My understanding is that the storage device cache is opaque, that is, drives tend to lie, saying the write is done when it is in cache, and depend on having enough internal power capacity to flush on power loss.

Re: Async I/O on Linux in databases

#77

Post by jeffbee »

What's baffling to me about this post is that anyone would believe that io_uring was even capable of speeding up this workload by 10x. Unless your profile suggests that syscall entry is taking > 90% of your CPU time, that is impossible. The only thing io_uring can do for you is reduce your syscall count, so the upper bound of its utility is whatever you are currently spending on sysenter/exit.

Re: Async I/O on Linux in databases

#78

Post by mhuffman »

jorangreef wrote: post #60
osigurdson wrote: post #43

Earlier quoted context omitted.

Suggest watching the Tigerbeatle video link in the article. There they discuss bitrot, "fsync gate", how Postgres used fsync wrong for 30 years, etc. It is very interesting even as pure entertainment.

Thanks! Great to hear you enjoyed our talk. Most of it is simply putting the spotlight on UW-Madison’s work on storage faults. Just to emphasize again that this blog post here is really quite different, since it does not fsync and breaks durability. Not what we do in TigerBeetle or would recommend or encourage. See also: https://news.ycombinator.com/item?id=44624065

Hi! I don't have a need for your products directly, but I was very intrigued when I saw TB's demo and talk on ThePrimeagen YT channel. I have be developing software for a looooong time and it was a breath of fresh air in a sea of startups to see a company champion optimization, speed, and security without going too deep in the weeds and slowing development. These days, that typically comes more as an afterthought or as a response to an incident. Or not at all. I would recommend any developer with an open mind to read this short document[0]. I have been integrating it into my own company's development practices with good results.

[0]https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TI...

Re: Async I/O on Linux in databases

#79

Post by codys »

cryptonector wrote: post #75
codys wrote: post #66

Earlier quoted context omitted.

> > you lose the durability guarantee that makes databases useful. ... the data might still be sitting in kernel buffers, not yet written to stable storage. > No! That's because you stopped using fsync. It's nothing to do with your code being async. From that section, it sounds like OP was tossing data into the io_uring submition queue and calling it "done" at that point (ie: not waiting for the io_uring completion q…

Correct, TFA needs to wait for the completion of _all_ writes to the WAL, which is what `fsync()` was doing. Waiting only for the completion of the "completion record" does not ensure that the "intent record" made it to the WAL. In the event of a power failure it is entirely possible that the intent record did not make it but the completion record did, and then on recovery you'll have to panic.

Yes, but I suspect there might be some confusion by the author and others between "io_uring completion of a write" (ie: io_uring sends its completion queue event that corresponds to a previous submission queue event) and "fsync completion" (as you've put as "completion of all writes", though note that fsync the api is fd scoped and the io_uring operation for fsync has file range support).

The CQEs on a write indicate something different compared to the CQE of an fsync operation on that same range.

Re: Async I/O on Linux in databases

#80

Post by jorangreef »

mhuffman wrote: post #78
jorangreef wrote: post #60

Earlier quoted context omitted.

Thanks! Great to hear you enjoyed our talk. Most of it is simply putting the spotlight on UW-Madison’s work on storage faults. Just to emphasize again that this blog post here is really quite different, since it does not fsync and breaks durability. Not what we do in TigerBeetle or would recommend or encourage. See also: https://news.ycombinator.com/item?id=44624065

Hi! I don't have a need for your products directly, but I was very intrigued when I saw TB's demo and talk on ThePrimeagen YT channel. I have be developing software for a looooong time and it was a breath of fresh air in a sea of startups to see a company champion optimization, speed, and security without going too deep in the weeds and slowing development. These days, that typically comes more as an afterthought or…

Appreciate your taking the time to write these kind words. Great to hear that TigerStyle has been making an impact on your company’s developer practices!
Post reply on HN