Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

pganalyze.com

Post reply on HN
Print view View original

21–30 of 159 posts

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#21

Post by __s »

martinald wrote: post #13

I sort of had to chuckle at the 20k IOPS AWS instance, given even a consumer $100-200 NVMe gives ~1million+ IOPS these days. I suspect now we have PCIe 5.0 NVMes this will go up to I always do wonder how much "arbitrary" cloud limits on things like this cause so many issues. I'm sure that async IO is very helpful anyway, but I bet on a 1million IOPS NVMe it is nowhere near as important. We're effectively optimising c…

Even worse on Azure where we had to ask customers to scale up vcpu to increase iops

https://azure.microsoft.com/en-us/pricing/details/managed-di...

Increasing vcpu also opened up more disk slots to try improve situation with disk striping

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#22

Post by merb »

anonymars wrote: post #20
martinald wrote: post #13

I sort of had to chuckle at the 20k IOPS AWS instance, given even a consumer $100-200 NVMe gives ~1million+ IOPS these days. I suspect now we have PCIe 5.0 NVMes this will go up to I always do wonder how much "arbitrary" cloud limits on things like this cause so many issues. I'm sure that async IO is very helpful anyway, but I bet on a 1million IOPS NVMe it is nowhere near as important. We're effectively optimising c…

> given even a consumer $100-200 NVMe gives ~1million+ IOPS these days In the face of sustained writes? For how long?

sustained reads would not even give 1 mio iops in that case. Maybe wen you only read the same file that fits into the nvme cache. Which probably never happens in a production database..

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#23

Post by spwa4 »

nu11ptr wrote: post #2

Is this new async. I/O feature for Linux only? I know Windows has IOCP and also now an IORing implementation of its own (Less familiar with macOS capabilities other than POSIX AIO). https://learn.microsoft.com/en-us/windows/win32/api/ioringap... Update: Most of the comments below seem to be missing the fact that Windows now also has an IORing implementation, as I mentioned above. Comparison article here: https://wind…

Yes, although Windows has had async I/O since Windows NT 3.1, their API is still not supported by Postgres.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#25

Post by brulard »

binary132 wrote: post #15
martinald wrote: post #13

I sort of had to chuckle at the 20k IOPS AWS instance, given even a consumer $100-200 NVMe gives ~1million+ IOPS these days. I suspect now we have PCIe 5.0 NVMes this will go up to I always do wonder how much "arbitrary" cloud limits on things like this cause so many issues. I'm sure that async IO is very helpful anyway, but I bet on a 1million IOPS NVMe it is nowhere near as important. We're effectively optimising c…

Meanwhile people are running things on raspberry pi home clusters thinking they’re winning

Maybe they are. With NVMe hat, you get decent IO performance

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#26

Post by maherbeg »

martinald wrote: post #13

I sort of had to chuckle at the 20k IOPS AWS instance, given even a consumer $100-200 NVMe gives ~1million+ IOPS these days. I suspect now we have PCIe 5.0 NVMes this will go up to I always do wonder how much "arbitrary" cloud limits on things like this cause so many issues. I'm sure that async IO is very helpful anyway, but I bet on a 1million IOPS NVMe it is nowhere near as important. We're effectively optimising c…

instance store on aws can give up to 3.3mil iops https://aws.amazon.com/blogs/aws/now-available-i3-instances-... - the main problem is just using networked storage.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#27

Post by perching_aix »

maherbeg wrote: post #26
martinald wrote: post #13

I sort of had to chuckle at the 20k IOPS AWS instance, given even a consumer $100-200 NVMe gives ~1million+ IOPS these days. I suspect now we have PCIe 5.0 NVMes this will go up to I always do wonder how much "arbitrary" cloud limits on things like this cause so many issues. I'm sure that async IO is very helpful anyway, but I bet on a 1million IOPS NVMe it is nowhere near as important. We're effectively optimising c…

instance store on aws can give up to 3.3mil iops https://aws.amazon.com/blogs/aws/now-available-i3-instances-... - the main problem is just using networked storage.

Instance store is also immediately wiped when the instance is halted / restarted, which can theoretically happen at any time, for example by a mystery instance failure, or a patching tool that's helpfully restarting your boxes during offhours.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#28

Post by arp242 »

Tostino wrote: post #14
niux wrote: post #3

I recently deployed Postgres on a dedicated Hetzner EX-44 server (20 cores, 64GB RAM, 2x 512GB NVMe SSDs in RAID 1) for €39/month. The price-to-performance ratio is exceptional, providing enterprise-level capacity at a fraction of typical cloud costs. For security, I implemented TailScale which adds only ~5ms of latency while completely eliminating public network exposure - a worthwhile tradeoff for the significant s…

I would absolutely use another backup utility (additionally if you want) if I were you (barman, pgbackrest, etc). You are just wrapping pgdump, which is not a full featured backup solution. Great for a snapshot... Use some of the existing tools and you get point-in-time recovery, easy restores to hot standbys for replication, a good failover story, backup rotations, etc.

Snapshots are backups. Not sufficient backups for some cases, but perfectly fine for others.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#29

Post by codegeek »

martinald wrote: post #13

I sort of had to chuckle at the 20k IOPS AWS instance, given even a consumer $100-200 NVMe gives ~1million+ IOPS these days. I suspect now we have PCIe 5.0 NVMes this will go up to I always do wonder how much "arbitrary" cloud limits on things like this cause so many issues. I'm sure that async IO is very helpful anyway, but I bet on a 1million IOPS NVMe it is nowhere near as important. We're effectively optimising c…

You probably already know this but I will say it anyway. These cloud services like AWS are not succeeding in enterprise because they have outdated hardware. They succeed because in enterprise, CIOs and CTOs want something that is known, has a brand and everyone else uses it. It's like the old adage of "No one got fired for using IBM". Now it is "No one gets fired for hosting with AWS no matter how ridiculous the cost and corresponding feature is".

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#30

Post by ARandomerDude »

seunosewa wrote: post #18
skeptrune wrote: post #11

How close is this to the way MySQL does it with InnoDB? It appears to be about the same.

Yep. It's a low hanging fruit they should've picked years ago. They will eventually figure out using b-trees for tables too.

Can you elaborate on this B-tree part of your comment? I know B-tree is the default index type in pg, but it sounds like there’s more to the story that I’m not familiar with.
Post reply on HN