Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

pganalyze.com

Post reply on HN
Print view View original

81–90 of 159 posts

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#81

Post by znpy »

p_ing wrote: post #35

Is io_uring still plagued by security issues enabled by it's use? Or have those largely been fixed? My understanding was many Linux admins (or even distros by default?) were disabling io_uring.

Disabling io_uring because “guy on the internet said so” or “$faang_company says so” is beyond dumb.

One should evaluate the risk according to their specific use case.

It can be a good idea to disable it of you run untrusted workloads (eg: other people’s containers, sharing the same kernel) but if you have a kernel on a machine (virtual or real) dedicated to your own workload you can pretty much keep using io_uring. There are other technologies to enforce security (eg: selinux emand similar).

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#82

Post by password4321 »

greenavocado wrote: post #48
p_ing wrote: post #33

Earlier quoted context omitted.

Yeah, surprise Linux had to play catch up to a Windows 1994 release! Same with the scheduler, I'd argue Windows does OOM better than Linux today... Windows even had the concept of io_uring before, but network only with Registered I/O back in the Windows 8 (8.1?) days. Linux still lacks the "all I/O is async" NT has. The underlying kernel and executive of Windows aren't primitive pieces of trash. They're quite advance…

How difficult would it be to completely tear out the Windows desktop experience and just use the system and display drivers without the rest? Has anybody attempted such a feat?

Free-as-in-beer Hyper-V Server 2019 is in extended support (only security updates) until 2029.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#83

Post by wtallis »

znpy wrote: post #81
p_ing wrote: post #35

Is io_uring still plagued by security issues enabled by it's use? Or have those largely been fixed? My understanding was many Linux admins (or even distros by default?) were disabling io_uring.

Disabling io_uring because “guy on the internet said so” or “$faang_company says so” is beyond dumb. One should evaluate the risk according to their specific use case. It can be a good idea to disable it of you run untrusted workloads (eg: other people’s containers, sharing the same kernel) but if you have a kernel on a machine (virtual or real) dedicated to your own workload you can pretty much keep using io_uring.…

I think at this point, those "other technologies to enforce security" are the main area of concern for io_uring users: if those other security layers don't know about io_uring they won't apply any restrictions to it.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#84

Post by 9dev »

niux wrote: post #16
Tostino wrote: post #14

Earlier quoted context omitted.

I would absolutely use another backup utility (additionally if you want) if I were you (barman, pgbackrest, etc). You are just wrapping pgdump, which is not a full featured backup solution. Great for a snapshot... Use some of the existing tools and you get point-in-time recovery, easy restores to hot standbys for replication, a good failover story, backup rotations, etc.

The reason I wrote my own tool is because I couldn't find anything for Pg17 at the time and pgbackrest seemed overkill for my needs. Also, the CLI handles backup rotations as well. Barman looks interesting though, I'll definitely have a look, thanks!

pgbackrest only looks scary because it’s so flexible, but the defaults work great in almost all cases. The most complex thing you’ll need to do is creating a storage bucket to write to, and configure the appropriate storage provider in pgbackrest's config file.

When it’s set up properly, it’s solid as rock. I’d really recommend you check it out again; it likely solves everything you did more elegantly, and also covers a ton of things you didn’t think of. Been there, done that :)

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#85

Post by 9dev »

codegeek wrote: post #29
martinald wrote: post #13

I sort of had to chuckle at the 20k IOPS AWS instance, given even a consumer $100-200 NVMe gives ~1million+ IOPS these days. I suspect now we have PCIe 5.0 NVMes this will go up to I always do wonder how much "arbitrary" cloud limits on things like this cause so many issues. I'm sure that async IO is very helpful anyway, but I bet on a 1million IOPS NVMe it is nowhere near as important. We're effectively optimising c…

You probably already know this but I will say it anyway. These cloud services like AWS are not succeeding in enterprise because they have outdated hardware. They succeed because in enterprise, CIOs and CTOs want something that is known, has a brand and everyone else uses it. It's like the old adage of "No one got fired for using IBM". Now it is "No one gets fired for hosting with AWS no matter how ridiculous the cost…

> No one gets fired for hosting with AWS no matter how ridiculous the cost and corresponding feature is

Actually, AWS is so expensive, hosting everything we ran on Hetzner there would have simply depleted our funding, and the company would not exist anymore.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#89

Post by lousken »

Tostino wrote: post #14
niux wrote: post #3

I recently deployed Postgres on a dedicated Hetzner EX-44 server (20 cores, 64GB RAM, 2x 512GB NVMe SSDs in RAID 1) for €39/month. The price-to-performance ratio is exceptional, providing enterprise-level capacity at a fraction of typical cloud costs. For security, I implemented TailScale which adds only ~5ms of latency while completely eliminating public network exposure - a worthwhile tradeoff for the significant s…

I would absolutely use another backup utility (additionally if you want) if I were you (barman, pgbackrest, etc). You are just wrapping pgdump, which is not a full featured backup solution. Great for a snapshot... Use some of the existing tools and you get point-in-time recovery, easy restores to hot standbys for replication, a good failover story, backup rotations, etc.

do you still need it? haven't managed pg for a while, but shouldn't pg17 have some solution for backups?

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#90

Post by adgjlsfhk1 »

brulard wrote: post #25
binary132 wrote: post #15

Earlier quoted context omitted.

Meanwhile people are running things on raspberry pi home clusters thinking they’re winning

Maybe they are. With NVMe hat, you get decent IO performance

It's still moderately bad. Raspberry pi is limited to 2 gen3 pcie lanes which is ~4-8x slower than the drive (and you will likely be further limited by cpu speed)
Post reply on HN