Live data from Hacker News

Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

pganalyze.com

121–130 of 159 posts

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#121
post #81
post #35

Is io_uring still plagued by security issues enabled by it's use? Or have those largely been fixed? My understanding was many Linux admins (or even distros by default?) were disabling io_uring.

Disabling io_uring because “guy on the internet said so” or “$faang_company says so” is beyond dumb. One should evaluate the risk according to their specific use case. It can be a good idea to disable it of you run untrusted workloads (eg: other people’s containers, sharing the same kernel) but if you have a kernel on a machine (virtual or real) dedicated to your own workload you can pretty much keep using io_uring.…

Most users don't know what io_uring is, nor would they have read any of thosee articles. They are not HN readers with low-level understanding of operating systems workingm and don't even remotely have the knowledge of how any of these technologies work, nor do they have the time to get it. With a very few exceptions, users have to rely entirely on software vendors and service providers to do this for them, and trust they know what they're doing.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#122
post #111

Postgres is such a cool project, I have so much respect for its maintainers and community! For me its the second most impactful OSS project in the business tech world behind Linux itself. A real public good to be cherished and praised.

I recently started a ML project using text data and the choice was between MySQL and Postgres. Having looked at the respective features and pros and cons, the choice was immediately obvious. Also, with pgvector and https://postgresml.com available, the choice for Postgres was even easier.

Postgres is Enough: https://gist.github.com/cpursley/c8fb81fe8a7e5df038158bdfe0f...

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#123
post #2

Is this new async. I/O feature for Linux only? I know Windows has IOCP and also now an IORing implementation of its own (Less familiar with macOS capabilities other than POSIX AIO). https://learn.microsoft.com/en-us/windows/win32/api/ioringap... Update: Most of the comments below seem to be missing the fact that Windows now also has an IORing implementation, as I mentioned above. Comparison article here: https://wind…

I am not a Windows guy but I (with help) managed to get IOCP working for this in a basic prototype. Will share publicly soon. I also sketched out an IoRing version (if you are interested in helping debug and flesh that out let me know!). Main learnings: the IOCP version can't do asynchronous flush! Which we want. The IoRing version can! But it can't do scatter/gather AKA vector I/O yet! Which is an essential feature…

Look into Registered I/O for sockets.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#124
post #23

Earlier quoted context omitted.

Yes, although Windows has had async I/O since Windows NT 3.1, their API is still not supported by Postgres.

FWIW, there are prototype patches for an IOCP based io_method. We just couldn't get them into an acceptable state for PG 18. I barely survived getting in what we did...

I didn't mean that as a criticism on Postgres, certainly not on Postgres developers. I really look forward to evaluating this change. I'm just reminiscing a bit about admiring Windows NT ... and being incredibly disappointed by the market.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#125
post #111

Postgres is such a cool project, I have so much respect for its maintainers and community! For me its the second most impactful OSS project in the business tech world behind Linux itself. A real public good to be cherished and praised.

I recently started a ML project using text data and the choice was between MySQL and Postgres. Having looked at the respective features and pros and cons, the choice was immediately obvious. Also, with pgvector and https://postgresml.com available, the choice for Postgres was even easier.

Why not MongoDB?

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#126

Does anyone know when the update allowing more concurrent connections is dropping, so we can stop using pgbouncer?

I recently read a great article exploring what would change if they were to switch from processes to threads for each connection. Running a connection pooler didn't seem so bad to me after reading it. https://medium.com/@tusharmalhotra_81114/why-postgresql-choo...

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#127

Earlier quoted context omitted.

I recently started a ML project using text data and the choice was between MySQL and Postgres. Having looked at the respective features and pros and cons, the choice was immediately obvious. Also, with pgvector and https://postgresml.com available, the choice for Postgres was even easier.

Why not MongoDB?

I guess they didn’t need anything web scale.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#128

Earlier quoted context omitted.

I recently started a ML project using text data and the choice was between MySQL and Postgres. Having looked at the respective features and pros and cons, the choice was immediately obvious. Also, with pgvector and https://postgresml.com available, the choice for Postgres was even easier.

Why not MongoDB?

The question should be the other way around: why mongodb? It’s not ACID compliant so has major down sides…

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#130
post #111

Postgres is such a cool project, I have so much respect for its maintainers and community! For me its the second most impactful OSS project in the business tech world behind Linux itself. A real public good to be cherished and praised.

You can get no better recommendation, that a phrase I once heard at a presentation by Richard Hipp primary author of SQLite. As he was explaining how he analyzed new features and design decisions he says some of his guidance is: "What would Postgres do?"
Post reply on HN