Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

351–360 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#351

Post by anarazel »

scottlamb wrote: post #350
anarazel wrote: post #345

Earlier quoted context omitted.

> That said, I think there have been efforts to use io_uring on Linux. I'm not sure how that would work with the process per connection model. Haven't been following it... There's some minor details that are easier with threads in that context, but on the whole it doesn't make much of a difference.

I don't understand how it works with thread per connection either. io_uring is designed for systems that have a thread and ring per core, for you to give it a bunch of IO to do at once (batches and chains), and your threads to do other work in the meantime. The syscall cost is amortized or even (through IORING_SETUP_SQPOLL) eliminated. If your code is instead designed to be synchronous and thus can only do one IO at…

> io_uring is designed for systems that have a thread and ring per core

That's not needed to benefit from io_uring

> for you to give it a bunch of IO to do at once (batches and chains), and your threads to do other work in the meantime.

You can see substantial gains even if you just submit multiple IOs at once, and then block waiting for any of them to complete. The cost of blocking on IO is amortized to some degree over multiple IOs. Of course it's even better to not block at all...

> If your code is instead designed to be synchronous and thus can only do one IO at a time and needs a syscall to block on it, I don't think there's much if any benefit in using io_uring.

We/I have done the work to issue multiple IOs at a time as part of the patchset introducing AIO support (with among others, an io_uring backend). There's definitely more to do, particularly around index scans, but ...

Re: PostgreSQL reconsiders its process-based model

#352

Post by ComputerGuru »

imtringued wrote: post #189
ComputerGuru wrote: post #132

Earlier quoted context omitted.

Because global variables can be confined to individual cpp files, exclusively visible in that compilation unit. It makes them far easier to reason with than hoisting them to the "global and globally visible" option if you just use a gargantuan struct. Which is why a more invasive refactor might be required.

Just use thread local variables. I abuse them for ridiculous things.

Yeah, I was really into that before there was even a cross-compiler/cross-platform syntax for declaring TLS values in C++ but have since “upgraded” to avoiding TLS altogether where possible. The quality of the implementations vary greatly from compiler and platform to compiler and platform, you run into weird issues with thread_at exit if they’re not primitive types, they run afoul of any fibers/coroutines/etc that have since become extremely prevalent, and a few other things.

Re: PostgreSQL reconsiders its process-based model

#353

Post by jupp0r »

megous wrote: post #320
jupp0r wrote: post #291

Earlier quoted context omitted.

The problems it causes for Postgres are outlined in the article on LWN.

> Globals work well enough when each server process has its own set... PostgreSQL uses a process model. So the article just states that globals work fine for PostgreSQL > Knizhnik has already done a threads port of PostgreSQL. The global-variable problem, he said, was not that difficult. I see no big problem based on information from person who did some porting already.

Knizhnik made these variables thread local, which is fine if you have a fixed association of threads to data. This looses some flexibility if your runtime needs to incorporate multiple sessions on one thread (for example to hide IO latency) in the future. In the end, the best solution is to associate the data that belongs to a session with the session itself, making it independent on which thread it's running on. This is described by Knizhnik as "cumbersome", which is exactly why people should have not started with global variables in the first place. (No blame, Postgres is from 1986 and times were very different back then).

Re: PostgreSQL reconsiders its process-based model

#354

Post by pphysch »

devjab wrote: post #283
pphysch wrote: post #260

Earlier quoted context omitted.

The issue here isn't "rewriting" per se but "stopping development". You shouldn't stop development on your important products. Letting a couple of your talented programmers loose on a greenfield reimplementation is a perfectly sane strategic move. Stopping development on important products because you are 100% certain that the reimplementation will be successful by $DEADLINE is a foolish gamble.

Isn’t that how you end up with Python 2 and 3 though?

Yes, that was a rough migration process, but the long-term result is we have an improved language and growing community instead of Python going the way of PHP and Perl.

Between the 3 P's, Python's strategic decisions in 2000s were clearly the most successful.

And it wasn't a total rewrite.

Re: PostgreSQL reconsiders its process-based model

#355

Post by pphysch »

hurril wrote: post #317
pphysch wrote: post #260

Earlier quoted context omitted.

The issue here isn't "rewriting" per se but "stopping development". You shouldn't stop development on your important products. Letting a couple of your talented programmers loose on a greenfield reimplementation is a perfectly sane strategic move. Stopping development on important products because you are 100% certain that the reimplementation will be successful by $DEADLINE is a foolish gamble.

The big problem there is that the people you are letting loose on the alternative, are lost from the original, so O loses steam that A gains. You still have to produce bug fixes and features to _both_ O and A to keep them in sync. So you essentially have a doubled required production rate to be delivered using the same staff. So in order for there to be a net gain, the gang working on the alternative have to be able…

Getting some Mythical Man Month vibes here. Productivity isn't a zero-sum game.

Re: PostgreSQL reconsiders its process-based model

#356

Post by cakoose »

bourgeoismedia wrote: post #270
cakoose wrote: post #247

Earlier quoted context omitted.

I think there might be a terminology mix-up here. In C, a global variable with the `static` keyword is is still mutable. So it typically can't be constant-folded/inlined. The `static` modifier in that context just means that the symbol is not exported, so other ".c" files can't access it.

A static variable in C is mutable in the same sense that a local variable is, but since it's not visible outside the current compilation unit the optimizer is allowed to observe that it's never actually modified or published and constant fold it away. Check out the generated assembly for this simple program, notice that kBase is folded even though it's not marked const: https://godbolt.org/z/h45vYo5x5

It is also possible for a link-time optimizer to observe that a non-static global variable is never modified and optimize that away too.

But the Postgres mailing list is talking about 2000 global variables being a hurdle to multi-threading. I doubt they just didn't realize that most of them can be optimized into constants.

Re: PostgreSQL reconsiders its process-based model

#357

Post by anarazel »

cakoose wrote: post #356
bourgeoismedia wrote: post #270

Earlier quoted context omitted.

A static variable in C is mutable in the same sense that a local variable is, but since it's not visible outside the current compilation unit the optimizer is allowed to observe that it's never actually modified or published and constant fold it away. Check out the generated assembly for this simple program, notice that kBase is folded even though it's not marked const: https://godbolt.org/z/h45vYo5x5

It is also possible for a link-time optimizer to observe that a non-static global variable is never modified and optimize that away too. But the Postgres mailing list is talking about 2000 global variables being a hurdle to multi-threading. I doubt they just didn't realize that most of them can be optimized into constants.

Yea. Just about none of them could be optimized to constants because, uh, they're not constant. We're not perfect, but we do add const etc to TU level statics/globals that are actually read only. And if they are actually read only, we don't care about them in the context of threading anyway, since they wouldn't need any different behaviour anyway.

Re: PostgreSQL reconsiders its process-based model

#358

Post by CoolCold »

chasil wrote: post #163
blinkingled wrote: post #156

Earlier quoted context omitted.

https://oracle-base.com/articles/12c/multithreaded-model-usi... Probably still requires the parameter to be set.

Contrast this to Microsoft SQL Server: $ systemctl status mssql-server ● mssql-server.service - Microsoft SQL Server Database Engine Loaded: loaded (/usr/lib/systemd/system/mssql-server.service; disabled; vendor preset: disabled) Active: active (running) since Mon 2023-06-19 15:48:05 CDT; 1min 18s ago Docs: https://docs.microsoft.com/en-us/sql/linux Main PID: 2125 (sqlservr) Tasks: 123 CGroup: /system.slice/mssql-ser…

I'm not sure what I wonder on more - seeing its not enabled on boot or seeing mssql under systemd

Re: PostgreSQL reconsiders its process-based model

#359

Post by Sesse__ »

IshKebab wrote: post #75
Sesse__ wrote: post #30

Earlier quoted context omitted.

Here's MySQL's all-session-globals-in-one-place-class: https://github.com/mysql/mysql-server/blob/8.0/sql/sql_class... I believe I can safely say that nobody acknowledges and understands the complexity of all state within that class, and that whatever incentives there may be to simplify it are not enough for that to actually happen. (It ends on line 4692)

Right but that would still be true if they were globals instead. Putting all the globals in a class doesn't make any difference to how much state you have.

> Putting all the globals in a class doesn't make any difference to how much state you have.

I didn't make any claims about the _amount_ of state. My claim was that “you're forced to acknowledge and understand the complexity of your state” (i.e., moving it all together in one place helps understanding the state) is plain-out wrong.

Re: PostgreSQL reconsiders its process-based model

#360

Post by jeltz »

imtringued wrote: post #189
ComputerGuru wrote: post #132

Earlier quoted context omitted.

Because global variables can be confined to individual cpp files, exclusively visible in that compilation unit. It makes them far easier to reason with than hoisting them to the "global and globally visible" option if you just use a gargantuan struct. Which is why a more invasive refactor might be required.

Just use thread local variables. I abuse them for ridiculous things.

That is the plan for PostgreSQL.
Post reply on HN