Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

181–190 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#181

Post by anarazel »

masklinn wrote: post #32
cactusfrog wrote: post #26

Earlier quoted context omitted.

This is different because there isn’t a whole ecosystem of packages that depend on access to a thread unsafe C API. Getting the GIL out of core Python isn’t too challenging. Getting all of the packages that depend on Python’s C API working is.

An other component of the Gil story is that removing the Gil require adding fine grained locks, which (aside from making VM development more complicated) significantly increases lock traffic and thus runtime costs, which noticeably impacts single-threaded performance, which is of major import. Postgres starts from a share-nothing architecture, it’s quite a bit easier to evaluate the addition of sharing.

Postgres already shares a lot of state between processes via shared memory. There's not a whole lot that would initially change from a concurrency perspective.

Re: PostgreSQL reconsiders its process-based model

#182

Post by Yoric »

mattashii wrote: post #150
niccl wrote: post #139

Earlier quoted context omitted.

yes, but postmaster is still running to roll back the transaction. If you crash a single multi-threaded process, you may lose postmaster as well and then sadness would ensue

The threaded design wouldn't necessarily be single-process, it would just not have 1 process for every connection. Things like crash detection could still be handled in a separate process. The reason to use threading in most cases is to reduce communication and switching overhead, but for low-traffic backends like a crash handler the overhead of it being a process is quite limited - when it gets triggered context swi…

Seconded. For instance, Firefox' crash reporter has always been a separate process, even at the time Firefox was mostly single-process, single-threaded. Last time I checked, this was still the case.

Re: PostgreSQL reconsiders its process-based model

#183

Post by ants_a »

usefulcat wrote: post #121
johannes1234321 wrote: post #63

Earlier quoted context omitted.

The code already is multithreaded. They have shared state just across multiple processes instead of threads within a process. They might even reduce complexity that way.

It's not the same at all for global variables, of which pgsql apparently has around a couple thousand. If every process is single threaded, you don't have to consider the possibility of race conditions when accessing any of those ~2000 global variables. And you can pretty much guarantee that little if any of the existing code was written with that possibility in mind.

Those global variables would be converted to thread locals and most of the code would be oblivious of the change. This is not the hard part of the change.

Re: PostgreSQL reconsiders its process-based model

#184

Post by anarazel »

newaccount74 wrote: post #105

A big advantage of the process-based model is its resilience against many classes of errors. If a bug in PostgreSQL (or in an extension) causes the server to crash, then only that process will crash. Postmaster will detect the child process termination, and send an error message to the client. The connection will be lost, but other connections will be unaffected. It's not foolproof (there are ways to bring the whole…

We would still have a separate supervisor process of we moved connections to threads.

Re: PostgreSQL reconsiders its process-based model

#185

Post by wizofaus »

SigmundA wrote: post #178
wizofaus wrote: post #141

Earlier quoted context omitted.

You're implying that only an OS can provide memory separation between units of execution - at least in .NET AppDomains give you the same protection within a single process, so why couldn't postgres have its own such mechanism? I'd also think with a database engine shared state is not just in-memory - i.e. one process can potentially corrupt the behaviour of another by what it writes to disk, so moving to a single-pro…

No AppDomains are not as good as processes, I have tried to go that route before, you cannot stop unruly code reliably in an app domain (you must use thread.abort() which is not good) and memory can still leak in any native code used there. The only reliable way to stop bad code like say an infinite loop is to run in another process even in .Net. They also removed Appdomain in later versions of .Net because they had…

Not claiming they're as good, just noting that there are alternative ways to provide memory barriers, though obviously if it's not enforced at the language/runtime level, it requires either super strong developer disciple or the use of some other tool to do so. I can't find anything suggesting AppDomains have been removed completely though, just they're not fully supported on non-Windows platforms, which is interesting, I wonder if that means they do have OS-level support.

Re: PostgreSQL reconsiders its process-based model

#186

Post by tylerhou »

bb88 wrote: post #166
carstenhag wrote: post #126

Earlier quoted context omitted.

The person probably implied that Postgres should switch to another toolchain that guarantees more things at compile time, so probably Rust.

You can take a chunk of code and just rewrite it in Rust. You'll learn a lot quickly by this.

The boundaries within database code are not clear. There are too many interlocking parts to take a nontrivial chunk and rewrite it Rust.

Re: PostgreSQL reconsiders its process-based model

#187

Post by [deleted] »

idiomaticrust wrote: post #115
eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

He is right. Such rewrites cause a lot of problems if your compiler doesn't help you with avoiding data races. But there is another way.

[deleted]

Re: PostgreSQL reconsiders its process-based model

#188

Post by duped »

eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

That's an awful message with the only sensible reply.

Re: PostgreSQL reconsiders its process-based model

#189

Post by imtringued »

ComputerGuru wrote: post #132
FooBarWidget wrote: post #108

Earlier quoted context omitted.

I don't get it. How is a 2000-member structure any different from having 2000 global variables? How is maintaining the struct possibly harder than maintaining the globals? Refactoring globals to struct members is semantically nearly identical, it may as well just be a mechanical, cosmetic change, while also giving the possibility to move to a threaded architecture.

Because global variables can be confined to individual cpp files, exclusively visible in that compilation unit. It makes them far easier to reason with than hoisting them to the "global and globally visible" option if you just use a gargantuan struct. Which is why a more invasive refactor might be required.

Just use thread local variables.

I abuse them for ridiculous things.

Re: PostgreSQL reconsiders its process-based model

#190

Post by SigmundA »

wizofaus wrote: post #185
SigmundA wrote: post #178

Earlier quoted context omitted.

No AppDomains are not as good as processes, I have tried to go that route before, you cannot stop unruly code reliably in an app domain (you must use thread.abort() which is not good) and memory can still leak in any native code used there. The only reliable way to stop bad code like say an infinite loop is to run in another process even in .Net. They also removed Appdomain in later versions of .Net because they had…

Not claiming they're as good, just noting that there are alternative ways to provide memory barriers, though obviously if it's not enforced at the language/runtime level, it requires either super strong developer disciple or the use of some other tool to do so. I can't find anything suggesting AppDomains have been removed completely though, just they're not fully supported on non-Windows platforms, which is interesti…

https://learn.microsoft.com/en-us/dotnet/api/system.appdomai...

"On .NET Core, the AppDomain implementation is limited by design and does not provide isolation, unloading, or security boundaries. For .NET Core, there is exactly one AppDomain. Isolation and unloading are provided through AssemblyLoadContext. Security boundaries should be provided by process boundaries and appropriate remoting techniques."

AppDomains pretty much only allowed you to load unload assemblies and provided little else. If you wanted to stop bad code you still used Thread.Abort which left your runtime in a potentially bad state due to no isolation between threads.

The only way to do something like an AppDomain to replace process isolation would be to re-write the whole OS in a memory safe language similar to https://en.wikipedia.org/wiki/Midori_(operating_system) / https://en.wikipedia.org/wiki/Singularity_(operating_system)

Post reply on HN