Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

111–120 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#111

Post by slashdev »

refulgentis wrote: post #81
slashdev wrote: post #78

Earlier quoted context omitted.

Thank you for sharing your ideological views, but this is not the appropriate venue for that. If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that. All engineering is trade offs. As soon as you make blanket statements that X is always bad, you’ve transitioned into the realm of ideology. Now presumably you mean to say it’s…

Global mutable state being a poor choice in software architecture isn’t an ideology. There is no ideology that argues it is awesome. If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that. All engineering is trade offs. As soon as you start telling people they’re making blanket statements that X is always bad, you’ve transit…

It's awesome where performance considerations are paramount. It's awesome in databases. It's awesome in embedded software. It's awesome in operating system kernels.

The fact is sometimes it's good. Saying it's universally bad is going beyond the realm of logic and evidence and into the realm of ideology.

Re: PostgreSQL reconsiders its process-based model

#113

Post by rburhum »

Sorry if I offend anybody, but this sounds like such a bad idea. I have been running various versions of postgres in production for 15 years with thousands of processes on super beefy machines, and I can tell you without a doubt that sometimes those processes crash - specially if you are running any of the extensions. Nevertheless, Postgres has 99% of the time proven to be resilient. The idea that a bad client can bring the whole cluster down because it hit a bug sounds scary. Every try creating a spatial index on thousands/millions of records that have nasty overly complex or badly digitized geometries? Sadly, crashes are part of that workflow, and changing this from process to threading would mean all the other clients also crashing and cutting connections. This as a potential problem because I want to avoid context switching overhead or cache misses, no thanks.

Re: PostgreSQL reconsiders its process-based model

#115

Post by idiomaticrust »

eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

He is right. Such rewrites cause a lot of problems if your compiler doesn't help you with avoiding data races.

But there is another way.

Re: PostgreSQL reconsiders its process-based model

#116

Post by withinboredom »

SoylentOrange wrote: post #79
Alekhine wrote: post #50

Earlier quoted context omitted.

I have no idea why that isn't standard practice in every codebase. I should be able to figure out your code without having to ask, or dig through issues or commit messages. Just tell me what it's for!

Because it takes a lot of time and because the comments can get outdated. I also want this for all my code bases. But do I always do this myself? No, especially on green field projects. I will sometimes go back and annotate them later.

Even outdated comments can tell you the original purpose of the code, which helps if you're looking for a bug. Especially if you're looking for a bug.

If someone didn't take the time to update the comments and the reviewers didn't point it out, then you've probably found the bug because someone was cowboying some shitty code.

Re: PostgreSQL reconsiders its process-based model

#117

Post by ComputerGuru »

Diggsey wrote: post #61
comboy wrote: post #48

Earlier quoted context omitted.

I've never really been limited by CPU when running postgres (few TB instances). The bottleneck is always IO. Do others have different experience? Plus there's elegance and a feeling of being in control when you know query is associated with specific process which you can deal with and monitor just like any other process. But I'm very much clueless about internals, so this is a question rather than an opinion.

It's not just CPU - memory usage is also higher. In particular, idle connections still consume signficant memory, and this is why PostgreSQL has so much lower connection limits than eg. MySQL. Pooling can help in some cases, but pooling also breaks some important PostgreSQL features (like prepared statements...) since poolers generally can't preserve session state. Other features (eg. notify) are just incompatible wi…

> solvable without a full switch to a threaded model (eg. by having pooling built-in and session-state-aware).

Yeeeeesssss, but solving that is solving the hardest part of switching to a threaded model. It requires the team to come terms with the global state and encapsulating session state in a non-global struct.

Re: PostgreSQL reconsiders its process-based model

#118

Post by ComputerGuru »

Alekhine wrote: post #50
BoardsOfCanada wrote: post #20

I recently looked through the source code of postgresql and every source files starts with a (really good) description of what the file is supposed to do, which made it really easy to get in to the code compared to other open source projects I've seen. So thanks for that.

I have no idea why that isn't standard practice in every codebase. I should be able to figure out your code without having to ask, or dig through issues or commit messages. Just tell me what it's for!

It kind of is in rust now, with module-level documentation given its own specific AST representation instead of just being a comment at the top of the file (a file is a module).

Re: PostgreSQL reconsiders its process-based model

#119

Post by hnarn »

idiomaticrust wrote: post #115
eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

He is right. Such rewrites cause a lot of problems if your compiler doesn't help you with avoiding data races. But there is another way.

> But there is another way.

Ok?

Re: PostgreSQL reconsiders its process-based model

#120

Post by blinkingled »

pjmlp wrote: post #112

So when we as a whole decided that multiprocessing is a much better approach from security and application stability point of view, they decide to go with threads?

Horses for courses I guess - purely threaded vs purely MP both have different set of tradeoffs and shoehorning one over the other always fails some use cases. The article says they are also considering the possibility of having to keep both process and thread models indefinitely for this and other reasons.

I know nothing of PG internals but I can see why process per connection model doesn't work for large machines and/or high number of connections. One way to do it would be to keep connection handling per thread and still keep multiprocess approach where it makes sense for security and doesn't add linear overheads.

Post reply on HN