Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

81–90 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#81

Post by refulgentis »

slashdev wrote: post #78
jupp0r wrote: post #73

Please don't use mutable global state in your work. Global variables are universally bad and don't provide much of a benefit. The number of desirable architectural refactoring that I've witnessed turning into a muddy mess because of them is daunting. This is one more example of this.

Thank you for sharing your ideological views, but this is not the appropriate venue for that. If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that. All engineering is trade offs. As soon as you make blanket statements that X is always bad, you’ve transitioned into the realm of ideology. Now presumably you mean to say it’s…

Global mutable state being a poor choice in software architecture isn’t an ideology. There is no ideology that argues it is awesome.

If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that.

All engineering is trade offs. As soon as you start telling people they’re making blanket statements that X is always bad, you’ve transitioned into the realm of nitpicking.

Re: PostgreSQL reconsiders its process-based model

#82

Post by ddorian43 »

ethbr0 wrote: post #53
devit wrote: post #42

Earlier quoted context omitted.

Use pgbouncer

Was curious about this as an architectural solution as well. We're really talking about X-per-client as the primary reason to move away from processes, right? So if you can get most of the benefit via pooling... why inherit the pain of porting? Presumably latency jitter would be a difficult problem with pools, but it seems easier (and safer) than porting processes -> threads. Disclaimer: High performance / low latenc…

> We're really talking about X-per-client as the primary reason to move away from processes, right?

Many other things too. Like better sharing of caches. Lower overhead of thread instead of process. Etc. (read the thread)

Re: PostgreSQL reconsiders its process-based model

#83

Post by wielebny »

Having been using and administering a lot of PostgreSQL servers, I hope they don't lose any stability over this.

I've seen (and reported) bugs that caused panics/segfaults in specific psql processes. Not just connections, also processes related to wal writing or replication. The way it's built right now, a child process can be just forced to quit and it does not affect other processes. Hopefully switching into thread won't force whole PostgreSQL to panic and shut down.

Re: PostgreSQL reconsiders its process-based model

#84

Post by megous »

saulrh wrote: post #13
mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

Also, even if a 2k-member structure is obnoxious, consider the alternative - having to think about and manage 2k global variables is probably even worse!

Each set of globals is in a module it relates to, not in some central file where everything has to be in one struct.

If anything, it's probably easier to understand.

Re: PostgreSQL reconsiders its process-based model

#85

Post by BoardsOfCanada »

levkk wrote: post #68

Pretty sure Tom Lane said this will be a disaster in that same pgsql-hackers thread. Not entirely sure what benefits the multi-threaded model will have when you can easily saturate the entire CPU with just 128 connections and a pooler. So I doubt there is consensus or even strong desire from the community to undertake this boil the ocean project. On the other hand, having the ability to shut down and cleanup the enti…

From the article:

> Tom Lane said: "I think this will be a disaster. There is far too much code that will get broken". He added later that the cost of this change would be "enormous", it would create "more than one security-grade bug", and that the benefits would not justify the cost.

Re: PostgreSQL reconsiders its process-based model

#86

Post by nneonneo »

sargun wrote: post #60

I'm curious if they can take advantage of vfork / CLONE_VM, to get the benefits of sharing memory and lower overhead context switches, with the trade of still getting benefits from the scheduler, and sysadmin-friendliness. The other thing that might be interesting is FUTEX_SWAP / UMCG. Although it doesn't remove the overhead induced by context switches entirely (specifically, you would still deal with TLB misses), yo…

Per the article, Postgres has many, many global variables, many of which track per-session state; much session state is “freed” via process exit rather than being explicitly cleaned up. Switching to CLONE_VM requires these problems to all be solved.

Re: PostgreSQL reconsiders its process-based model

#87

Post by megous »

mattashii wrote: post #45
RcouF1uZ4gsC wrote: post #4

This would be one those places where a language like Rust would be helpful. In C/C++ with undefined behavior and crashes, process isolation makes a lot of sense to limit the blast radius. Rust borrow checker gives you at compile time a lot of the safety that you would rely on process isolation for.

Yes, but note that the blast radius of a PostgreSQL process crash is already "the whole system reboots", so there are not a lot of differences between process- and thread-based PostgreSQL written in C. Rewriting in Rust would be interesting, but it would also probably be too invasive to make it worthwile at all - all code in PostgreSQL is C, while not all code in PostgreSQL interacts with the intrinsics of processes…

PostgreSQL process crash may also just be, one query fails.

Re: PostgreSQL reconsiders its process-based model

#88

Post by zackmorris »

EGreg wrote: post #14

I hope they don't do it. I've had a similar situation with PHP, where we had written quite a large engine ( https://github.com/Qbix/Platform ) with many features ( https://qbix.com/features.pdf ) . It took advantage of the fact that PHP isolated each script and gave it its own global variables, etc. In fact, much of the request handling did stuff like this: Q_Request::requireFields(['a', 'b', 'c']); $uri = Q_Dispatch…

I've been using PHP for decades and have found its isolated process model to be about the best around, certainly for any mainstream language. Also Symfony's Process component encapsulates most of the errata around process management in a cross-platform way:

https://symfony.com/doc/current/components/process.html

Going from a working process implementation to async/threads with shared memory is pretty much always a mistake IMHO, especially if it's only done for performance reasons. Any speed gains will be eclipsed by endless whack-a-mole bug fixes, until the code devolves into something unrecognizable. Especially when there are other approaches similar to map-reduce and scatter-gather arrays where data is processed in a distributed fashion and then joined into a final representation through mechanisms like copy-on-write, which are supported by very few languages outside of PHP and the functional programming world.

The real problem here is the process spawning and context-switching overhead of all versions of Windows. I'd vote to scrap their process code in its entirety and write a new version based on atomic operations/lists/queues/buffers/rings with no locks and present an interface which emulates the previous poor behavior, then run it through something like a SAT solver to ensure that any errata that existing software depends on is still present. Then apps could opt to use the direct unix-style interface and skip the cruft, or refactor their code to use the new interface.

Apple did something similar to this when OS X was released, built on a mostly POSIX Darwin, NextSTEP, Mach and BSD Unix. I have no idea how many times Microsoft has rewritten their process model or if they've succeeded in getting performance on par with their competitors (unlikely).

Edit: I realized that the PHP philosophy may not make a lot of sense to people today. In the 90s, OS code was universally terrible, so for example the graphics libraries of Mac and Windows ran roughly 100 times slower than they should for various reasons, and developers wrote blitters to make it possible for games to run in real time. That was how I was introduced to programming. PHP encapsulated the lackluster OS calls in a cross-platform way, using existing keywords from popular languages to reduce the learning curve to maybe a day (unlike Perl/Ruby, which are weird in a way that can be fun but impractical to grok later). So it's best to think of PHP more like something like Unity, where the nonsense is abstracted and developers can get down to business. Even though it looks like Javascript with dollar signs on the variables. It's also more like the shell, where it tries to be as close as possible to bare-metal performance, even while restricted to the 100x interpreter slowdown of languages like Python. I find that PHP easily saturates the processor when doing things in a data-driven way by piping bytes around.

Re: PostgreSQL reconsiders its process-based model

#89

Post by mhh__ »

SoylentOrange wrote: post #79
Alekhine wrote: post #50

Earlier quoted context omitted.

I have no idea why that isn't standard practice in every codebase. I should be able to figure out your code without having to ask, or dig through issues or commit messages. Just tell me what it's for!

Because it takes a lot of time and because the comments can get outdated. I also want this for all my code bases. But do I always do this myself? No, especially on green field projects. I will sometimes go back and annotate them later.

They can get outdated but they usually don't. It's a good litmus test for if a file is too big / small if it's purpose is hard to nail down.

Re: PostgreSQL reconsiders its process-based model

#90

Post by mynonameaccount »

"the benefits would not justify the cost". PostgreSQL, like any software, at some point in it's life need to be refactored. Why not refactor with a thread model. Of course there will be bugs. Of course it will be difficult. But I think it is a worthwhile endeavor. Doesn't sound like this will happen but a new project would be cool.
Post reply on HN