Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

91–100 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#91

Post by megous »

refulgentis wrote: post #81
slashdev wrote: post #78

Earlier quoted context omitted.

Thank you for sharing your ideological views, but this is not the appropriate venue for that. If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that. All engineering is trade offs. As soon as you make blanket statements that X is always bad, you’ve transitioned into the realm of ideology. Now presumably you mean to say it’s…

Global mutable state being a poor choice in software architecture isn’t an ideology. There is no ideology that argues it is awesome. If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that. All engineering is trade offs. As soon as you start telling people they’re making blanket statements that X is always bad, you’ve transit…

Using globals is simpler, it's also pretty natural in event driven architectures. Passing everything via function arguments is welcome for library code, but there's little point to using it in application code. It just complicates things.

Re: PostgreSQL reconsiders its process-based model

#92

Post by cakoose »

loeg wrote: post #7
mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

Yeah. I think as a straightforward, easily correct transition from 2000 globals, a giant structure isn't an awful idea. It's not like the globals were organized before! You're just making the ambient state (awful as it is) explicit.

> I think as a straightforward, easily correct transition from 2000 globals, a giant structure isn't an awful idea.

Agree.

> It's not like the globals were organized before!

Using a struct with 2000 fields loses some encapsulation.

When a global is defined in a ".c" file (and not exported via a ".h" file), it can only be accessed in that one ".c" file, sort of like a "private" field in a class.

Switching to a single struct would mean that all globals can be accessed by all code.

There's probably a way to define things that allows you to regain some encapsulation, though. For example, some spin on the opaque type pattern: https://stackoverflow.com/a/29121847/163832

Re: PostgreSQL reconsiders its process-based model

#93

Post by ilyt »

comboy wrote: post #48
mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

I've never really been limited by CPU when running postgres (few TB instances). The bottleneck is always IO. Do others have different experience? Plus there's elegance and a feeling of being in control when you know query is associated with specific process which you can deal with and monitor just like any other process. But I'm very much clueless about internals, so this is a question rather than an opinion.

>I've never really been limited by CPU when running postgres (few TB instances). The bottleneck is always IO.

Throw a few NVMe drives at it and it might.

Re: PostgreSQL reconsiders its process-based model

#94

Post by ilyt »

shepardrtc wrote: post #29
mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

I think this is a situation where a message-passing Actor-based model would do well. Maybe pass variable updates to a single writer process/thread through channels or a queue. Years ago I wrote an algorithmic trader in Python (and Cython for the hotspots) using Multiprocessing and I was able to get away with a lot using that approach. I had one process receiving websocket updates from the exchange, another process wr…

That would most likely be several times slower than current model

Re: PostgreSQL reconsiders its process-based model

#95

Post by ilyt »

levkk wrote: post #68

Pretty sure Tom Lane said this will be a disaster in that same pgsql-hackers thread. Not entirely sure what benefits the multi-threaded model will have when you can easily saturate the entire CPU with just 128 connections and a pooler. So I doubt there is consensus or even strong desire from the community to undertake this boil the ocean project. On the other hand, having the ability to shut down and cleanup the enti…

>Not entirely sure what benefits the multi-threaded model will have when you can easily saturate the entire CPU with just 128 connections and a pooler.

That the all of those would work faster because of performance benefits, as mentioned in article

Re: PostgreSQL reconsiders its process-based model

#96

Post by topspin »

nextaccountic wrote: post #51
topspin wrote: post #38

Earlier quoted context omitted.

> I'm honestly surprised it took them so long to reach this conclusion. I'm not. You can get a long way with conventional IPC, and OS processes provide a lot of value. For most PostgreSQL instances the TLB flush penalty is at least 3rd or 4th on the list of performance concerns, far below prevailing storage and network bottlenecks. I share the concerns cited in this LWN story. Reworking this massive code base around…

From what I gather postgres isn't doing conventional IPC but instead it uses shared memory, which means the same mechanism threads use but with way higher complexity

As does Oracle, and others. I'm aware.

IPC, to me, includes the conventional shared memory resources (memory segments, locks, semaphores, condition variable, etc.) used by these systems: resources acquired by processes for the purpose of communication with other processes.

I get it though. The most general concept of shared memory is not coupled to an OS "process." You made me question whether my concept of term IPC was valid, however. So what does one do when a question appears? Stop thinking immediately and consult a language model!

Q: Is shared memory considered a form of interprocess communication?

GPT-4: Yes, shared memory is indeed considered a form of interprocess communication (IPC). It's one of the several mechanisms provided by an operating system to allow processes to share and exchange data.

...

Why does citing ChatGPT make me feel so ugly inside?

Re: PostgreSQL reconsiders its process-based model

#97

Post by baggy_trough »

xwdv wrote: post #33
baggy_trough wrote: post #8

I hope they are conservative about this, because even the smartest and best programmers in the world cannot create bug free multithreaded code.

Nonsense, multithreaded code can be written as bug free as regular code. No need to fear.

In theory, yes. In practice, no.

Re: PostgreSQL reconsiders its process-based model

#98

Post by ilyt »

ethbr0 wrote: post #53
devit wrote: post #42

Earlier quoted context omitted.

Use pgbouncer

Was curious about this as an architectural solution as well. We're really talking about X-per-client as the primary reason to move away from processes, right? So if you can get most of the benefit via pooling... why inherit the pain of porting? Presumably latency jitter would be a difficult problem with pools, but it seems easier (and safer) than porting processes -> threads. Disclaimer: High performance / low latenc…

The reasons are explained in article. Read the article

Re: PostgreSQL reconsiders its process-based model

#99

Post by 0xbadcafebee »

So compromise. Take the current process model, add threading and shared memory, with feature flags to limit number of processes and number of threads.

Want to run an extension that isn't threadsafe? Run with 10 processes, 1 threads. Want to run high-performance? Run with 1 process, 10 threads. Afraid of "stability issues"? Run with 1 process, 1 thread.

Will it be hard to do? Sure. Impossible? Not at all. Plan for it, give a very long runway, throw all your new features into the next major version branch, and tell people everything else is off the table for the next few years. If you're really sure threading is going to be increasingly necessary, better to start now than to wait until it's too late. But this idea of "oh it's hard", "oh it's dangerous", "too complicated", etc is bullshit. We've built fucking spaceships that visit other planets. We can make a database with threads that doesn't break. Otherwise we admit that basic software development using practices from the past 30 years is too much for us to figure out.

Re: PostgreSQL reconsiders its process-based model

#100

Post by aseipp »

uudecoded wrote: post #71

I am going to go ahead and trust Tom Lane on this one, over someone who is working on "serverless Postgres". Godspeed to the forthcoming fork.

Heikki Linnakangas is one of the top Postgres contributors of all time, he isn't just "someone." The fact he's working for a startup on a fork (that already exists, which you can run right now on your local machine) doesn't warrant any snide dismissal. Robert Haas admitted that it would be a huge amount of work and that it would only be achievable by a small few people anyway, Heikki being among them.

Anyway, I think there are definitely limits that are starting to appear with Postgres in some spots. This is probably one of the most difficult possible solutions to some of those problems, but even if they don't switch to a fully threaded model, being more CPU efficient, better connection handling, etc will all go a substantial way. Doing some of the really hard work is better than none of it, probably.

Post reply on HN