Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

231–240 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#231

Post by progmetaldev »

abhibeckert wrote: post #224
eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

Reminds me of PHP 6... For those who don't follow PHP closely - that version was an attempted refactor of the string implementation which essentially shut down nearly all work on PHP for a decade, stagnating the language until it became pretty terrible compared to other options. They finally gave up and started work on PHP 7 which uses the (perfectly good) PHP 5 strings. Ten years of wasted time by the best internal…

I've used PHP in the past (PHP 4 and 5), as well as some simple templated projects in PHP 7. I try to keep up on news with what is happening in the PHP world, and it's difficult because of the hate for the language. Is the solution to Unicode strings still to just use the "mb_*" functions?

I got my real professional start using PHP, and have built even financial systems in the language (since ported to .NET 6 for my ease of maintenance, and better number handling). I'm still very interested in the language itself, in case I ever have the need to freelance or provide a solution to a client that can't afford what I can build in .NET (although to be honest, at this point I'm roughly able to code at the same speed in .NET as in PHP, but with the added type-safety, although I know PHP has really stepped up in providing this).

Re: PostgreSQL reconsiders its process-based model

#232

Post by Icathian »

gjvc wrote: post #210
Icathian wrote: post #209

Earlier quoted context omitted.

Uh. Heikki is definitely not just "some guy". Dude is one of the top contributors to Postgres.

How does that make him immune to having dumb ideas? See, I'm judging the idea on merit. You're just defending your hero who has gone rogue.

I must have missed all the nuanced judgment in your original post. Maybe you can quote some for me.

Re: PostgreSQL reconsiders its process-based model

#233

Post by ethbr0 »

ilyt wrote: post #98
ethbr0 wrote: post #53

Earlier quoted context omitted.

Was curious about this as an architectural solution as well. We're really talking about X-per-client as the primary reason to move away from processes, right? So if you can get most of the benefit via pooling... why inherit the pain of porting? Presumably latency jitter would be a difficult problem with pools, but it seems easier (and safer) than porting processes -> threads. Disclaimer: High performance / low latenc…

The reasons are explained in article. Read the article

I appear to have missed them, then.

Could you point out, aside from the large numbers of clients I mentioned (and the development overhead of implementing multi-process memory management code), what the article mentions is a primary drawback of using processes over threads?

Re: PostgreSQL reconsiders its process-based model

#234

Post by JdeBP »

tragomaskhalos wrote: post #19

Worked on a codebase which was separate processes, each of which has a shedload of global variables. It was a nightmare working out what was going on, not helped by the fact that there was no naming convention for the globals, plus they were not declared in a single place. I believe their use was a performance move, ie having the linker pin a var to a specific memory location rather than copying it to the stack as a…

Per discussion on this very page, in the headlined article, and in the mailing list discussion it references, PostgreSQL is not in that category. It has lots of static storage duration variables, which do not necessarily have external linkage.

Robert Haas pointed out in one message that an implementation pattern was to use things like file-scope static storage duration variables to provide session-local state for individual components. This is why they've been arguing against a single giant structure declared in "session.h" as an approach, as it requires every future addition to session state to touch the central core of the entire program.

They want to keep the advantage of the fact that these variables are in fact not global. They are local; and the problem is rather that they have static storage duration and are not per-thread, and thus are not per-session in a thread-per-session model.

Re: PostgreSQL reconsiders its process-based model

#235

Post by anarazel »

wmf wrote: post #191
bb88 wrote: post #134

This reminds me of this poster: "You must be this tall..." https://bholley.net/blog/2015/must-be-this-tall-to-write-mul... Back about a decade ago I was "auditing" someone else's threaded code. And couldn't figure it out. But he was the company's "golden child" so by default it must be working code because he wrote it. And then it started causing deadlocks in prod. "What do you want me to do about it? It's the golden…

The thing is... multi-process with a bespoke shared memory system isn't better than multithreading; it's much worse.

I'm not sure if I'd judge it as harshly, but you have a good point: A lot of debugging / validation tooling understands threads, but not memory shared between processes.

Re: PostgreSQL reconsiders its process-based model

#236

Post by anarazel »

ethbr0 wrote: post #53
devit wrote: post #42

Earlier quoted context omitted.

Use pgbouncer

Was curious about this as an architectural solution as well. We're really talking about X-per-client as the primary reason to move away from processes, right? So if you can get most of the benefit via pooling... why inherit the pain of porting? Presumably latency jitter would be a difficult problem with pools, but it seems easier (and safer) than porting processes -> threads. Disclaimer: High performance / low latenc…

pgbouncer is not transparent, you loose features, particularly when using the pooling mode actually allowing a larger number of active concurrent connections. Solving those issues is a lot easier with threads than with processes.

Re: PostgreSQL reconsiders its process-based model

#237

Post by robomc »

abhibeckert wrote: post #224
eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

Reminds me of PHP 6... For those who don't follow PHP closely - that version was an attempted refactor of the string implementation which essentially shut down nearly all work on PHP for a decade, stagnating the language until it became pretty terrible compared to other options. They finally gave up and started work on PHP 7 which uses the (perfectly good) PHP 5 strings. Ten years of wasted time by the best internal…

I have the "Professional PHP6" book which I feel like should be a collectors item or something.

Weird book IMO, because it has a lot of content that's just about general software development, rather than anything to do with PHP specifically, or the theoretical PHP6 APIs in particular.

Re: PostgreSQL reconsiders its process-based model

#238

Post by throwawaylinux »

wmf wrote: post #191
bb88 wrote: post #134

This reminds me of this poster: "You must be this tall..." https://bholley.net/blog/2015/must-be-this-tall-to-write-mul... Back about a decade ago I was "auditing" someone else's threaded code. And couldn't figure it out. But he was the company's "golden child" so by default it must be working code because he wrote it. And then it started causing deadlocks in prod. "What do you want me to do about it? It's the golden…

The thing is... multi-process with a bespoke shared memory system isn't better than multithreading; it's much worse.

By bespoke you mean using standard interfaces to create shared memory pools?

They do roll some of their own locking primitives, but that's not particularly unusual in a large portable program (and quite likely what they wanted is/was not available in glibc or other standard libraries, at least when first written).

Re: PostgreSQL reconsiders its process-based model

#239

Post by __turbobrew__ »

wmf wrote: post #191
bb88 wrote: post #134

This reminds me of this poster: "You must be this tall..." https://bholley.net/blog/2015/must-be-this-tall-to-write-mul... Back about a decade ago I was "auditing" someone else's threaded code. And couldn't figure it out. But he was the company's "golden child" so by default it must be working code because he wrote it. And then it started causing deadlocks in prod. "What do you want me to do about it? It's the golden…

The thing is... multi-process with a bespoke shared memory system isn't better than multithreading; it's much worse.

In Linux, multi process with shared memory regions is basically just threads. The kernel doesn’t know anything about threads, it knows about processes and it lets you share memory regions between those processes if you so desire.

Re: PostgreSQL reconsiders its process-based model

#240

Post by rcxdude »

dfox wrote: post #203
ilyt wrote: post #93

Earlier quoted context omitted.

>I've never really been limited by CPU when running postgres (few TB instances). The bottleneck is always IO. Throw a few NVMe drives at it and it might.

Throw a ridiculous amount of RAM at it is more correct assessment. NVMe reads are still an “I/O” and that is slow. And for at least 10 years buying enough RAM to have all off the interesting parts of OLTP psql database either in shared_buffers or in the OS-level buffer cache is completely feasible.

an array of modern SSDs can get to a similar bandwidth to RAM, albeit with significantly worse latency still. It's not that hard to push the bottleneck elsewhere in a lot of workloads. High performance fileservers, for example, need pretty beefy CPUs to keep up.
Post reply on HN