Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

291–300 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#291

Post by jupp0r »

megous wrote: post #91
refulgentis wrote: post #81

Earlier quoted context omitted.

Global mutable state being a poor choice in software architecture isn’t an ideology. There is no ideology that argues it is awesome. If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that. All engineering is trade offs. As soon as you start telling people they’re making blanket statements that X is always bad, you’ve transit…

Using globals is simpler, it's also pretty natural in event driven architectures. Passing everything via function arguments is welcome for library code, but there's little point to using it in application code. It just complicates things.

The problems it causes for Postgres are outlined in the article on LWN.

Re: PostgreSQL reconsiders its process-based model

#292

Post by kristiandupont »

eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

Yeah.

Without being familiar with the Postgres source, this seems to be what I call a "somersault problem": hard to break down into sub-goals. I have heard that the Postgres codebase is solid which makes it easier but it's still mature and highly complex. It doesn't sound feasible to me.

[link redacted]

Re: PostgreSQL reconsiders its process-based model

#293

Post by mindwok »

Shorel wrote: post #265
rburhum wrote: post #113

Sorry if I offend anybody, but this sounds like such a bad idea. I have been running various versions of postgres in production for 15 years with thousands of processes on super beefy machines, and I can tell you without a doubt that sometimes those processes crash - specially if you are running any of the extensions. Nevertheless, Postgres has 99% of the time proven to be resilient. The idea that a bad client can br…

Reading your comment makes me think it is not only a good idea, it is a necessity. Relying on crashing as a bug recovery system is a good idea? Crashing is just part of the workflow? That's insane, and a good argument against PostgreSQL in any production system. It is possible PostgreSQL doesn't migrate to a thread based model, and I am not arguing they should. But debug and patch the causes of these crashes? Absolut…

They are still debugging and patching the causes. The crash detection is just to try and prevent a single bug from bringing down the whole system.

Re: PostgreSQL reconsiders its process-based model

#294

Post by szundi »

wizofaus wrote: post #141
vbezhenar wrote: post #137

Earlier quoted context omitted.

Of course it will. That's better than continue working with damaged memory structures and unpredictable consequences. For database it's more important than ever. Imagine writing corrupted data because other thread went crazy.

You're implying that only an OS can provide memory separation between units of execution - at least in .NET AppDomains give you the same protection within a single process, so why couldn't postgres have its own such mechanism? I'd also think with a database engine shared state is not just in-memory - i.e. one process can potentially corrupt the behaviour of another by what it writes to disk, so moving to a single-pro…

For a decades old codebase probably only the OS can.

Point is it getting worse if this is changed.

Re: PostgreSQL reconsiders its process-based model

#296

Post by neilwilson »

It's always amazed me with databases why they don't go the other way.

Create an operating system specifically for the database and make it so you boot the database.

Databases seem to spend most of their time working around the operating system abstractions. So why not look at the OS, and streamline it for database use - dropping all the stuff a database will never need.

That then is a completely separate project which is far easier to get started rather than shoehorning the database into an operating system thread model that is already a hack of the process model.

Re: PostgreSQL reconsiders its process-based model

#297

Post by thesnide »

cmrdporcupine wrote: post #206

It sounds like the specific concerns here are actually around buffer pool management performance in and around the TLB: "Once you have a significant number of connections we end up spending a *lot* of time in TLB misses, and that's inherent to the process model, because you can't share the TLB across processes. " Many of the comments here seem to be missing this and talking about CPU-boundedness generally and thread-…

Exactly my thinking. If the problem is TLB evictions, why not improve these? PG isn't the only software that is hit by those.

And if you carefully craft your CPU scheduling and address spaces mapping, you can reduce them by a lot.

Yet rewrites are always easier and more sexy. At first.

Re: PostgreSQL reconsiders its process-based model

#298

Post by withinboredom »

jgilias wrote: post #269
elteto wrote: post #158

Earlier quoted context omitted.

Indeed. I recently found this piece of code: if (X) assert(false); // we never do X, ever, anywhere. Then I look over to the other pane, where I have a different, but related file open: if (exact same X) { do_useful_stuff(); } It got a chuckle out of me.

Did you update the comment? :-)

// there are two kinds of mutually exclusive commentors

enum kinds { writers; readers; updaters; }

Re: PostgreSQL reconsiders its process-based model

#299

Post by kqr »

abhibeckert wrote: post #224
eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

Reminds me of PHP 6... For those who don't follow PHP closely - that version was an attempted refactor of the string implementation which essentially shut down nearly all work on PHP for a decade, stagnating the language until it became pretty terrible compared to other options. They finally gave up and started work on PHP 7 which uses the (perfectly good) PHP 5 strings. Ten years of wasted time by the best internal…

On the other hand, there's also the case of the Lunar Module guidance software that was hard-coded to run exactly every two seconds. If the previous subroutine call was still running when the next one was due, the previous one was harshly terminated (with weird side effects).

One of the main programmers suggested making it so that the next guidance routine wouldn't run until the previous one was done. This would make the code less sensitive to race conditions and allow more useful functionality for the pilots (who were the actual users and did seem to want it). However everyone assumed the two-second constant was implicitly embedded everywhere.

It wasn't -- only in a few places -- and with that fixed the code got more general and the proof of concept ran better than ever in about every simulator available. The amount of control it gave pilots was years ahead of the curve. But it never got a chance to fly on a real mission because what was there was "good enough" and nobody bothered to try.

In our combined comments there's a lesson about growing experiments and figuring out how to achieve failure quickly.

Re: PostgreSQL reconsiders its process-based model

#300

Post by dialogbox »

neilwilson wrote: post #296

It's always amazed me with databases why they don't go the other way. Create an operating system specifically for the database and make it so you boot the database. Databases seem to spend most of their time working around the operating system abstractions. So why not look at the OS, and streamline it for database use - dropping all the stuff a database will never need. That then is a completely separate project whic…

I'm not sure what you mean by OS. If you mean a whole new kernel, it will take decades. They can support only small number of HW. If you mean a specialized linux distro, many companies does that already.

I don't know how that can make it easier the process based / thread based problem.

Post reply on HN