Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

301–310 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#301

Post by girvo »

neilwilson wrote: post #296

It's always amazed me with databases why they don't go the other way. Create an operating system specifically for the database and make it so you boot the database. Databases seem to spend most of their time working around the operating system abstractions. So why not look at the OS, and streamline it for database use - dropping all the stuff a database will never need. That then is a completely separate project whic…

That was/is part of the promise of the whole unikernel thing, no?

https://mirage.io/ or similar could then let you boot your database. That said, it's not really taken off from what I can tell, so I'm guessing there's more to it than that.

Re: PostgreSQL reconsiders its process-based model

#302

Post by dialogbox »

Shorel wrote: post #265
rburhum wrote: post #113

Sorry if I offend anybody, but this sounds like such a bad idea. I have been running various versions of postgres in production for 15 years with thousands of processes on super beefy machines, and I can tell you without a doubt that sometimes those processes crash - specially if you are running any of the extensions. Nevertheless, Postgres has 99% of the time proven to be resilient. The idea that a bad client can br…

Reading your comment makes me think it is not only a good idea, it is a necessity. Relying on crashing as a bug recovery system is a good idea? Crashing is just part of the workflow? That's insane, and a good argument against PostgreSQL in any production system. It is possible PostgreSQL doesn't migrate to a thread based model, and I am not arguing they should. But debug and patch the causes of these crashes? Absolut…

It's all about trade off.

Building a database which is never gonna crash might be possible but at what cost? Can you name any single real world system archived that? Also, there can be a regression. More tests? Sure but again, at what cost?

While we are trying to get there, having a crash proof architecture is also a very practical approach.

Re: PostgreSQL reconsiders its process-based model

#303

Post by anarazel »

thesnide wrote: post #297
cmrdporcupine wrote: post #206

It sounds like the specific concerns here are actually around buffer pool management performance in and around the TLB: "Once you have a significant number of connections we end up spending a *lot* of time in TLB misses, and that's inherent to the process model, because you can't share the TLB across processes. " Many of the comments here seem to be missing this and talking about CPU-boundedness generally and thread-…

Exactly my thinking. If the problem is TLB evictions, why not improve these? PG isn't the only software that is hit by those. And if you carefully craft your CPU scheduling and address spaces mapping, you can reduce them by a lot. Yet rewrites are always easier and more sexy. At first.

The TLB issue are more a hardware issue than a software/OS one. To my knowledge neither x86 nor arm provide a way to partially share TLB contents between processes/entities. Tlb entries can be tagged with a process context identifier, but that's an all or nothing thing. Either the entire address space is shared, or it's not.

> Yet rewrites are always easier and more sexy. At first.

Moving to threads would not at all be a rewrite.

Re: PostgreSQL reconsiders its process-based model

#304

Post by mrweasel »

neilwilson wrote: post #296

It's always amazed me with databases why they don't go the other way. Create an operating system specifically for the database and make it so you boot the database. Databases seem to spend most of their time working around the operating system abstractions. So why not look at the OS, and streamline it for database use - dropping all the stuff a database will never need. That then is a completely separate project whic…

You mean like Microsoft SQL Server, which basically runs a small OS on top of Windows or Linux?

This is actually part of the reason why Microsoft was able to port SQL Server to Linux fairly easily, if I recall correctly.

Re: PostgreSQL reconsiders its process-based model

#305

Post by lenkite »

eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

Feel like the PostgreSQL Core Team should just build a new database from scratch using what they have learned from experience instead of attempting such a fundamental architectural migration. It would give them more freedom to change things also. Call it "postgendb" and provide a data migrator.

Re: PostgreSQL reconsiders its process-based model

#306

Post by anarazel »

girvo wrote: post #301
neilwilson wrote: post #296

It's always amazed me with databases why they don't go the other way. Create an operating system specifically for the database and make it so you boot the database. Databases seem to spend most of their time working around the operating system abstractions. So why not look at the OS, and streamline it for database use - dropping all the stuff a database will never need. That then is a completely separate project whic…

That was/is part of the promise of the whole unikernel thing, no? https://mirage.io/ or similar could then let you boot your database. That said, it's not really taken off from what I can tell, so I'm guessing there's more to it than that.

Imo unikernels are a complicated solution in search of a problem, which turns out to not exist.

There certainly are times OSs get in the way. But it's hard enough to write a good database, we don't need to maintain a third of an OS in addition.

Re: PostgreSQL reconsiders its process-based model

#307

Post by dialogbox »

lukeschlather wrote: post #9

This sounds like a problem that would border on the complexity of replacing the GIL in Ruby or Python. The performance benefits are obvious but it seems like the correctness problems would be myriad and a constant source of (unpleasant) surprises.

Even the performance benefits are not big enough compare to the GIL.

Biggest problem of the process model might be the cost of having too many DB connections. Each client need a dedicated server process. Memory usage and the context switching overhead. Or if there is no connection pool, connection time overhead is very high.

This problem has been well addressed with a connection pool. Or having a middle ware instead of exposing the DB directly. That works very well so far.

Oracle has been supporting the thread based model and it's been usable for decades. I remember I tried the thread based configuration option (MTS or shared server) in 1990s. But no one likes that at least within my Oracle DBA network.

It would be a great research project but it would be a big problem if the community pushs this too early.

Re: PostgreSQL reconsiders its process-based model

#308

Post by girvo »

anarazel wrote: post #306
girvo wrote: post #301

Earlier quoted context omitted.

That was/is part of the promise of the whole unikernel thing, no? https://mirage.io/ or similar could then let you boot your database. That said, it's not really taken off from what I can tell, so I'm guessing there's more to it than that.

Imo unikernels are a complicated solution in search of a problem, which turns out to not exist. There certainly are times OSs get in the way. But it's hard enough to write a good database, we don't need to maintain a third of an OS in addition.

Yeah indeed, that was my feeling on it as well. As much as Linux et al might get in ones way at times, what we get for free by relying on them is too useful to ignore for most tasks I think.

That said, perhaps at AWS or Google scale that would be different? I wonder if they've looked at this stuff internally.

Re: PostgreSQL reconsiders its process-based model

#309

Post by wiz21c »

devjab wrote: post #283
pphysch wrote: post #260

Earlier quoted context omitted.

The issue here isn't "rewriting" per se but "stopping development". You shouldn't stop development on your important products. Letting a couple of your talented programmers loose on a greenfield reimplementation is a perfectly sane strategic move. Stopping development on important products because you are 100% certain that the reimplementation will be successful by $DEADLINE is a foolish gamble.

Isn’t that how you end up with Python 2 and 3 though?

Python3 changes were many "little" things; some more fundamental than other (unicode str). So I guess they were able to split the work in tiny pieces and, ultimately, were able to manage the project...

Re: PostgreSQL reconsiders its process-based model

#310

Post by rossmohax »

dialogbox wrote: post #300
neilwilson wrote: post #296

It's always amazed me with databases why they don't go the other way. Create an operating system specifically for the database and make it so you boot the database. Databases seem to spend most of their time working around the operating system abstractions. So why not look at the OS, and streamline it for database use - dropping all the stuff a database will never need. That then is a completely separate project whic…

I'm not sure what you mean by OS. If you mean a whole new kernel, it will take decades. They can support only small number of HW. If you mean a specialized linux distro, many companies does that already. I don't know how that can make it easier the process based / thread based problem.

This project could borrow a lot from unikernels. If they mandate running it as a VM, there is no HW to support.
Post reply on HN