It's always amazed me with databases why they don't go the other way. Create an operating system specifically for the database and make it so you boot the database. Databases seem to spend most of their time working around the operating system abstractions. So why not look at the OS, and streamline it for database use - dropping all the stuff a database will never need. That then is a completely separate project whic…
PostgreSQL reconsiders its process-based model
331–340 of 377 posts
Re: PostgreSQL reconsiders its process-based model
#332With the process model, it encourages using separate memory for each process; this forces developers to think about things like memory consistency and availability and gives them more flexibility in terms of scalability across multiple CPU cores or even hosts. Processes are far better abstractions than threads for modeling concurrent systems since their logic is fundamentally the same regardless of whether they run across different CPU cores or different hosts.
> The overhead of cross-process context switches is inherently higher than switching between threads in the same process
I remember researching this a while back. It depends on the specific OS and hardware. It's not so straight forward and this is something which tends to change over time and the differences are usually insignificant anyway.
Also, it's important not to conflate performance with scalability - These two characteristics are orthogonal at best and oftentimes conflicting.
Oftentimes, to scale horizontally, a system needs to incur a performance penalty as additional work is required to route and coordinate actions across multiple CPUs or hosts. A scalable system can service a much larger number (or even sometimes theoretically unlimited) number of requests but it will typically perform worse than a non-scalable system if you judge it on a requests-per-CPU-core basis.
Re: PostgreSQL reconsiders its process-based model
#333Earlier quoted context omitted.
I don't expect you or others to buy into any particular code change at this point, or to contribute time into it. Just to accept that it's a worthwhile goal. If the implementation turns out to be a disaster, then it won't be accepted, of course. But I'm optimistic. The reply is much more reasonable than this blanket assertion of a disaster.
As an outsider it doesn't sound like something a few people could spin off in a branch in a couple months and see how code review goes. They're talking about doing it over multiple (yearly?) releases. It seems like it'll take a lot of expert attention, which won't be available for other work and the changes themselves will impact all other ongoing work. I'm not trying to naysay it per se, bc again I don't have techni…
They could fork, they could add threading to some sub systems and roll it out over several versions.
I don't know enough about the code but, of course, it is a hard problem but the solution might be to build it from the ground up as a threaded system, using the skills learned over 30 years and taking the hit on the rebuild instead of reworking what is there.
I am most interested because I didn't realise there was a performance problem in the first place.
Re: PostgreSQL reconsiders its process-based model
#334Earlier quoted context omitted.
How does that make him immune to having dumb ideas? See, I'm judging the idea on merit. You're just defending your hero who has gone rogue.
Heikki is far from the only "senior" postgres contributor thinking that this is the right direction in the long term. > You're just defending your hero who has gone rogue. That doesn't sound like judging an idea on its merit, it sounds like judging a person for something you haven't analyzed yourself.
sounds like groupthink
Re: PostgreSQL reconsiders its process-based model
#335For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.
Yeah. Without being familiar with the Postgres source, this seems to be what I call a "somersault problem": hard to break down into sub-goals. I have heard that the Postgres codebase is solid which makes it easier but it's still mature and highly complex. It doesn't sound feasible to me. [link redacted]
Re: PostgreSQL reconsiders its process-based model
#336Earlier quoted context omitted.
Reminds me of PHP 6... For those who don't follow PHP closely - that version was an attempted refactor of the string implementation which essentially shut down nearly all work on PHP for a decade, stagnating the language until it became pretty terrible compared to other options. They finally gave up and started work on PHP 7 which uses the (perfectly good) PHP 5 strings. Ten years of wasted time by the best internal…
Things You Should Never Do https://www.joelonsoftware.com/2000/04/06/things-you-should-... An oldie but a goodie
Re: PostgreSQL reconsiders its process-based model
#337Earlier quoted context omitted.
What makes you think that it will require that many changes? There will be some widespread mechanical changes (which can be verified to be complete with a bit of low level work, like a script using objdump/nm to look for non-TLS mutable variables) and some areas changing more heavily (e.g. connection establishment, crash detection, signal handling, minor details of the locking code). But large portions of the code wo…
I'm not the person you asked and I don't have any particular knowledge of postgres internals. Experience with other systems has taught me that in a system that's been in active use and development for decades, entanglement will be deep, subtle, and pervasive. If this isn't true of postgres then it's an absolute freak anomaly of a codebase. It is that in other ways, so it's possible. But the article mentions there bei…
The 'that many changes' in question is a complete rewrite. Many changes across many files, yes, but nothing even approaching a rewrite.
> I don't have any particular knowledge of postgres
Judging by their bio, the person you're replying to does.
Re: PostgreSQL reconsiders its process-based model
#338Earlier quoted context omitted.
The reasons are explained in article. Read the article
I appear to have missed them, then. Could you point out, aside from the large numbers of clients I mentioned (and the development overhead of implementing multi-process memory management code), what the article mentions is a primary drawback of using processes over threads?
Re: PostgreSQL reconsiders its process-based model
#339For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.
Re: PostgreSQL reconsiders its process-based model
#340Why should TLB flush performance ever be a problem on big machines? You can have one process per core with 128 or more cores, never flush any TLB if you pin those processes. And as it is a database, shoveling data from/to disk/SSD is your main concern anyways.
That said, I think there have been efforts to use io_uring on Linux. I'm not sure how that would work with the process per connection model. Haven't been following it...