Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

101–110 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#101

Post by Ankhers »

MuffinFlavored wrote: post #34
mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

> if the hierarchy inside them is well organized is this another way to say "in a 2000 member structure, only 10 have significant voting power"?

This statement is not about people, it is about a C struct.

Re: PostgreSQL reconsiders its process-based model

#102

Post by timtom39 »

mynonameaccount wrote: post #90

"the benefits would not justify the cost". PostgreSQL, like any software, at some point in it's life need to be refactored. Why not refactor with a thread model. Of course there will be bugs. Of course it will be difficult. But I think it is a worthwhile endeavor. Doesn't sound like this will happen but a new project would be cool.

> like any software, at some point in it's life need to be refactored.

This is simply not true for most software. Software has a product life cycle like everything else and major refactors/rewrites should be weighed carefully against cost/risk of the refactor. Many traditional engineering fields do much better at this analysis.

Although, because I run a contracting shop, I have personally profited greatly by clients thinking this is true and being unable to convince them otherwise.

Re: PostgreSQL reconsiders its process-based model

#103

Post by smsm42 »

mynonameaccount wrote: post #90

"the benefits would not justify the cost". PostgreSQL, like any software, at some point in it's life need to be refactored. Why not refactor with a thread model. Of course there will be bugs. Of course it will be difficult. But I think it is a worthwhile endeavor. Doesn't sound like this will happen but a new project would be cool.

"Difficult" doesn't even begin to do it justice. Making a code which has 2k global variables and probably order of magnitude as many underlying assumptions (the code should know that now every time you touch X you may be influenced or influence all other threads that may touch X) is a gargantuan task, and will absolutely for sure involve many iterations which any sane person would never let anywhere near valuable data (and how long would it take until you'd consider it safe enough?). And making this all performant - given that shared-state code requires completely different approach to thinking about workload distribution, something that performs when running in isolated processes may very well get bogged down in locking or cache races hell when sharing the state - would be even harder. I am not doubting Postgres has some very smart people - much, much smarter than me, in any case - but I'd say it could be more practical to write new core from scratch than trying to "refactor" the core that organically grew for decades with assumptions of share-nothing model.

Re: PostgreSQL reconsiders its process-based model

#104

Post by djur »

mynonameaccount wrote: post #90

"the benefits would not justify the cost". PostgreSQL, like any software, at some point in it's life need to be refactored. Why not refactor with a thread model. Of course there will be bugs. Of course it will be difficult. But I think it is a worthwhile endeavor. Doesn't sound like this will happen but a new project would be cool.

What you're talking about is a rewrite, not a refactor.

Re: PostgreSQL reconsiders its process-based model

#105

Post by newaccount74 »

A big advantage of the process-based model is its resilience against many classes of errors.

If a bug in PostgreSQL (or in an extension) causes the server to crash, then only that process will crash. Postmaster will detect the child process termination, and send an error message to the client. The connection will be lost, but other connections will be unaffected.

It's not foolproof (there are ways to bring the whole server down), but it does protect against many error conditions.

It is possible to trap on some exceptions in a threaded environment, but cleaning up after eg. an attempted NULL pointer dereference is going to be very difficult or impossible.

Re: PostgreSQL reconsiders its process-based model

#106

Post by akira2501 »

SoylentOrange wrote: post #79
Alekhine wrote: post #50

Earlier quoted context omitted.

I have no idea why that isn't standard practice in every codebase. I should be able to figure out your code without having to ask, or dig through issues or commit messages. Just tell me what it's for!

Because it takes a lot of time and because the comments can get outdated. I also want this for all my code bases. But do I always do this myself? No, especially on green field projects. I will sometimes go back and annotate them later.

Trying to understand what I previously wrote and why I wrote it takes more time than I ever care to spend. I'd much rather have the comments, plus at this point, by making them a "first class" part of my code, I find them much easier to write and I find the narrative style I use incredibly useful in laying out a new structure but also in refactoring old ones.

Re: PostgreSQL reconsiders its process-based model

#107

Post by TeMPOraL »

topspin wrote: post #96
nextaccountic wrote: post #51

Earlier quoted context omitted.

From what I gather postgres isn't doing conventional IPC but instead it uses shared memory, which means the same mechanism threads use but with way higher complexity

As does Oracle, and others. I'm aware. IPC, to me, includes the conventional shared memory resources (memory segments, locks, semaphores, condition variable, etc.) used by these systems: resources acquired by processes for the purpose of communication with other processes. I get it though. The most general concept of shared memory is not coupled to an OS "process." You made me question whether my concept of term IPC…

I always understood IPC, "interprocess communication", in general sense, as anything and everything that can be used by processes to communicate with each other - of course with a narrowing provision that common use of the term refers to those means that are typically used for that purpose, are relatively efficient, and the process in question run on the same machine.

In that view, I always saw shared memory as IPC, in that it is a tool commonly used to exchange data between processes, but of course it is not strictly tied to any process in particular. This is similar to files, which if you squint are a form of IPC too, and are also not tied to any specific process.

> Why does citing ChatGPT make me feel so ugly inside?

That's probably because, in cases like this, it's not much different to stating it yourself, but is more noisy.

Re: PostgreSQL reconsiders its process-based model

#108

Post by FooBarWidget »

mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

I don't get it. How is a 2000-member structure any different from having 2000 global variables? How is maintaining the struct possibly harder than maintaining the globals? Refactoring globals to struct members is semantically nearly identical, it may as well just be a mechanical, cosmetic change, while also giving the possibility to move to a threaded architecture.

Re: PostgreSQL reconsiders its process-based model

#109

Post by phamilton »

comboy wrote: post #48
mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

I've never really been limited by CPU when running postgres (few TB instances). The bottleneck is always IO. Do others have different experience? Plus there's elegance and a feeling of being in control when you know query is associated with specific process which you can deal with and monitor just like any other process. But I'm very much clueless about internals, so this is a question rather than an opinion.

I've generally had buffer-cache hit rates in the 99.9% range, which ends up being minimal read I/O. (This is on AWS Aurora, where these bo disk cache and so shared_buffers is the primary cache, but an equivalent measure for vanilla postgres exists.)

In those scenarios,there's very little read I/O. CPU is the primary bottleneck. That's why we run up as many as 10 Aurora readers (autoscaled with traffic).

Post reply on HN