PostgreSQL reconsiders its process-based model • Hacker News Board

Re: PostgreSQL reconsiders its process-based model

#271

Post by paulddraper » Tue, Jun 20, 2023, 4:32 AM UTC

pphysch wrote: post #260

SoftTalker wrote: post #241

Earlier quoted context omitted.

Things You Should Never Do https://www.joelonsoftware.com/2000/04/06/things-you-should-... An oldie but a goodie

The issue here isn't "rewriting" per se but "stopping development". You shouldn't stop development on your important products. Letting a couple of your talented programmers loose on a greenfield reimplementation is a perfectly sane strategic move. Stopping development on important products because you are 100% certain that the reimplementation will be successful by $DEADLINE is a foolish gamble.

The problem is if it snowballs

Re: PostgreSQL reconsiders its process-based model

#272

Post by j16sdiz » Tue, Jun 20, 2023, 4:33 AM UTC

mattashii wrote: post #223

gremlinsinc wrote: post #124

Earlier quoted context omitted.

Maybe a better option would be finding a team to create nugres, aka a fork for this and other experiments. So that mainline remains stable.

There are several forks of PostgreSQL, in various levels of license, additional features and activity. However, maintaining a fork in addition to a main project is inherently more expensive than maintaining just a single project, so adding features to new major releases of the main project is generally preferred over forking every release into its own, newly named, project. After all, that is what we have major (feat…

This won't work well for a multiyear project.. Either you have to stall the release process, divide it into smaller parts or fork.

Re: PostgreSQL reconsiders its process-based model

#273

Post by paulddraper » Tue, Jun 20, 2023, 4:34 AM UTC

Shorel wrote: post #265

rburhum wrote: post #113

Sorry if I offend anybody, but this sounds like such a bad idea. I have been running various versions of postgres in production for 15 years with thousands of processes on super beefy machines, and I can tell you without a doubt that sometimes those processes crash - specially if you are running any of the extensions. Nevertheless, Postgres has 99% of the time proven to be resilient. The idea that a bad client can br…

Reading your comment makes me think it is not only a good idea, it is a necessity. Relying on crashing as a bug recovery system is a good idea? Crashing is just part of the workflow? That's insane, and a good argument against PostgreSQL in any production system. It is possible PostgreSQL doesn't migrate to a thread based model, and I am not arguing they should. But debug and patch the causes of these crashes? Absolut…

Cars are designed with airbags?!

Like, they are supposed to crash?!?

Re: PostgreSQL reconsiders its process-based model

#274

Post by paulddraper » Tue, Jun 20, 2023, 4:38 AM UTC

wmf wrote: post #191

bb88 wrote: post #134

This reminds me of this poster: "You must be this tall..." https://bholley.net/blog/2015/must-be-this-tall-to-write-mul... Back about a decade ago I was "auditing" someone else's threaded code. And couldn't figure it out. But he was the company's "golden child" so by default it must be working code because he wrote it. And then it started causing deadlocks in prod. "What do you want me to do about it? It's the golden…

The thing is... multi-process with a bespoke shared memory system isn't better than multithreading; it's much worse.

It kinda is though. The process barrier is better at enforcing careful deliberate interactions

Re: PostgreSQL reconsiders its process-based model

#275

Post by anarazel » Tue, Jun 20, 2023, 4:38 AM UTC

Shorel wrote: post #265

rburhum wrote: post #113

Sorry if I offend anybody, but this sounds like such a bad idea. I have been running various versions of postgres in production for 15 years with thousands of processes on super beefy machines, and I can tell you without a doubt that sometimes those processes crash - specially if you are running any of the extensions. Nevertheless, Postgres has 99% of the time proven to be resilient. The idea that a bad client can br…

Reading your comment makes me think it is not only a good idea, it is a necessity. Relying on crashing as a bug recovery system is a good idea? Crashing is just part of the workflow? That's insane, and a good argument against PostgreSQL in any production system. It is possible PostgreSQL doesn't migrate to a thread based model, and I am not arguing they should. But debug and patch the causes of these crashes? Absolut…

We do fix crashes etc, even if the postgres manages to restart.

I think the post upthread references an out-of-core extension we don't control, which in turn depends on many external libraries it doesn't control either...

Re: PostgreSQL reconsiders its process-based model

#276

Post by anarazel » Tue, Jun 20, 2023, 4:44 AM UTC

yxre wrote: post #267

h0l0cube wrote: post #263

Earlier quoted context omitted.

> Well, yes. They did. They did it by making the single worst strategic mistake that any software company can make: > They decided to rewrite the code from scratch. Absolutely not what's being proposed for Postgres.

Process isolation affects so many things in C. The strategy change is going to require changes to so many modules that it will either be a re-write or buggy. In practical terms, if every line needs to be audited and updated, it is a re-write

What makes you think that it will require that many changes? There will be some widespread mechanical changes (which can be verified to be complete with a bit of low level work, like a script using objdump/nm to look for non-TLS mutable variables) and some areas changing more heavily (e.g. connection establishment, crash detection, signal handling, minor details of the locking code). But large portions of the code won't need to change. Note that we/postgres already shares a lot of state across processes.

Re: PostgreSQL reconsiders its process-based model

#277

Post by jpgvm » Tue, Jun 20, 2023, 4:44 AM UTC

anarazel wrote: post #258

jpgvm wrote: post #256

A close to impossible task, if anyone can do it's probably Heikki though. Unfortunately I expect this to go the way of zheap et al. Fundamental design changes like this have just had such a rough time of succeeding thus far. I think for such a change to work it probably needs not just the support of Neon but also of say Microsoft (current stewards of Citus) that have larger engineering resources to throw at the probl…

I know that at least people from EDB (Robert) and Microsoft (Thomas, me) are quite interested in eventually making this transition, it's not just Heikki. Personally I won't have a lot of cycles for the next release or two, but after that...

That gives me some faith, I hope everyone is able to come together to make it happen.

Re: PostgreSQL reconsiders its process-based model

#278

Post by anarazel » Tue, Jun 20, 2023, 4:47 AM UTC

rashidujang wrote: post #266

jpgvm wrote: post #256

A close to impossible task, if anyone can do it's probably Heikki though. Unfortunately I expect this to go the way of zheap et al. Fundamental design changes like this have just had such a rough time of succeeding thus far. I think for such a change to work it probably needs not just the support of Neon but also of say Microsoft (current stewards of Citus) that have larger engineering resources to throw at the probl…

Hey I'm fairly new to the who's who in the PostgreSQL world, would you mind telling why Heikki might be able to pull this off?

Not who you asked, but: He is a longtime contributor who has written/resigned important parts of postgres (WAL format, concurrent WAL insertion, 2PC support, parts of SSI support, much more). And he is just a nice person to work with.

Re: PostgreSQL reconsiders its process-based model

#279

Post by rtpg » Tue, Jun 20, 2023, 4:47 AM UTC

Shorel wrote: post #265

rburhum wrote: post #113

Sorry if I offend anybody, but this sounds like such a bad idea. I have been running various versions of postgres in production for 15 years with thousands of processes on super beefy machines, and I can tell you without a doubt that sometimes those processes crash - specially if you are running any of the extensions. Nevertheless, Postgres has 99% of the time proven to be resilient. The idea that a bad client can br…

Reading your comment makes me think it is not only a good idea, it is a necessity. Relying on crashing as a bug recovery system is a good idea? Crashing is just part of the workflow? That's insane, and a good argument against PostgreSQL in any production system. It is possible PostgreSQL doesn't migrate to a thread based model, and I am not arguing they should. But debug and patch the causes of these crashes? Absolut…

We don't want stuff to crash. But we also want data integrity to be maintained. We also want things to work. In a world with extensions written in C to support a lot of cool things with Postgres, you want to walk and chew bubblegum on this front.

Though to your point, a C extension can totally destroy your data in other ways, and there are likely ways to add more barriers. And hey, we should fix bugs!

Re: PostgreSQL reconsiders its process-based model

#280

Post by Hayvok » Tue, Jun 20, 2023, 4:52 AM UTC

progmetaldev wrote: post #231

abhibeckert wrote: post #224

Earlier quoted context omitted.

Reminds me of PHP 6... For those who don't follow PHP closely - that version was an attempted refactor of the string implementation which essentially shut down nearly all work on PHP for a decade, stagnating the language until it became pretty terrible compared to other options. They finally gave up and started work on PHP 7 which uses the (perfectly good) PHP 5 strings. Ten years of wasted time by the best internal…

I've used PHP in the past (PHP 4 and 5), as well as some simple templated projects in PHP 7. I try to keep up on news with what is happening in the PHP world, and it's difficult because of the hate for the language. Is the solution to Unicode strings still to just use the "mb_*" functions? I got my real professional start using PHP, and have built even financial systems in the language (since ported to .NET 6 for my…

I believe so - most (all?) string functions have an mb_ equivalent, for working on multibyte strings.

Regular PHP strings are actually pretty great, since you can treat them like byte arrays. Fun fact: PHPs streaming API has an “in-memory” option and it’s… just a string under the hood.

Just don’t forget to use multibyte functions when you’re handling things like user input.