Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

121–130 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#121

Post by usefulcat »

johannes1234321 wrote: post #63
baggy_trough wrote: post #8

I hope they are conservative about this, because even the smartest and best programmers in the world cannot create bug free multithreaded code.

The code already is multithreaded. They have shared state just across multiple processes instead of threads within a process. They might even reduce complexity that way.

It's not the same at all for global variables, of which pgsql apparently has around a couple thousand.

If every process is single threaded, you don't have to consider the possibility of race conditions when accessing any of those ~2000 global variables. And you can pretty much guarantee that little if any of the existing code was written with that possibility in mind.

Re: PostgreSQL reconsiders its process-based model

#122

Post by blinkingled »

chasil wrote: post #24

Oracle has similar problems. On UNIX systems, Oracle uses a multi-process model, and you can see these: $ ps -ef | grep smon USER PID PPID STARTED TIME %CPU %MEM COMMAND oracle 22131 1 Mar 28 3:09 0.0 4.0 ora_smon_yourdb Windows forks processes about 100x slower than Linux, so Oracle runs threaded on that platform in one great big PID. Sybase was the first major database that fully adopted threads from an architectur…

Didn't Oracle switch to threaded model in 12c - at least on Linux I remember there being a parameter to do that - it dropped the number of processes significantly.

Re: PostgreSQL reconsiders its process-based model

#123

Post by mycall »

hnarn wrote: post #119
idiomaticrust wrote: post #115

Earlier quoted context omitted.

He is right. Such rewrites cause a lot of problems if your compiler doesn't help you with avoiding data races. But there is another way.

> But there is another way. Ok?

Microsoft SQL Server has SQLOS which is another way [0].

[0] https://www.thegeekdiary.com/what-is-sql-server-operating-sy...

Re: PostgreSQL reconsiders its process-based model

#124

Post by gremlinsinc »

eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

Maybe a better option would be finding a team to create nugres, aka a fork for this and other experiments. So that mainline remains stable.

Re: PostgreSQL reconsiders its process-based model

#125

Post by ComputerGuru »

masklinn wrote: post #32
cactusfrog wrote: post #26

Earlier quoted context omitted.

This is different because there isn’t a whole ecosystem of packages that depend on access to a thread unsafe C API. Getting the GIL out of core Python isn’t too challenging. Getting all of the packages that depend on Python’s C API working is.

An other component of the Gil story is that removing the Gil require adding fine grained locks, which (aside from making VM development more complicated) significantly increases lock traffic and thus runtime costs, which noticeably impacts single-threaded performance, which is of major import. Postgres starts from a share-nothing architecture, it’s quite a bit easier to evaluate the addition of sharing.

> which (aside from making VM development more complicated) significantly increases lock traffic and thus runtime costs, which noticeably impacts single-threaded performance, which is of major import.

I don't think that's a fair characterization of the trade offs. Acquiring uncontended mutexes is basically free (and fairly side-effect free) so single-threaded performance will not be noticeably impacted.

Every large C project I'm aware of (read: kernels) that has publicly switched from coarse locks to fine-grained locks has considered it to be a huge win with little to no impact on single-threaded performance. You can even gain performance if you chop up objects or allocations into finer-grained blobs to fit your finer-grained locking strategy because it can play nicer with cache friendliness (accessing one bit of code doesn't kick the other bits of code out of the cache).

Re: PostgreSQL reconsiders its process-based model

#126

Post by carstenhag »

hnarn wrote: post #119
idiomaticrust wrote: post #115

Earlier quoted context omitted.

He is right. Such rewrites cause a lot of problems if your compiler doesn't help you with avoiding data races. But there is another way.

> But there is another way. Ok?

The person probably implied that Postgres should switch to another toolchain that guarantees more things at compile time, so probably Rust.

Re: PostgreSQL reconsiders its process-based model

#127

Post by chc »

hnarn wrote: post #119
idiomaticrust wrote: post #115

Earlier quoted context omitted.

He is right. Such rewrites cause a lot of problems if your compiler doesn't help you with avoiding data races. But there is another way.

> But there is another way. Ok?

I think it's meant to imply the solution given in their username ("idiomatic Rust").

Re: PostgreSQL reconsiders its process-based model

#128

Post by datavirtue »

eclipticplane wrote: post #72

For the record, I think this will be a disaster. There is far too much code that will get broken, largely silently, and much of it is not under our control. regards, tom lane (via https://lwn.net/ml/pgsql-hackers/4178104.1685978307@sss.pgh.... ) If Tom Lane says it will be a disaster, I believe it will be a disaster.

This should be considered a research effort, assuming it will be a complete rewrite. In light of that, you should not draw down resources from the established code base to work on it.

Ignoring the above, first state the explicit requirements driving this change and let people weigh in on those. This sounds like a geeky dev itch.

Re: PostgreSQL reconsiders its process-based model

#129

Post by gremlinsinc »

mynonameaccount wrote: post #90

"the benefits would not justify the cost". PostgreSQL, like any software, at some point in it's life need to be refactored. Why not refactor with a thread model. Of course there will be bugs. Of course it will be difficult. But I think it is a worthwhile endeavor. Doesn't sound like this will happen but a new project would be cool.

a better option would just create an experimental fork that has a different name and is obviously a different product but based on the original source. That way pg gets updates and remains stable and if they fail, they fail and it doesn't hurt all the pg in production.

Re: PostgreSQL reconsiders its process-based model

#130

Post by treis »

devit wrote: post #42
EvanAnderson wrote: post #40

Earlier quoted context omitted.

> Windows forks processes about 100x slower than Linux... I work with a Windows-based COTS webapp that uses Postgres w/o any connection pooling. It's nearly excruciating to use because it spins-up new Postgres processes for each page load. If not for the fact that the Postgres install is "turnkey" with the app I'd just move Postgres over to a Linux machine.

Use pgbouncer

That helps a lot but it's not a replacement for large number of persistent connections. If you had that you could simplify things in the application layer and do interesting things with the DB.
Post reply on HN