Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

21–30 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#21

Post by paulddraper »

mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

> I'm honestly surprised it took them so long to reach this conclusion.

On the contrary, it's been discussed for ages. But it's a huge change, with only modest advantages.

I'm skeptical of the ROI to be honest. Not that is doesn't have value, but that it has more value than the effort.

Re: PostgreSQL reconsiders its process-based model

#22

Post by mihaic »

loeg wrote: post #7
mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

Yeah. I think as a straightforward, easily correct transition from 2000 globals, a giant structure isn't an awful idea. It's not like the globals were organized before! You're just making the ambient state (awful as it is) explicit.

Exactly, if you're now forced to put everything in one place you're forced to acknowledge and understand the complexity of your state, and might have incentives to simplify it.

Re: PostgreSQL reconsiders its process-based model

#24

Post by chasil »

Oracle has similar problems.

On UNIX systems, Oracle uses a multi-process model, and you can see these:

  $ ps -ef | grep smon

  USER      PID  PPID  STARTED   TIME %CPU %MEM COMMAND
  oracle  22131     1   Mar 28   3:09  0.0  4.0 ora_smon_yourdb
Windows forks processes about 100x slower than Linux, so Oracle runs threaded on that platform in one great big PID.

Sybase was the first major database that fully adopted threads from an architectural perspective, and Microsoft SQL Server has certainly retained and improved on that model.

Re: PostgreSQL reconsiders its process-based model

#26

Post by cactusfrog »

lukeschlather wrote: post #9

This sounds like a problem that would border on the complexity of replacing the GIL in Ruby or Python. The performance benefits are obvious but it seems like the correctness problems would be myriad and a constant source of (unpleasant) surprises.

This is different because there isn’t a whole ecosystem of packages that depend on access to a thread unsafe C API. Getting the GIL out of core Python isn’t too challenging. Getting all of the packages that depend on Python’s C API working is.

Re: PostgreSQL reconsiders its process-based model

#28

Post by lukeschlather »

narrator wrote: post #17
lukeschlather wrote: post #9

This sounds like a problem that would border on the complexity of replacing the GIL in Ruby or Python. The performance benefits are obvious but it seems like the correctness problems would be myriad and a constant source of (unpleasant) surprises.

The correctness problem should be handled by a suite of automated tests which PostgreSQL has. If all tests pass, the application must work correctly. The project is too big, and has too many developers to make much progress without full test coverage. Where else would up-to-date documentation regarding the correct behavior of PostgreSQL exist? In some developers head? SQLite is pretty famous for there extreme approac…

Parallelism is often incredibly hard to write automated tests for, and this will most likely create parallelism issues that were not dreamed of by the authors of the test suite.

Re: PostgreSQL reconsiders its process-based model

#29

Post by shepardrtc »

mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

I think this is a situation where a message-passing Actor-based model would do well. Maybe pass variable updates to a single writer process/thread through channels or a queue.

Years ago I wrote an algorithmic trader in Python (and Cython for the hotspots) using Multiprocessing and I was able to get away with a lot using that approach. I had one process receiving websocket updates from the exchange, another process writing them to an order book that used a custom data structure, and multiple other processes reading from that data structure. Ran well enough that trade decisions could be made in a few thousand nanoseconds on an average EC2 instance. Not sure what their latency requirements are, though I imagine they may need to be faster.

Obviously mutexes are the bottleneck for them at this point, and while my idea might be a bit slower than a low-load situation, perhaps it would be faster when you start getting to higher load.

Re: PostgreSQL reconsiders its process-based model

#30

Post by Sesse__ »

mihaic wrote: post #22
loeg wrote: post #7

Earlier quoted context omitted.

Yeah. I think as a straightforward, easily correct transition from 2000 globals, a giant structure isn't an awful idea. It's not like the globals were organized before! You're just making the ambient state (awful as it is) explicit.

Exactly, if you're now forced to put everything in one place you're forced to acknowledge and understand the complexity of your state, and might have incentives to simplify it.

Here's MySQL's all-session-globals-in-one-place-class: https://github.com/mysql/mysql-server/blob/8.0/sql/sql_class...

I believe I can safely say that nobody acknowledges and understands the complexity of all state within that class, and that whatever incentives there may be to simplify it are not enough for that to actually happen.

(It ends on line 4692)

Post reply on HN