Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

131–140 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#131

Post by zeroimpl »

rburhum wrote: post #113

Sorry if I offend anybody, but this sounds like such a bad idea. I have been running various versions of postgres in production for 15 years with thousands of processes on super beefy machines, and I can tell you without a doubt that sometimes those processes crash - specially if you are running any of the extensions. Nevertheless, Postgres has 99% of the time proven to be resilient. The idea that a bad client can br…

However, it's already the case that if a postgres process crashes, the whole cluster gets restarted. I've occasionally seen this message:

    WARNING: terminating connection because of crash of another server process
    DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
    HINT: In a moment you should be able to reconnect to the database and repeat your command.
    LOG: all server processes terminated; reinitializing

Re: PostgreSQL reconsiders its process-based model

#132

Post by ComputerGuru »

FooBarWidget wrote: post #108
mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

I don't get it. How is a 2000-member structure any different from having 2000 global variables? How is maintaining the struct possibly harder than maintaining the globals? Refactoring globals to struct members is semantically nearly identical, it may as well just be a mechanical, cosmetic change, while also giving the possibility to move to a threaded architecture.

Because global variables can be confined to individual cpp files, exclusively visible in that compilation unit. It makes them far easier to reason with than hoisting them to the "global and globally visible" option if you just use a gargantuan struct. Which is why a more invasive refactor might be required.

Re: PostgreSQL reconsiders its process-based model

#133

Post by datavirtue »

paulddraper wrote: post #21
mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

> I'm honestly surprised it took them so long to reach this conclusion. On the contrary, it's been discussed for ages. But it's a huge change, with only modest advantages. I'm skeptical of the ROI to be honest. Not that is doesn't have value, but that it has more value than the effort.

Yeah, and you will run headlong into other unforseen real world issues. You may never reach the performance goals.

Re: PostgreSQL reconsiders its process-based model

#134

Post by bb88 »

This reminds me of this poster: "You must be this tall..."

https://bholley.net/blog/2015/must-be-this-tall-to-write-mul...

Back about a decade ago I was "auditing" someone else's threaded code. And couldn't figure it out. But he was the company's "golden child" so by default it must be working code because he wrote it.

And then it started causing deadlocks in prod.

"What do you want me to do about it? It's the golden child's code. He's not even gonna show up til 2pm today."

Re: PostgreSQL reconsiders its process-based model

#135

Post by pasc1878 »

cakoose wrote: post #92
loeg wrote: post #7

Earlier quoted context omitted.

Yeah. I think as a straightforward, easily correct transition from 2000 globals, a giant structure isn't an awful idea. It's not like the globals were organized before! You're just making the ambient state (awful as it is) explicit.

> I think as a straightforward, easily correct transition from 2000 globals, a giant structure isn't an awful idea. Agree. > It's not like the globals were organized before! Using a struct with 2000 fields loses some encapsulation. When a global is defined in a ".c" file (and not exported via a ".h" file), it can only be accessed in that one ".c" file, sort of like a "private" field in a class. Switching to a single…

No that is what a static in a .c file is for.

A plain global can be accessed from other compiled units - agreed with no .h entry it is my=uch more error prone e.g. you don't know the type but the variables name is exposed to other objects

Re: PostgreSQL reconsiders its process-based model

#136

Post by adwn »

ceeam wrote: post #39
adwn wrote: post #27

Earlier quoted context omitted.

Yes, and some of those reasons are even listed in the article.

TLB misses? They are just a detail of particular CPU implementation, and the architectures change. Also, aren't they per core and not per process? What would that solve then to switch to MT?

> TLB misses? They are just a detail of particular CPU implementation, and the architectures change.

TLBs are "just a detail" of roughly 100% of server, desktop, and mobile CPUs.

> Also, aren't they per core and not per process? What would that solve then to switch to MT?

TLB entries are per address space. Threads share an address space, processes do not.

Re: PostgreSQL reconsiders its process-based model

#137

Post by vbezhenar »

wielebny wrote: post #83

Having been using and administering a lot of PostgreSQL servers, I hope they don't lose any stability over this. I've seen (and reported) bugs that caused panics/segfaults in specific psql processes. Not just connections, also processes related to wal writing or replication. The way it's built right now, a child process can be just forced to quit and it does not affect other processes. Hopefully switching into thread…

Of course it will. That's better than continue working with damaged memory structures and unpredictable consequences. For database it's more important than ever. Imagine writing corrupted data because other thread went crazy.

Re: PostgreSQL reconsiders its process-based model

#138

Post by mastax »

It would be interesting to have something between threads and processes. I'll call them heavy-threads for sake of discussion.

Like light-threads, heavy-threads would share the same process-security-boundary and therefore switching between them would be cheap. No need to flush TLB, I$, D$.

Like processes, heavy-threads would have mostly-separate address spaces by default. Similar to forking a process, they could share read-only mappings for shared libraries, code, COW global variables, and explicitly defined shared writable memory regions.

Like processes, heavy-threads would isolate failure states. A C++ exception, UNIX signal, segfault, etc. would kill only the heavy-thread responsible.

Re: PostgreSQL reconsiders its process-based model

#139

Post by niccl »

zeroimpl wrote: post #131
rburhum wrote: post #113

Sorry if I offend anybody, but this sounds like such a bad idea. I have been running various versions of postgres in production for 15 years with thousands of processes on super beefy machines, and I can tell you without a doubt that sometimes those processes crash - specially if you are running any of the extensions. Nevertheless, Postgres has 99% of the time proven to be resilient. The idea that a bad client can br…

However, it's already the case that if a postgres process crashes, the whole cluster gets restarted. I've occasionally seen this message: WARNING: terminating connection because of crash of another server process DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. HINT: In a momen…

yes, but postmaster is still running to roll back the transaction. If you crash a single multi-threaded process, you may lose postmaster as well and then sadness would ensue

Re: PostgreSQL reconsiders its process-based model

#140

Post by avgcorrection »

hnarn wrote: post #119
idiomaticrust wrote: post #115

Earlier quoted context omitted.

He is right. Such rewrites cause a lot of problems if your compiler doesn't help you with avoiding data races. But there is another way.

> But there is another way. Ok?

Don’t mind the gimmick gallery (username).
Post reply on HN