Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL reconsiders its process-based model

lwn.net

Post reply on HN
Print view View original

201–210 of 377 posts

Re: PostgreSQL reconsiders its process-based model

#201

Post by rsaxvc »

mihaic wrote: post #2

I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…

This is how I made my fork of libtcc lock-free.

Mainline has a lock so that all backends can use global variables, but only one instance can do codegen at a time.

It was a giant refactoring Especially fun was when multiple compilation units used the same static variable name, but it all worked in the end.

Re: PostgreSQL reconsiders its process-based model

#202

Post by nologic01 »

Alekhine wrote: post #50
BoardsOfCanada wrote: post #20

I recently looked through the source code of postgresql and every source files starts with a (really good) description of what the file is supposed to do, which made it really easy to get in to the code compared to other open source projects I've seen. So thanks for that.

I have no idea why that isn't standard practice in every codebase. I should be able to figure out your code without having to ask, or dig through issues or commit messages. Just tell me what it's for!

the average programmer thinks they are writting significantly above average clean code, so no need to document it :-)

Re: PostgreSQL reconsiders its process-based model

#203

Post by dfox »

ilyt wrote: post #93
comboy wrote: post #48

Earlier quoted context omitted.

I've never really been limited by CPU when running postgres (few TB instances). The bottleneck is always IO. Do others have different experience? Plus there's elegance and a feeling of being in control when you know query is associated with specific process which you can deal with and monitor just like any other process. But I'm very much clueless about internals, so this is a question rather than an opinion.

>I've never really been limited by CPU when running postgres (few TB instances). The bottleneck is always IO. Throw a few NVMe drives at it and it might.

Throw a ridiculous amount of RAM at it is more correct assessment. NVMe reads are still an “I/O” and that is slow. And for at least 10 years buying enough RAM to have all off the interesting parts of OLTP psql database either in shared_buffers or in the OS-level buffer cache is completely feasible.

Re: PostgreSQL reconsiders its process-based model

#204

Post by timcobb »

BoardsOfCanada wrote: post #85
levkk wrote: post #68

Pretty sure Tom Lane said this will be a disaster in that same pgsql-hackers thread. Not entirely sure what benefits the multi-threaded model will have when you can easily saturate the entire CPU with just 128 connections and a pooler. So I doubt there is consensus or even strong desire from the community to undertake this boil the ocean project. On the other hand, having the ability to shut down and cleanup the enti…

From the article: > Tom Lane said: "I think this will be a disaster. There is far too much code that will get broken". He added later that the cost of this change would be "enormous", it would create "more than one security-grade bug", and that the benefits would not justify the cost.

You can think of this as an opportunity to rewrite in Rust.

Re: PostgreSQL reconsiders its process-based model

#205

Post by wizofaus »

SigmundA wrote: post #190
wizofaus wrote: post #185

Earlier quoted context omitted.

Not claiming they're as good, just noting that there are alternative ways to provide memory barriers, though obviously if it's not enforced at the language/runtime level, it requires either super strong developer disciple or the use of some other tool to do so. I can't find anything suggesting AppDomains have been removed completely though, just they're not fully supported on non-Windows platforms, which is interesti…

https://learn.microsoft.com/en-us/dotnet/api/system.appdomai... "On .NET Core, the AppDomain implementation is limited by design and does not provide isolation, unloading, or security boundaries. For .NET Core, there is exactly one AppDomain. Isolation and unloading are provided through AssemblyLoadContext. Security boundaries should be provided by process boundaries and appropriate remoting techniques." AppDomains p…

Is that saying global variables are shared between AppDomains on .NET core then? Scary if so, we have a bunch of .NET framework code we're looking at porting to .NET core in the near future, and I know it relies on AppDomain separation currently. It's not the first framework->Core conversation I've done, but I don't remember changes in AppDomain behaviour causing any issues the first time.

As it happens I already know there are bits of code currently not working "as expected" exactly because of AppDomain separation - i.e. attempting to use a shared-memory cache to improve performance and in one or two cases in an attempt to share state, and I got the impression whoever wrote that code didn't understand that there even were two AppDomains involved, and used various ugly hacks to "fall back" to alternative means of state-sharing, but in fact the fall-back is the only thing that actually ever works.

Re: PostgreSQL reconsiders its process-based model

#206

Post by cmrdporcupine »

It sounds like the specific concerns here are actually around buffer pool management performance in and around the TLB: "Once you have a significant number of connections we end up spending a *lot* of time in TLB misses, and that's inherent to the process model, because you can't share the TLB across processes. "

Many of the comments here seem to be missing this and talking about CPU-boundedness generally and thread-per-request vs process etc models, but this seems orthogonal to that, and is actually quite specific about the VM subsystem and seems like a legitimate bottleneck with the approach Postgres has to take for buffer/page mgmt with the process model it has now.

I'm no Postgres hacker (or a Linux kernel hacker), and I only did a 6 month stint doing DB internals, but it feels to me like perhaps the right answer here is that instead of Postgres getting deep down in the weeds refactoring and rewriting to a thread based model -- with all the risks in that that people have pointed out -- some assistance could be reached for by working on specific targeted patches in the Linux kernel?

The addition of e.g. userfaultfd shows that there is room for innovation and acceptance of changes in and around kernel re: page management. Some new flags for mmap, shm_open, etc. to handle some specific targeted use cases to help Postgres out?

Also wouldn't be the first time that people have done custom kernel patches or tuning parameters to crank performance out of a database.

Re: PostgreSQL reconsiders its process-based model

#207

Post by fdr »

dfox wrote: post #199
cogman10 wrote: post #176

Earlier quoted context omitted.

I think my bigger fear is around security. A process per connection keeps things pretty secure for that connection regardless of what the global variables are doing (somewhat hard to mess that up with no concurrency going on in a process). Merge all that into one process with many threads and it becomes a nightmare problem to ensure some random addon didn't decide to change a global var mid processing (which causes w…

All postgres processes run under the same system user and all the access checking happens completely in userspace.

Access checking, yes, but the scope of memory corruption does increase unavoidably, given the main thing the pgsql-hackers investigating threads want: one virtual memory context when toggling between concurrent work.

Of course, there's a huge amount of shared space already, so a willful corruption can already do virtually anything. But, more is more.

Re: PostgreSQL reconsiders its process-based model

#210

Post by gjvc »

Icathian wrote: post #209
gjvc wrote: post #198

surely this is "Some guy reconsiders the process-based model of PostgreSQL"

Uh. Heikki is definitely not just "some guy". Dude is one of the top contributors to Postgres.

How does that make him immune to having dumb ideas? See, I'm judging the idea on merit.

You're just defending your hero who has gone rogue.

Post reply on HN