361–370 of 377 posts
"no objections" "consent"
Have you ever tried to move a large organization forward in a certain direction? It’s really hard. At some point you have to make a decision.
Earlier quoted context omitted.
I don't understand how it works with thread per connection either. io_uring is designed for systems that have a thread and ring per core, for you to give it a bunch of IO to do at once (batches and chains), and your threads to do other work in the meantime. The syscall cost is amortized or even (through IORING_SETUP_SQPOLL) eliminated. If your code is instead designed to be synchronous and thus can only do one IO at…
> io_uring is designed for systems that have a thread and ring per core That's not needed to benefit from io_uring > for you to give it a bunch of IO to do at once (batches and chains), and your threads to do other work in the meantime. You can see substantial gains even if you just submit multiple IOs at once, and then block waiting for any of them to complete. The cost of blocking on IO is amortized to some degree…
> > io_uring is designed for systems that have a thread and ring per core
> That's not needed to benefit from io_uring
90% sure I read Axboe saying that's what he designed io_uring for. If it helps in other scenarios, though, great.
> Of course it's even better to not block at all...
Out of curiosity, is that something you ever want/hope to achieve in PostgreSQL? Many high-performance systems use this model, but switching a synchronous system in plain C to it sounds uncomfortably exciting, both in terms of the transition itself and the additional complexity of maintaining the result. To me it seems like a much riskier change than the process->thread one discussed here that Tom Lane already stated will be a disaster.
> We/I have done the work to issue multiple IOs at a time as part of the patchset introducing AIO support (with among others, an io_uring backend). There's definitely more to do, particularly around index scans, but ...
Nice.
Is the benefit you're getting simply from adding IO parallelism where there was none, or is there also a CPU reduction?
Is having a large number of rings (as when supporting a large number of incoming connections) practical? I'm thinking of each ring being a significant reserved block of RAM, but maybe in this scenario that's not really true. A smallish ring for a smallish number of IOs for the query is enough.
Speaking of large number of incoming connections, would/could the process->thread change be a step toward having a thread per active query rather than per (potentially idle) connection? To me it seems like it could be: all the idle ones could just be watched over by one thread and queries dispatched. That'd be a nice operational improvement if it meant folks no longer needed a pooler [1] to get decent performance. All else being equal, fewer moving parts is more pleasant...
[1] or even if they only needed one layer of pooler instead of two, as I read some people have!
Earlier quoted context omitted.
Right but that would still be true if they were globals instead. Putting all the globals in a class doesn't make any difference to how much state you have.
> Putting all the globals in a class doesn't make any difference to how much state you have. I didn't make any claims about the _amount_ of state. My claim was that “you're forced to acknowledge and understand the complexity of your state” (i.e., moving it all together in one place helps understanding the state) is plain-out wrong.
Earlier quoted context omitted.
> io_uring is designed for systems that have a thread and ring per core That's not needed to benefit from io_uring > for you to give it a bunch of IO to do at once (batches and chains), and your threads to do other work in the meantime. You can see substantial gains even if you just submit multiple IOs at once, and then block waiting for any of them to complete. The cost of blocking on IO is amortized to some degree…
Oh, I hadn't realized until now I was talking with someone actually doing this work. Thanks for popping into this discussion! > > io_uring is designed for systems that have a thread and ring per core > That's not needed to benefit from io_uring 90% sure I read Axboe saying that's what he designed io_uring for. If it helps in other scenarios, though, great. > Of course it's even better to not block at all... Out of cu…
> Out of curiosity, is that something you ever want/hope to achieve in PostgreSQL? Many high-performance systems use this model, but switching a synchronous system in plain C to it sounds uncomfortably exciting, both in terms of the delta and the additional complexity of maintaining the result. To me it seems like a much riskier change than the process->thread one discussed here that Tom Lane already stated will be a disaster.
Depends on how you define it. In a lot of scenarios you can avoid blocking by scheduling IO in a smart way - and I think we can quite far towards that for a lot of workloads and the wins are substantial. But that obviously cannot alone guarantee that you never block.
I think we can get quite far avoiding blocking, but I don't think we're going to a complete asynchronous model in the foreseeable future. But it seems more feasible to incrementally make common blocking locations support asynchronicity. E.g. when a query scans multiple partitions, switch to processing a different partition while waiting for IO.
> Is having a large number of rings (as when supporting a large number of incoming connections) practical? I'm thinking of each ring being a significant reserved block of RAM, but maybe in this scenario that's not really true. A smallish ring for a smallish number of IOs for the query is enough.
It depends on the kernel version etc. The amount of memory isn't huge but initially it was affected by RLIMIT_MEMLOCK... That's one reason why the AIO patchset has a smaller number of io_uring "instances" than the allowed connections. The other reason is that we need to be able to complete IOs that other backends started (otherwise there would be deadlocks), which in turn requires having the file descriptor for each ring available in all processes... Which wouldn't be fun with a high max_connections.
> Speaking of large number of incoming connections, would/could the process->thread be a step toward having a thread per active query rather than per (potentially idle) connection?
Yes. Moving to threads really mainly would be to make subsequent improvements more realistic...
> That'd be a nice operational improvement if it meant folks no longer needed a pooler [1] to get decent performance. All else being equal, fewer moving parts is more pleasant...
You'd likely often still want a pooler on the "application server" side, to avoid TCP / SSL connection establishment overhead. But that can be a quite simple implementation.
I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…
> I'm honestly surprised it took them so long to reach this conclusion. I'm not. You can get a long way with conventional IPC, and OS processes provide a lot of value. For most PostgreSQL instances the TLB flush penalty is at least 3rd or 4th on the list of performance concerns, far below prevailing storage and network bottlenecks. I share the concerns cited in this LWN story. Reworking this massive code base around…
Postmaster would just share the already shared memory between processes (containing also the locks). That explicit part of memory would opt-in to thread -like sharing and thus get faster/less tlb switching and lower memory usage. While all the rest of the state would still be per-process and safe.
tl;dr super share the existing shared memory area with kernel patch
All operating systems not supporting it would keep working as is.
Earlier quoted context omitted.
> I'm honestly surprised it took them so long to reach this conclusion. I'm not. You can get a long way with conventional IPC, and OS processes provide a lot of value. For most PostgreSQL instances the TLB flush penalty is at least 3rd or 4th on the list of performance concerns, far below prevailing storage and network bottlenecks. I share the concerns cited in this LWN story. Reworking this massive code base around…
Would something like opt-in sharing of pages between processes that oracle has been trying to get into kernel be the correct option: https://lwn.net/ml/linux-kernel/cover.1682453344.git.khalid.... Postmaster would just share the already shared memory between processes (containing also the locks). That explicit part of memory would opt-in to thread -like sharing and thus get faster/less tlb switching and lower memory…
Earlier quoted context omitted.
You're implying that only an OS can provide memory separation between units of execution - at least in .NET AppDomains give you the same protection within a single process, so why couldn't postgres have its own such mechanism? I'd also think with a database engine shared state is not just in-memory - i.e. one process can potentially corrupt the behaviour of another by what it writes to disk, so moving to a single-pro…
.NET is a managed-language with a VM. In such language, a memory error in managed-code will often trigger a jump back to the VM, where they can attempt to recover from there. For native code, there's no such safety net. Likewise, even for managed language, an error in the interpreter code will still crash the VM, since there's nothing to fallback to anymore.
I'm honestly surprised it took them so long to reach this conclusion. > That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way. As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no…
I've never really been limited by CPU when running postgres (few TB instances). The bottleneck is always IO. Do others have different experience? Plus there's elegance and a feeling of being in control when you know query is associated with specific process which you can deal with and monitor just like any other process. But I'm very much clueless about internals, so this is a question rather than an opinion.
Not sure what's going on here, but one connection per process seems... ancient. Using threaded model is difficult, how about pre-fork? Some connections per process is a good improvement.
The issue is costlier (runtime, complexity, memory) resource sharing, not the cost of the fork itself. Pre-forking isn't going to help with any of that.
Earlier quoted context omitted.
I think it’s very dependent on how you use words like “rewrite” or “refactor”. The point the author makes about the two page function, and all the bug-fixes (lessons learned) makes sense only if you “rewrite” from scratch without looking at the history. You can absolutely “rewrite” the function in a manner that is “refactoring”, but will often get called “rewrite” in the real world. This may be because “refactor” is…
There's currently a trend towards shitting on microservices-everything, imo largely justified. But missing from that is that identifying a logical feature and moving it to a microservice is one of the safer ways to begin a gradual rewrite of a critical system. And usually possible to get cross-dept buyin for various not always wholesome reasons. It may not always be the best technical solution but it's often the best…
Why not identify that same logical feature and move it into a library. How does a Microservice add value here?
Identifying, extracting and adding tests to logical features has been the sane way to rewrite software for ages. Michael feathers even wrote a book about it [1]. This ship of theseus approach works because it's incremental and allows for a complete rewrite without ever having non-functional software in between.
Adding a REST/networking/orchestration boundary to make it a Microservice just for the sake of extracting it adds a lot of complexity for no gain.
Microservice can be the right architecture, but not if all you want is to extract a library.
[1] https://understandlegacycode.com/blog/key-points-of-working-...