Live data from Hacker News

Subtly Bad Things Linux May Be Doing To PostgreSQL

rhaas.blogspot.in

71–80 of 95 posts

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#71
post #69

Issues like the ones raised are why all sufficiently advanced database engine designs tend to evolve toward a kernel bypass architecture. From the perspective of a database engine, operating systems do a lot of "dumb" things with resource management and scheduling in ways that are essentially impossible to avoid that the database engine has enough context to do intelligently on its own. OS bypass in a database can ha…

At this point why aren't you just building the database on top of a minimal OS? Then run that in a VM, say

> Then run that in a VM

You think that VMs aren't subject to the host OS scheduler, caching, and memory allocation quirks?

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#73

Issues like the ones raised are why all sufficiently advanced database engine designs tend to evolve toward a kernel bypass architecture. From the perspective of a database engine, operating systems do a lot of "dumb" things with resource management and scheduling in ways that are essentially impossible to avoid that the database engine has enough context to do intelligently on its own. OS bypass in a database can ha…

I assert that open source databases don't do it because it's a bad idea. It's the kind of you do when you have a lot of spare engineering resources and not many innovative ideas. Not only is the initial devepment expensive, so is the maintenance burden. It makes every new idea cost more to implement. Postgres has been extraordinarily innovative; offering things like transactional DDL, advanced indexing, first-class e…

You both overestimate the engineering complexity and underestimate the benefits. I've both designed and worked on a couple different bypass kernels as well PostgreSQL internals over the years.

You are correct that the initial development is steep. However, once the infrastructure is there it really is not much different than working with the operating system infrastructure and you gain a level of predictability and stability in terms of behavior that saves engineering time. Also, bypass implementations have almost no locking internally (either "lock-free" types or heavier types) which reduces complexity considerably.

Some bypass kernel code bases allow you to compile with the bypass implementation disabled, using highly-optimized PostgreSQL-like internals. I've seen and run quite a few comparative benchmarks on the same design with and without bypass enabled, as well as absolute benchmarks against engines like PostgreSQL. We don't have to guess about single node performance.

Broadly speaking, a properly designed bypass kernel buys you 2-3x the throughput of a highly optimized non-bypass kernel in my experience. If it was only 25% no one would bother. Furthermore, for massively parallel databases, you essentially require a bypass kernel to design a well-behaved system due to the adaptive operation scheduling requirements.

I agree that it is a lot of work but it is also entirely worth it if you need to either (1) maximize throughput on a single node and (2) build a well-behaved massively parallel database kernel. The differences are not trivial.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#74
post #60

Issues like the ones raised are why all sufficiently advanced database engine designs tend to evolve toward a kernel bypass architecture. From the perspective of a database engine, operating systems do a lot of "dumb" things with resource management and scheduling in ways that are essentially impossible to avoid that the database engine has enough context to do intelligently on its own. OS bypass in a database can ha…

These two issues are actually perfect counterpoint to bypassing kernel caching and scheduling. There is no way to overcome first issue in userspace, second is trigered by what amounts to too agressive caching in userspace, and third is something that you should not be doing anyway (I'm not exactly sure if there is something better that kernel can do in that case, except doing write-thru caching on writes which has it…

I think you aren't understanding what a bypass kernel does. It literally takes control of the physical resources to the extent kernel interfaces exist that allow it to do so and, at least in the case of Linux, the level of control possible is quite high. That means taking control of the CPU, physical memory, disk I/O, network I/O, etc so that they can be scheduled and managed from userspace. All at the same time starting with bypass kernel initialization. For obvious reasons, it is usually a bad idea to run other non-trivial processes on the same machine because they will tend to be resource starved.

Once you have these resources, you can organize them and use them as you see fit. Because it is not going to the kernel for any resources or buffering or scheduling or memory etc, there is little opportunity for the OS to do the wrong thing with resources that already are tightly controlled by the runtime. However, this is also why it is an "all or nothing" kind of situation.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#75

Issues like the ones raised are why all sufficiently advanced database engine designs tend to evolve toward a kernel bypass architecture. From the perspective of a database engine, operating systems do a lot of "dumb" things with resource management and scheduling in ways that are essentially impossible to avoid that the database engine has enough context to do intelligently on its own. OS bypass in a database can ha…

How does this work? I don't see how its possible to fight the scheduler or how caching works via userspace. Maybe I don't understand this stuff, but maybe the kernel should have some bypass API for high performance applications, instead of coders finding curious ways to fight it.

The kernel does have bypass APIs, or APIs that can be used for that purpose. However, you can't use just a little; once you start down that path you need to bypass everything.

To be clear, while the bypass APIs are simple to use you actually have to know what you are doing since you become responsible for doing things the OS used to do for you. I/O scheduling, disk caching, process scheduling, memory management, etc all have to be reimplemented in userspace.

It is why I mentioned that the skill set required to do bypass kernels is fairly rarified. You can't just reimplement what the OS already does, you need to implement something that is different than the OS design but also better at providing functionality the OS provides for the use case. You are essentially writing a purpose-optimized OS without the device drivers.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#76

Earlier quoted context omitted.

I assert that open source databases don't do it because it's a bad idea. It's the kind of you do when you have a lot of spare engineering resources and not many innovative ideas. Not only is the initial devepment expensive, so is the maintenance burden. It makes every new idea cost more to implement. Postgres has been extraordinarily innovative; offering things like transactional DDL, advanced indexing, first-class e…

You both overestimate the engineering complexity and underestimate the benefits. I've both designed and worked on a couple different bypass kernels as well PostgreSQL internals over the years. You are correct that the initial development is steep. However, once the infrastructure is there it really is not much different than working with the operating system infrastructure and you gain a level of predictability and s…

Or you could, you know, fix the OS like they are trying to do in TFA.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#77

For 1: I don't remember exactly which machines use NUMA, but I thought it was limited to 1st Opterons (and the behaviour makes sense) 2: Not sure, this may be specific to FS, or something that has to do with the behaviour of MMAPed files however I don't know how do you guarantee that what you're writing corresponds to a single block in the FS (unless you're writing directly to /dev/sda and even then)

1 affects any multi-socket Intel system after Nehalem, and any multi-socket Opteron+ AMD system.

So basically most physical servers.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#78

Earlier quoted context omitted.

I assert that open source databases don't do it because it's a bad idea. It's the kind of you do when you have a lot of spare engineering resources and not many innovative ideas. Not only is the initial devepment expensive, so is the maintenance burden. It makes every new idea cost more to implement. Postgres has been extraordinarily innovative; offering things like transactional DDL, advanced indexing, first-class e…

You both overestimate the engineering complexity and underestimate the benefits. I've both designed and worked on a couple different bypass kernels as well PostgreSQL internals over the years. You are correct that the initial development is steep. However, once the infrastructure is there it really is not much different than working with the operating system infrastructure and you gain a level of predictability and s…

Is there a middle ground here somewhere? In that the kernel developers create some sort of DB specific hooks that allow some of the kernel bypass mechanisms to be implemented?

What are the key things that a kernel bypass version does different? Can these be separated out in a concise way which would lead to multiple DB implementations being able to use these same interfaces? Essentially for any major DB system, you'd want the kernel tailored anyway - you're not going to be doing much else on your DB server (are you?)

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#79
post #59
post #8

Earlier quoted context omitted.

same here. Was a FreeBSD user, but find hard to find VM a few years ago that support FreeBSD, may switch back, if this issue is NOT addressed.

First issue is relevant only on systems that have more than one NUMA node, which is probably every meaningful physical server and essentially no VM (at least on Xen, multiprocessor VMs are single NUMA node), as it does not make much sense to advertise NUMA topology to guest VMs. Second issue is relevant for postgresql mostly only if you use very large shared_buffers which anyway is not recommended for general workloa…

NUMA can absolutely ping you in virtual servers, but without access to the hypervisor you'll never know why it's happening (JVMs straddling NUMA regions have caused me pain in the past, when the guest was split across memory regions).

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#80
post #24

Earlier quoted context omitted.

Sure, if you are running stats across everything in a nontrivial and frequently changing way, then you have a great ally in an RDBMS. But I don't believe many people do that, because usually that sort of stuff is pretty damn predictable, executed offline, or can be consolidated from shards. However, if you have any of the following: (1) vastly different security requirements for different parts of your datastore (2)…

I counter that many people have met each of your numbers for the past 20 years using commercial RDBMS. I can't think of anything that is magnificently easier or better at solving your numbers, especially all together. #4 seems less relevant, is it really cheaper than operationalizing a distributed system? These days, likely for situations where consistency can be relaxed. Not so for many business workloads. Can you e…

Haha, went out and these comments got downvoted to pluto. Honestly though, I haven't heard a decent argument in response other than "lazy is good". Sure, but architecturally, you're basically in the "engineers run the architecture" or "its an architecture of convenience for business purposes" camp. I'm in the former, I'd like to hope that some nontrivial subset of the participants here are in the former, but most are no doubt in the latter. People get upset when you slight their world. That's understandable. The TLDR is: even if people made a lot of stuff happen 20 years ago; it doesn't justify using the same methods today, and discussing the tradeoffs is constructive not dismissive.
Post reply on HN