Subtly Bad Things Linux May Be Doing To PostgreSQL • Hacker News Board

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#91

Post by deafbybeheading » Fri, Apr 04, 2014, 7:47 AM UTC

jandrewrogers wrote: post #87

jeffdavis wrote: post #83

Earlier quoted context omitted.

Any number we have is going to be sensitive to the workload, so I think it's unfair to say 2-3x without a lot of context. Also, you dismiss ideas that help the database and the OS work together better. For instance, I did "synchronized scans" for postgres. It coordinates sequential scans to start from the block another scan is already reading, improving cache behavior and dramatically reducing seeks. This could have…

Jeff, I am familiar with your work, I lurked on the PostgreSQL hackers mailing list for years when I was hacking on that database. :-) I am not dismissing the coordination of OS and database, it just has really deep limits because the OS must hide information critical to optimizing database throughput. While the increased throughput is a complex function of hardware, workload, etc, it is also consistently substantial…

Stupid (and somewhat tangential) question: how do bypass kernels work with virtualization, if at all?

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#92

Post by cookiecaper » Fri, Apr 04, 2014, 7:57 AM UTC

kev009 wrote: post #76

jandrewrogers wrote: post #73

Earlier quoted context omitted.

You both overestimate the engineering complexity and underestimate the benefits. I've both designed and worked on a couple different bypass kernels as well PostgreSQL internals over the years. You are correct that the initial development is steep. However, once the infrastructure is there it really is not much different than working with the operating system infrastructure and you gain a level of predictability and s…

Or you could, you know, fix the OS like they are trying to do in TFA.

The argument is that "fix" is relative. Things that improves Postgres's performance may negatively impact other applications. The suggestion, therefore, is that Postgres takes control of these tasks for its own purposes, and then they don't have to worry about the implications for other systems or wait on anyone else.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#93

Post by jeffdavis » Fri, Apr 04, 2014, 3:48 PM UTC

jandrewrogers wrote: post #87

jeffdavis wrote: post #83

Earlier quoted context omitted.

Any number we have is going to be sensitive to the workload, so I think it's unfair to say 2-3x without a lot of context. Also, you dismiss ideas that help the database and the OS work together better. For instance, I did "synchronized scans" for postgres. It coordinates sequential scans to start from the block another scan is already reading, improving cache behavior and dramatically reducing seeks. This could have…

Jeff, I am familiar with your work, I lurked on the PostgreSQL hackers mailing list for years when I was hacking on that database. :-) I am not dismissing the coordination of OS and database, it just has really deep limits because the OS must hide information critical to optimizing database throughput. While the increased throughput is a complex function of hardware, workload, etc, it is also consistently substantial…

Perhaps it's just because I've never seen a good implementation of a bypass, and I might agree if I had seen one. Like many things, maybe it just takes the right people to make it successful.

Postgres leaves a lot of performance on the table in much more basic ways, too, so I certainly am not suggesting that postgres is anywhere near optimal.

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#94

Post by Dylan16807 » Sat, Apr 05, 2014, 5:43 AM UTC

ori_b wrote: post #71

pekk wrote: post #69

Earlier quoted context omitted.

At this point why aren't you just building the database on top of a minimal OS? Then run that in a VM, say

> Then run that in a VM You think that VMs aren't subject to the host OS scheduler, caching, and memory allocation quirks?

>scheduler

eh, it could pin threads at realtime priority

>caching

vm wouldn't use the OS cache

>memory allocation

it can easily allocate memory up front

Re: Subtly Bad Things Linux May Be Doing To PostgreSQL

#95

Post by profquail » Sun, Apr 06, 2014, 2:28 AM UTC

ams6110 wrote: post #72

forkandwait wrote: post #2

Anybody know of a FreeBSD comparison?

Dragonfly is probably the best BSD to run for database performance. http://www.dragonflybsd.org/performance/

That chart is 18 months old. FreeBSD 10 got a number of SMP-related improvements (and some for ZFS as well, if I'm not mistaken). It'd be interesting to see an up-to-date comparison with FreeBSD 10 (and maybe 11-CURRENT), DragonFlyBSD 3.6, and NetBSD 6.1 (and maybe a couple of Linux distros for good measure).