Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

phoronix.com

Post reply on HN
Print view View original

71–80 of 177 posts

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#71

Post by adrian_b »

MBCook wrote: post #38
TacticalCoder wrote: post #19

Earlier quoted context omitted.

AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine. So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL. That is not going to be 100% of the installed PostgreSQL DBs out there in…

So perhaps this is a regression specifically in the arm64 code, or said differently maybe it’s a performance bug that has been there for a long time but covered up by the scheduler part that was removed?

The following messages concluded that using huge pages mitigates the regression, while not using huge pages reproduces it.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#72

Post by adrian_b »

Deeg9rie9usi wrote: post #60

Once again phoronix shoot out an article without further researching nor letting the mail thread in question cool down. The follow up mails make clear that the issue is more or less a non-issue since the benchmark is wrong.

The following up mails conclude that the regression happens only when huge pages are not used.

While using huge pages whenever possible is the right solution and this should be enough for PostgreSQL, perhaps there are applications that cannot use huge pages and which are affected by the regression.

So I do not think that it is right to just ignore what happened.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#73

Post by adrian_b »

anal_reactor wrote: post #66
alienchow wrote: post #63

Earlier quoted context omitted.

I'm not familiar with the jargon either, but based on some reading it comes down to how the latest kernel treats process preempts. Postgres uses spinlocks to hold shared memory for very critical processes. Spinlocks are an infinite loop with no sleep to attempt to hold a lock, thus "spinning". Previous kernels allowed spinlocking processes to run with PREEMPT_NONE. This flag tells the kernel to let the locking proces…

Why does it only appear on arm64 and not x86?

It was not architecture-related. Not using huge pages also reproduced the regression on x86.

I do not know why using huge pages mitigates the regression, but it could be just because when the application uses huge pages it uses spinlocks much less frequently so the additional delays do not accumulate enough to cause a significant performance reduction.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#74

Post by wongogue »

pmontra wrote: post #50
master_crab wrote: post #24

Earlier quoted context omitted.

For production Postgres, i would assume it’s close to almost no effect? If someone is running postgres in a serious backend environment, i doubt they are using Ubuntu or even touching 7.x for months (or years). It’ll be some flavor of Debian or Red Hat still on 6.x (maybe even 5?). Those same users won’t touch 7.x until there has been months of testing by distros.

A customer of mine is running on Ubuntu 22.04 and the plan is to upgrade to 26.04 in Q1 2027. We'll have to add performance regression to the plan.

Are you running ARM servers?

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#75

Post by tijsvd »

anal_reactor wrote: post #51

Can someone explain to me what's the problem? I have very little knowledge of Linux kernel, but I'm curious. I've tried reading a little, but it's jargon over jargon.

From what I understand in the follow up: postgres uses shared memory for buffers. This shared memory is read by a new connection while locked.

In postgres, connections are handled with a process fork, not a new thread. If such a fork first reads memory, even if it already exists, that causes a minor page fault, which goes back to the kernel so it can update memory mapping tables.

The operation under lock is only a few instructions, but if it takes longer than expected, then that causes lock contention. Regression in the kernel handling minor faults?

The whole thing is then made worse because it's a spinlock, causing all waiting processes to contend over the cpus which adds to kernel processing.

Mitigated by using huge pages, which dramatically reduces the number of mapping entries and faults. I reckon that it could also be mitigated in postgres by pre-faulting all shared memory early?

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#76

Post by GandalfHN »

jeffbee wrote: post #7
lfittl wrote: post #2

Its worth reading this follow-up LKML post by Andres Freund (who works on Postgres): https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...

Funny how "use hugepages" is right there on the table and 99% of users ignore it.

[dead]

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#77

Post by jeltz »

adrian_b wrote: post #70
justinclift wrote: post #21

Earlier quoted context omitted.

Note that it's just not a single post, and there's additional further information in following the full thread. :)

Yes, and in the following messages the conclusion was that the regression is mitigated when using huge pages.

Which you always should use anyway if you can.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#78

Post by teekert »

Meekro wrote: post #8
dsr_ wrote: post #4

Nobody sensible runs the latest kernel; nobody running PG in production should be afraid of setting a non-default at either boot time or as a sysctl. So this will, most likely, be another step in building a PG database server (turn off pre-emption if your kernel is 7.0 or later and PG is pre-whatever-version). At worst it might become a permanent part of building a PG server and a FAQ... but if it affects one thing t…

> Nobody sensible runs the latest kernel From the article: "Linux 7.0 stable is due out in about two weeks. This is also the kernel version powering Ubuntu 26.04 LTS to be released later in April." Unfortunately, lots of people will be running it in less than a month. At the moment, it'll take a kernel patch (not a sysctl) to undo this-- hopefully something changes.

I think most people on enterprise-y systems wait for (at least) 26.04.1, the window is 3 years (when on 24.04, which is supported until ~2029-04-30, it's 1 year when on 22.04) starting now, hardly anyone switches immediately.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#79

Post by jeltz »

monocasa wrote: post #18

I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations.

PostgreSQL is old and had to support kernels which did not support spinlocks. But, yes, maybe PostgreSQL should stop doing so now that kernels do.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#80

Post by apelapan »

SoftTalker wrote: post #28
esafak wrote: post #23

Earlier quoted context omitted.

That's the advantage of LTS? 24.04 is the LTS, not the one you use, 22.04.

22.04 is also an LTS release, supported for another year still. https://ubuntu.com/about/release-cycle We're just now looking at moving production machines to 24.04.

If you are on a maintenance contract with Ubuntu, 22.04 is supported until 2032.

If it aint broken, don't fix it.

Post reply on HN