Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

phoronix.com

Post reply on HN
Print view View original

41–50 of 177 posts

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#41

Post by arjie »

FireBeyond wrote: post #3

Once upon a time, Linus would shout and yell about how the kernel should never "break" userspace (and I see in some places, some arguments of "It's not broken, it's just a performance regression" - personally I'd argue a 50% hit to performance of a pre-eminent database engine is ... quite the regression). Now, the kernel engineer who introduced the brand new mechanism (introduced in Linux 7.0) for handling pre-emptio…

Well, the reason he'd yell about it is that someone did it. If no one ever did it, he'd never yell and we'd never have the rule. So one can only imagine that this is one of those things where someone has to keep holding the line rather than one of those things where you set some rule and it self-holds.

Doubtless someone will have to do the yelling.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#42

Post by rvnx »

crcastle wrote: post #39
nine_k wrote: post #36

Earlier quoted context omitted.

Do they upgrade to the new LTS the day it is released?

Not historically.

and they are right, this is because a lot of junior sysadmins believe that newer = better.

But the reality:

  a) may get irreversible upgrades (e.g. new underlying database structure) 
  b) permanent worse performance / regression (e.g. iOS 26)
  c) added instability
  d) new security issues (litellm)
  e) time wasted migrating / debugging
  f) may need rewrite of consumers / users of APIs / sys calls
  g) potential new IP or licensing issues
etc.

A couple of the few reasons to upgrade something is:

  a) new features provide genuine comfort or performance upgrade (or... some revert)
  b) there is an extremely critical security issue
  c) you do not care about stability because reverting is uneventful and production impact is nil (e.g. Claude Code)
but 99% of the time, if ain't broke, don't fix it.

https://en.wikipedia.org/wiki/2024_CrowdStrike-related_IT_ou...

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#43

Post by Seattle3503 »

dsr_ wrote: post #4

Nobody sensible runs the latest kernel; nobody running PG in production should be afraid of setting a non-default at either boot time or as a sysctl. So this will, most likely, be another step in building a PG database server (turn off pre-emption if your kernel is 7.0 or later and PG is pre-whatever-version). At worst it might become a permanent part of building a PG server and a FAQ... but if it affects one thing t…

If you're running in a docker container you share the host kernel. You might not have a choice.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#45

Post by db48x »

MBCook wrote: post #38
TacticalCoder wrote: post #19

Earlier quoted context omitted.

AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine. So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL. That is not going to be 100% of the installed PostgreSQL DBs out there in…

So perhaps this is a regression specifically in the arm64 code, or said differently maybe it’s a performance bug that has been there for a long time but covered up by the scheduler part that was removed?

Could be either of those, or something else entirely. Or even measurement error.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#47

Post by zamalek »

TacticalCoder wrote: post #19
lfittl wrote: post #2

Its worth reading this follow-up LKML post by Andres Freund (who works on Postgres): https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...

AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine. So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL. That is not going to be 100% of the installed PostgreSQL DBs out there in…

It was later reproduced on the same machine without huge pages enabled. PICNIC?

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#48

Post by anarazel »

zamalek wrote: post #47
TacticalCoder wrote: post #19

Earlier quoted context omitted.

AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine. So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL. That is not going to be 100% of the installed PostgreSQL DBs out there in…

It was later reproduced on the same machine without huge pages enabled. PICNIC?

Yes, I did reproduce it (to a much smaller degree, but it's just a 48c/96t machine). But it's an absurd workload in an insane configuration. Not using huge pages hurts way more than the regression due to PREEMPT_LAZY does.

With what we know so far, I expect that there are just about no real world workloads that aren't already completely falling over that will be affected.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#49

Post by GandalfHN »

dsr_ wrote: post #4

Nobody sensible runs the latest kernel; nobody running PG in production should be afraid of setting a non-default at either boot time or as a sysctl. So this will, most likely, be another step in building a PG database server (turn off pre-emption if your kernel is 7.0 or later and PG is pre-whatever-version). At worst it might become a permanent part of building a PG server and a FAQ... but if it affects one thing t…

[dead]

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#50

Post by pmontra »

master_crab wrote: post #24
TacticalCoder wrote: post #19

Earlier quoted context omitted.

AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine. So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL. That is not going to be 100% of the installed PostgreSQL DBs out there in…

For production Postgres, i would assume it’s close to almost no effect? If someone is running postgres in a serious backend environment, i doubt they are using Ubuntu or even touching 7.x for months (or years). It’ll be some flavor of Debian or Red Hat still on 6.x (maybe even 5?). Those same users won’t touch 7.x until there has been months of testing by distros.

A customer of mine is running on Ubuntu 22.04 and the plan is to upgrade to 26.04 in Q1 2027. We'll have to add performance regression to the plan.
Post reply on HN