[deleted]
AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
31–40 of 177 posts
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#32Earlier quoted context omitted.
AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine. So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL. That is not going to be 100% of the installed PostgreSQL DBs out there in…
For production Postgres, i would assume it’s close to almost no effect? If someone is running postgres in a serious backend environment, i doubt they are using Ubuntu or even touching 7.x for months (or years). It’ll be some flavor of Debian or Red Hat still on 6.x (maybe even 5?). Those same users won’t touch 7.x until there has been months of testing by distros.
Ubuntu is used in many serious backend environments. Heroku runs tens of thousands (if not more) instances of Ubuntu on its fleet. Or at least it did through the teens and early 2020s.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#33Once upon a time, Linus would shout and yell about how the kernel should never "break" userspace (and I see in some places, some arguments of "It's not broken, it's just a performance regression" - personally I'd argue a 50% hit to performance of a pre-eminent database engine is ... quite the regression). Now, the kernel engineer who introduced the brand new mechanism (introduced in Linux 7.0) for handling pre-emptio…
This was my immediate thought - kernel doesn’t break software, or at least it didn’t used to.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#34Earlier quoted context omitted.
That may be the case, but it’s still not a great situation to be in and one has to wonder: if PostgreSQL is affected, what else is?
That's the big thing - PSQL will be tested, noticed, and fixed (and likely have a version that handles 7.0 by the time it's in common use). But other software won't and may not even be noticed, except as a (I hate using the term) enshittification. Better to introduce the "correct way" in 7.0 but not regress the old (translate the "correct" into the old if necessary) - and then in 8.0 or some future release implement…
Exactly, this is how it’s usually done. As the developer on the mailing list mentions, implementing a new low level construct in 7.0 and a performance regression that requires said new low level construct to mitigate is not great. You need a grace period in which both old and new approach is fast.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#35Earlier quoted context omitted.
Not nobody but not everybody upgrades to the newest distros immediately. That's the advantage of LTS. I've even found that a lot of programs have poorer support on 24.04 than 22.04 due to security changes, so I'm fine sticking with 22.04 as my main dev system.
That's the advantage of LTS? 24.04 is the LTS, not the one you use, 22.04.
All even number .04 releases are LTS in Ubuntu
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#36Earlier quoted context omitted.
For production Postgres, i would assume it’s close to almost no effect? If someone is running postgres in a serious backend environment, i doubt they are using Ubuntu or even touching 7.x for months (or years). It’ll be some flavor of Debian or Red Hat still on 6.x (maybe even 5?). Those same users won’t touch 7.x until there has been months of testing by distros.
Ubuntu is used in many serious backend environments. Heroku runs tens of thousands (if not more) instances of Ubuntu on its fleet. Or at least it did through the teens and early 2020s. https://devcenter.heroku.com/articles/stack
Do they upgrade to the new LTS the day it is released?
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#37Once upon a time, Linus would shout and yell about how the kernel should never "break" userspace (and I see in some places, some arguments of "It's not broken, it's just a performance regression" - personally I'd argue a 50% hit to performance of a pre-eminent database engine is ... quite the regression). Now, the kernel engineer who introduced the brand new mechanism (introduced in Linux 7.0) for handling pre-emptio…
It wouldn’t be the first time one of the other maintainers ran afoul of “Linus’s law“.
He may simply be waiting until more is known on exactly what’s causing it.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#38Its worth reading this follow-up LKML post by Andres Freund (who works on Postgres): https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...
AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine. So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL. That is not going to be 100% of the installed PostgreSQL DBs out there in…
So perhaps this is a regression specifically in the arm64 code, or said differently maybe it’s a performance bug that has been there for a long time but covered up by the scheduler part that was removed?
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#39Earlier quoted context omitted.
Ubuntu is used in many serious backend environments. Heroku runs tens of thousands (if not more) instances of Ubuntu on its fleet. Or at least it did through the teens and early 2020s. https://devcenter.heroku.com/articles/stack
Do they upgrade to the new LTS the day it is released?
Not historically.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#40Nobody sensible runs the latest kernel; nobody running PG in production should be afraid of setting a non-default at either boot time or as a sysctl. So this will, most likely, be another step in building a PG database server (turn off pre-emption if your kernel is 7.0 or later and PG is pre-whatever-version). At worst it might become a permanent part of building a PG server and a FAQ... but if it affects one thing t…
> Nobody sensible runs the latest kernel From the article: "Linux 7.0 stable is due out in about two weeks. This is also the kernel version powering Ubuntu 26.04 LTS to be released later in April." Unfortunately, lots of people will be running it in less than a month. At the moment, it'll take a kernel patch (not a sysctl) to undo this-- hopefully something changes.
Depends on your shop.
As someone with a heavy QA/Dev Opps background I don't think we have enough details.
Is it only ARM64 ? How many ARM64 PG DBs are running 96 cores?
However...
This is the most popular database in the world. Odds are this will effect a bunch of other lesser known applications.