[dead]
AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
81–90 of 177 posts
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#82Once again phoronix shoot out an article without further researching nor letting the mail thread in question cool down. The follow up mails make clear that the issue is more or less a non-issue since the benchmark is wrong.
The following up mails conclude that the regression happens only when huge pages are not used. While using huge pages whenever possible is the right solution and this should be enough for PostgreSQL, perhaps there are applications that cannot use huge pages and which are affected by the regression. So I do not think that it is right to just ignore what happened.
I agree with you. The lurid headlines of phoronix.com just annoy me...
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#83Earlier quoted context omitted.
Funny how "use hugepages" is right there on the table and 99% of users ignore it.
I’m absolutely flabbergasted by the performance left on the table; even by myself - just yesterday I learned Gentoo’s emerge can use git and be a billion times faster.
The time spent by emerge is utterly dwarfed by the time spent to build the packages, so who cares? Maybe it's different if installing a binary system but don't think most people are doing that.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#84Earlier quoted context omitted.
Why does it only appear on arm64 and not x86?
It was not architecture-related. Not using huge pages also reproduced the regression on x86. I do not know why using huge pages mitigates the regression, but it could be just because when the application uses huge pages it uses spinlocks much less frequently so the additional delays do not accumulate enough to cause a significant performance reduction.
The problem is the spinlock being interrupted by a minor fault (you're touching a page of memory for the first time, and the kernel needs to set it up the first time it's actually used)
If your pages are 1GB instead of 4kB, this happens much less often.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#85Earlier quoted context omitted.
Not historically.
and they are right, this is because a lot of junior sysadmins believe that newer = better. But the reality: a) may get irreversible upgrades (e.g. new underlying database structure) b) permanent worse performance / regression (e.g. iOS 26) c) added instability d) new security issues (litellm) e) time wasted migrating / debugging f) may need rewrite of consumers / users of APIs / sys calls g) potential new IP or licen…
all fair points, on the other hand, as a general rule, isn't it important to stay on currently-supported versions of pieces of software that you run?
ymmv, but in my experience projects like postgresql which have been reliable, tend to continue to be so.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#86Earlier quoted context omitted.
Not nobody but not everybody upgrades to the newest distros immediately. That's the advantage of LTS. I've even found that a lot of programs have poorer support on 24.04 than 22.04 due to security changes, so I'm fine sticking with 22.04 as my main dev system.
> ... not everybody upgrades to the newest distros immediately. While that's true, for new deployments the story is often "deploy on the latest release of things available at the time". So, there will probably be a substantial deployment of new projects / testing projects using the Linux 7.0 kernel along with the latest available software packages in a few weeks.
I would argue it's mainly inexperienced devs who deploy on the very latest. Once you get some more years under your belt you realize the value of LTS versions, even if you don't get the shiniest shiny.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#87Earlier quoted context omitted.
Ubuntu is used in many serious backend environments. Heroku runs tens of thousands (if not more) instances of Ubuntu on its fleet. Or at least it did through the teens and early 2020s. https://devcenter.heroku.com/articles/stack
Do they upgrade to the new LTS the day it is released?
Ubuntu's upgrade tools wait until the .1 release for LTSes, so your typical installation would wait at least half a year.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#88Earlier quoted context omitted.
So perhaps this is a regression specifically in the arm64 code, or said differently maybe it’s a performance bug that has been there for a long time but covered up by the scheduler part that was removed?
Could be either of those, or something else entirely. Or even measurement error.
Turns out the amd machine had huge tables enabled and after disabling those the regression was there on and too. So arm vs amd was a red herring.
Of course not a nice regression but you should not run PostgreSQL on large servers without huge pages enabled so thud regression will only hurt people who have a bad configuration. That said I think these bad configurations are common out there, especially in containerized environments where the one running PostgreSQL may not have the ability to enable huge pages.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#89Not sure why people have to upgrade to the newest major kernel version as soon as it is released.
If nobody tests and reports these things when the version is released the regression would not be fixed when people start using it in production.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#90Earlier quoted context omitted.
Could be either of those, or something else entirely. Or even measurement error.
Turns out the amd machine had huge tables enabled and after disabling those the regression was there on and too. So arm vs amd was a red herring. Of course not a nice regression but you should not run PostgreSQL on large servers without huge pages enabled so thud regression will only hurt people who have a bad configuration. That said I think these bad configurations are common out there, especially in containerized…
Still that huge a regression that affects multiple platforms doesn't sound too neat, did they narrow down the root cause?