Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

phoronix.com

Post reply on HN
Print view View original

101–110 of 177 posts

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#101

Post by reisse »

falcor84 wrote: post #95
anal_reactor wrote: post #69

Earlier quoted context omitted.

> Maybe we should, but requiring the use of a new low level facility that was introduced in the 7.0 kernel, to address a regression that exists only in 7.0+, seems not great. Completely right. This sounds like a communication failure. Maybe Linux maintainers should pick a few applications that have "priority support" and problems with these applications are also problems with Linux itself. Breaking Postgres is a seri…

I for one liked the old and simple WE DO NOT BREAK USERSPACE attitude. https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE

Not sure it is true anymore. I've encountered few userspace breaks in io_uring, at least.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#102

Post by gcr »

falcor84 wrote: post #95
anal_reactor wrote: post #69

Earlier quoted context omitted.

> Maybe we should, but requiring the use of a new low level facility that was introduced in the 7.0 kernel, to address a regression that exists only in 7.0+, seems not great. Completely right. This sounds like a communication failure. Maybe Linux maintainers should pick a few applications that have "priority support" and problems with these applications are also problems with Linux itself. Breaking Postgres is a seri…

I for one liked the old and simple WE DO NOT BREAK USERSPACE attitude. https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE

Performance regressions are different from ABI incompatibilities. If the kernel refused to do any work that slowed down any userspace program, the pace would go a lot slower.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#103

Post by Neywiny »

vasco wrote: post #67
Neywiny wrote: post #13

Earlier quoted context omitted.

Not nobody but not everybody upgrades to the newest distros immediately. That's the advantage of LTS. I've even found that a lot of programs have poorer support on 24.04 than 22.04 due to security changes, so I'm fine sticking with 22.04 as my main dev system.

Someone said "its fine nobody uses this" and someone else gave the world's biggest slam dunk of "Ubuntu in 1 month" and your reply is that "not everyone does it". How far from the point can you be! In the Linux world this is the worst possible scenario, distro with the largest adoption, LTS.

22.04 is still potentially more prevalent than 24.04 according to https://fr.archive.ubuntu.com/stats/stats_of_day-16.html?ver... . 26.04 will take some time before it's largely adopted.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#104

Post by Neywiny »

stingraycharles wrote: post #14
Neywiny wrote: post #13

Earlier quoted context omitted.

Not nobody but not everybody upgrades to the newest distros immediately. That's the advantage of LTS. I've even found that a lot of programs have poorer support on 24.04 than 22.04 due to security changes, so I'm fine sticking with 22.04 as my main dev system.

This seems to be brushing off a major performance regression just because you personally don’t upgrade for 4 years. I don’t think that’s common at all.

https://fr.archive.ubuntu.com/stats/stats_of_day-16.html?ver... no need to think, data backs it up

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#105

Post by shadowgovt »

gcr wrote: post #102
falcor84 wrote: post #95

Earlier quoted context omitted.

I for one liked the old and simple WE DO NOT BREAK USERSPACE attitude. https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE

Performance regressions are different from ABI incompatibilities. If the kernel refused to do any work that slowed down any userspace program, the pace would go a lot slower.

Or be a lot uglier. See: Microsoft replacing its own API surfaces with binary-compatible representations to workaround companies like Adobe adding perf improvements like bypassing the kernel-provided kernel object constructors because it saved them a few cycles to just hard-code the objects they wanted and memcpy them into existence.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#106

Post by pgaddict »

anarazel wrote: post #48
zamalek wrote: post #47

Earlier quoted context omitted.

It was later reproduced on the same machine without huge pages enabled. PICNIC?

Yes, I did reproduce it (to a much smaller degree, but it's just a 48c/96t machine). But it's an absurd workload in an insane configuration. Not using huge pages hurts way more than the regression due to PREEMPT_LAZY does. With what we know so far, I expect that there are just about no real world workloads that aren't already completely falling over that will be affected.

So why does it happen only with hugepages? Is the extra overhead / TLB pressure enough to trigger the issue in some way? Of is it because the regular pages get swapped out (which hugepages can't be)?

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#107

Post by anarazel »

pgaddict wrote: post #106
anarazel wrote: post #48

Earlier quoted context omitted.

Yes, I did reproduce it (to a much smaller degree, but it's just a 48c/96t machine). But it's an absurd workload in an insane configuration. Not using huge pages hurts way more than the regression due to PREEMPT_LAZY does. With what we know so far, I expect that there are just about no real world workloads that aren't already completely falling over that will be affected.

So why does it happen only with hugepages? Is the extra overhead / TLB pressure enough to trigger the issue in some way? Of is it because the regular pages get swapped out (which hugepages can't be)?

I don't fully know, but I suspect it's just that due to the minor faults and tlb misses there is terrible contention with the spinlock, regardless of the PREEMPT_LAZY when using 4k pages (that easily reproducible). Which is then made worse by preempting more with the lock held.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#108

Post by ndsipa_pomu »

vasco wrote: post #67
Neywiny wrote: post #13

Earlier quoted context omitted.

Not nobody but not everybody upgrades to the newest distros immediately. That's the advantage of LTS. I've even found that a lot of programs have poorer support on 24.04 than 22.04 due to security changes, so I'm fine sticking with 22.04 as my main dev system.

Someone said "its fine nobody uses this" and someone else gave the world's biggest slam dunk of "Ubuntu in 1 month" and your reply is that "not everyone does it". How far from the point can you be! In the Linux world this is the worst possible scenario, distro with the largest adoption, LTS.

Not trying to downplay the importance of this, but the LTS versions aren't until the first point release, so 26.04.1 (typically six months or so after the release).

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#109

Post by cr125rider »

aftbit wrote: post #25
lfittl wrote: post #2

Its worth reading this follow-up LKML post by Andres Freund (who works on Postgres): https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...

>If this somehow does end up being a reproducible performance issue (I still suspect something more complicated is going on), I don't see how userspace could be expected to mitigate a substantial perf regression in 7.0 that can only be mitigated by a default-off non-trivial functionality also introduced in 7.0.

They said the magic words to get Linus to start flipping tables. Never break userspace. Unusably slow is broken

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#110

Post by josh-sematic »

yunohn wrote: post #94
Maxion wrote: post #86

Earlier quoted context omitted.

I would argue it's mainly inexperienced devs who deploy on the very latest. Once you get some more years under your belt you realize the value of LTS versions, even if you don't get the shiniest shiny.

> kernel version powering Ubuntu 26.04 *LTS*

Yes it’s LTS but the point is that the LTS system has overlapping support so you can wait on an older LTS for a bit before upgrading to a newer one. And it’s somewhat prudent to do so if you value stability highly, because often a few new issues will be discovered and patched after LTS goes live for a bit.
Post reply on HN