Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

phoronix.com

Post reply on HN
Print view View original

61–70 of 177 posts

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#61

Post by shakna »

FireBeyond wrote: post #3

Once upon a time, Linus would shout and yell about how the kernel should never "break" userspace (and I see in some places, some arguments of "It's not broken, it's just a performance regression" - personally I'd argue a 50% hit to performance of a pre-eminent database engine is ... quite the regression). Now, the kernel engineer who introduced the brand new mechanism (introduced in Linux 7.0) for handling pre-emptio…

Freund seems to suggest that hugepages is the right way to run a system under this sort of load - which is the fix.

> Hah. I had reflexively used huge_pages=on - as that is the only sane thing to do with 10s to 100s of GB of shared memory and thus part of all my benchmarking infrastructure - during the benchmark runs mentioned above.

> Turns out, if I disable huge pages, I actually can reproduce the contention that Salvatore reported (didn't see whether it's a regression for me though). Not anywhere close to the same degree, because the bottleneck for me is the writes.

But, they can speak for themselves here [0].

[0] https://news.ycombinator.com/item?id=47646332

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#62

Post by IshKebab »

mgaunard wrote: post #55
jcalvinowens wrote: post #26

Earlier quoted context omitted.

> I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations. Yeah, exactly. "Doctor, help, somebody replaced my wooden hammer with a metal one, and now I can't hit myself in the face with it as many times." If you use spinlocks in userspace, you're gonna have a bad time.

Most people looking for performance will reach for the spinlock. The expectation is that the kernel should somehow detect applications that are spinning, and avoid preempting them early.

Well that seems like an unreasonable expectation no? Also isn't the point of spinlocks that they get released before the kernel does anything? Otherwise you could just use a futex... Which maybe you should do anyway...

https://matklad.github.io/2020/01/04/mutexes-are-faster-than...

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#63

Post by alienchow »

anal_reactor wrote: post #51

Can someone explain to me what's the problem? I have very little knowledge of Linux kernel, but I'm curious. I've tried reading a little, but it's jargon over jargon.

I'm not familiar with the jargon either, but based on some reading it comes down to how the latest kernel treats process preempts.

Postgres uses spinlocks to hold shared memory for very critical processes. Spinlocks are an infinite loop with no sleep to attempt to hold a lock, thus "spinning". Previous kernels allowed spinlocking processes to run with PREEMPT_NONE. This flag tells the kernel to let the locking process complete their work before doing anything. Now the latest kernel removed this functionality and is interrupting spinlocking processes. So if a process that is holding a lock gets interrupted, all other postgres spinlocks processes that need the same lock spin in place for way longer times, leading to performance degradation.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#65

Post by torginus »

TacticalCoder wrote: post #19
lfittl wrote: post #2

Its worth reading this follow-up LKML post by Andres Freund (who works on Postgres): https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...

AIUI in that thread they're saying "0.51x" the perf on a 96-core arm64 machine and they're also saying they cannot reproduce it on a 96-core amd64 machine. So it's not going to affect everybody both running PostgreSQL and upgrading to the latest kernel. Conditions seems to be: arm64, shitloads of core, kernel 7.0, current version of PostgreSQL. That is not going to be 100% of the installed PostgreSQL DBs out there in…

It's a huge issue of ARM based systems, that hardly anyone uses or tests things on them (in production).

Yes, Macs going ARM has been a huge boon, but I've also seen crazy regressions on AWS Graviton (compared to how its supposed to perform), on .NET (and node as well), which frankly I have no expertise or time digging into.

Which was the main reason we ultimately cancelled our migration.

I'm sure this is the same reason why its important to AWS.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#66

Post by anal_reactor »

alienchow wrote: post #63
anal_reactor wrote: post #51

Can someone explain to me what's the problem? I have very little knowledge of Linux kernel, but I'm curious. I've tried reading a little, but it's jargon over jargon.

I'm not familiar with the jargon either, but based on some reading it comes down to how the latest kernel treats process preempts. Postgres uses spinlocks to hold shared memory for very critical processes. Spinlocks are an infinite loop with no sleep to attempt to hold a lock, thus "spinning". Previous kernels allowed spinlocking processes to run with PREEMPT_NONE. This flag tells the kernel to let the locking proces…

Why does it only appear on arm64 and not x86?

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#67

Post by vasco »

Neywiny wrote: post #13
Meekro wrote: post #8

Earlier quoted context omitted.

> Nobody sensible runs the latest kernel From the article: "Linux 7.0 stable is due out in about two weeks. This is also the kernel version powering Ubuntu 26.04 LTS to be released later in April." Unfortunately, lots of people will be running it in less than a month. At the moment, it'll take a kernel patch (not a sysctl) to undo this-- hopefully something changes.

Not nobody but not everybody upgrades to the newest distros immediately. That's the advantage of LTS. I've even found that a lot of programs have poorer support on 24.04 than 22.04 due to security changes, so I'm fine sticking with 22.04 as my main dev system.

Someone said "its fine nobody uses this" and someone else gave the world's biggest slam dunk of "Ubuntu in 1 month" and your reply is that "not everyone does it". How far from the point can you be!

In the Linux world this is the worst possible scenario, distro with the largest adoption, LTS.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#68

Post by bob1029 »

I'm struggling a bit with why we need all these fancy dynamic preemption modes. Is this about hyperscalars shoving more VMs per physical machine? What does a person trying to host a software solution gain from this kernel change?

If a user wants to spin in an infinite loop all day every day, I don't see the problem with that. Even if the spinning will provably never do any useful work.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#69

Post by anal_reactor »

lfittl wrote: post #2

Its worth reading this follow-up LKML post by Andres Freund (who works on Postgres): https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...

> Maybe we should, but requiring the use of a new low level facility that was introduced in the 7.0 kernel, to address a regression that exists only in 7.0+, seems not great.

Completely right. This sounds like a communication failure. Maybe Linux maintainers should pick a few applications that have "priority support" and problems with these applications are also problems with Linux itself. Breaking Postgres is a serious regression.

Reminds me of a situation where Fedora couldn't be updated if you had Wine installed and one side of the argument was "user applications are user problem" while the other was "it's Wine, like come on".

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#70

Post by adrian_b »

justinclift wrote: post #21
lfittl wrote: post #2

Its worth reading this follow-up LKML post by Andres Freund (who works on Postgres): https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...

Note that it's just not a single post, and there's additional further information in following the full thread. :)

Yes, and in the following messages the conclusion was that the regression is mitigated when using huge pages.
Post reply on HN