AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy • Hacker News Board

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#151

Post by scottlamb » Mon, Apr 06, 2026, 4:11 AM UTC

justinclift wrote: post #125

scottlamb wrote: post #116

Earlier quoted context omitted.

> While using huge pages whenever possible is the right solution and this should be enough for PostgreSQL, perhaps there are applications that cannot use huge pages and which are affected by the regression. It will be more interesting to talk about those applications if and when they are found. And I wouldn't assume the solutions are limited to reverting this change, starting to use the new spinlock time-slice extens…

> It will be more interesting to talk about those applications if and when they are found. Redis recommend disabling hugepages: https://redis.io/docs/latest/operate/oss_and_stack/managemen... --- Actually, looks like they changed the log warning to be more specific, as it's just the "always" setting which seems to cause Redis grief? https://github.com/redis/redis/issues/3895

By "those applications" I'm talking about other applications affected by this regression. There are several apps in addition to Redis that recommend limiting the transparent huge page configuration. (Some of them recommend using explicit huge pages instead.) But it's quite possible none of them are affected by this regression, as it may be particular to apps using spinlocks. (Certainly the new rseq API mentioned in the thread is targeted at spinlock users.) It seems equally possible to me that some spinlock-using app has a regression irrespective of huge pages.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#152

Post by scottlamb » Mon, Apr 06, 2026, 4:25 AM UTC

db48x wrote: post #130

scottlamb wrote: post #117

Earlier quoted context omitted.

I doubt they explicitly said "I'll run without huge pages, which is an important AWS configuration". They probably just forgot a step. And "someone at Amazon" describes a lot of people; multiply your mental probability tables accordingly.

The number of people at Amazon is pretty much irrelevant; the org is going to ensure that someone is keeping an eye on kernel performance, but also that the work isn’t duplicative. Surely they would be testing the configuration(s) that they use in production? They’re not running RDS without hugepages turned on, right?

> The number of people at Amazon is pretty much irrelevant; the org is going to ensure that someone is keeping an eye on kernel performance, but also that the work isn’t duplicative.

I'd guess they have dozens of people across say a Linux kernel team, a Graviton hardware integration team, an EC2 team, and a Amazon RDS for PostgreSQL team who might at one point or another run a benchmark like this. They probably coordinate to an extent, but not so much that only one person would ever run this test. So yes it is duplicative. And they're likely intending to test the configurations they use in production, yes, but people just make mistakes.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#153

Post by db48x » Mon, Apr 06, 2026, 7:31 AM UTC

cperciva wrote: post #140

db48x wrote: post #46

Earlier quoted context omitted.

Heh. Did they at least add useful features to balance out that cost?

FreeBSD 15 has lots of useful features! And better performance on other benchmarks; I just need to track down what's going wrong with this particular one.

Certainly could be worse!

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#154

Post by db48x » Mon, Apr 06, 2026, 7:49 AM UTC

scottlamb wrote: post #152

db48x wrote: post #130

Earlier quoted context omitted.

The number of people at Amazon is pretty much irrelevant; the org is going to ensure that someone is keeping an eye on kernel performance, but also that the work isn’t duplicative. Surely they would be testing the configuration(s) that they use in production? They’re not running RDS without hugepages turned on, right?

> The number of people at Amazon is pretty much irrelevant; the org is going to ensure that someone is keeping an eye on kernel performance, but also that the work isn’t duplicative. I'd guess they have dozens of people across say a Linux kernel team, a Graviton hardware integration team, an EC2 team, and a Amazon RDS for PostgreSQL team who might at one point or another run a benchmark like this. They probably coord…

True; to err is human. But it is weird that they didn’t just fire up a standard RDS instance of one or more sizes and test those. After all, it’s already automated; two clicks on the website gets you a standard configuration and a couple more get you a 96c graviton cpu. I just wonder how the mistake happened.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#155

Post by linzhangrun » Mon, Apr 06, 2026, 7:53 AM UTC

fxtentacle wrote: post #112

lfittl wrote: post #2

Its worth reading this follow-up LKML post by Andres Freund (who works on Postgres): https://lore.kernel.org/lkml/yr3inlzesdb45n6i6lpbimwr7b25kqk...

.. which confirms all of my stereotypes. Looks like the AWS engineer who reported it used a m8g.24xlarge instance with 384 GB of RAM, but somehow didn't know or care to enable huge pages. And once enabling them, the performance regression disappears.

[flagged]

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#156

Post by adrian_b » Mon, Apr 06, 2026, 8:12 AM UTC

amluto wrote: post #127

anarazel wrote: post #113

Earlier quoted context omitted.

I really dislike the use of spinlocks in postgres (and have been replacing a lot of uses over time), but it's not always easy to replace them from a performance angle. On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Turns out that that increase in memory barriers causes regr…

> On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Now you've gotten me wondering. This issue is, in some sense, artificial: the actual conceptual futex unlock operation does not require sequential consistency. What's needed is (roughly, anyway) an release operation that sync…

Using LOCK CMPXCHG or even plain CMPXCHG does not make sense unless it is done in a loop, which tests the success of the operation.

Implementing locks does not need this kind of loops, which may greatly increase the overhead, but only loops that do simple loads, for detecting changes, or the invocation of a FUTEX_WAIT, which is equivalent with that.

Besides loops that wait for changes, any kind of lock may be implemented with atomic read-modify-write instructions (e.g. on x86 XCHG, LOCK XADD, LOCK BTS and so on, and equivalent instructions on Armv8.1-A or later ISAs) that are not used in loops, so they have predictable overhead. For example, a futex may be used by a thread that waits for multiple events, if the other threads use a locked bit-test-and-set on the futex variable to signal the occurrence of an event, where each event is assigned to a distinct bit.

CMPXCHG and the equivalent load-and-lock/store conditional are really needed far less often than some people use them. The culprit is a widely-quoted research paper that has shown that these instructions are more universal than simple atomic fetch-and-operation instructions, allowing the implementation of lock-free algorithms, but the fact that they can do more does not mean that they should also be used when their extra power is not necessary, because that is paid dearly by introducing non-deterministic overhead.

A simple atomic instruction has an overhead much greater than an access to the L1 cache or the L2 cache, but typically the overhead is similar to that of a simple access to the L3 cache and significantly lower than the overhead of a simple access to the main memory, which remains the most expensive operation in modern CPUs.

Moreover, while mutual exclusion can be implemented reasonably efficiently with locks, it is also used far more often than necessary. It is possible to implement shared buffers or message queues that use neither mutual exclusion nor optimistic access that may need to be retried (a.k.a. lock-free access), but instead of those they use dynamic partitioning of the shared resource, allowing concurrent accesses without interference.

Organizing the cooperation between threads around shared buffers/message queues is frequently much better than using mutual exclusion, which stalls all contending threads, serializing their execution, and also much better than lock-free access, which may need an unpredictable number of retries when contention is high.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#157

Post by menaerus » Mon, Apr 06, 2026, 9:12 AM UTC

devchix wrote: post #148

adrian_b wrote: post #70

Earlier quoted context omitted.

Yes, and in the following messages the conclusion was that the regression is mitigated when using huge pages.

This seems bad, Splunk advises you to turn off THP due its small read/write characteristics: https://help.splunk.com/en/splunk-enterprise/release-notes-a... Bad because as of Splunk 10.x, Splunk bundles postgres to integrate with their SOAR platform. Parenthetically, this practice of bundling stuff with Splunk is making vuln remediation a real pain. Splunk bundles its own python, mongod, and now postgres, instead of…

Huge pages and THP are not the same thing.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#158

Post by menaerus » Mon, Apr 06, 2026, 9:23 AM UTC

db48x wrote: post #154

scottlamb wrote: post #152

Earlier quoted context omitted.

> The number of people at Amazon is pretty much irrelevant; the org is going to ensure that someone is keeping an eye on kernel performance, but also that the work isn’t duplicative. I'd guess they have dozens of people across say a Linux kernel team, a Graviton hardware integration team, an EC2 team, and a Amazon RDS for PostgreSQL team who might at one point or another run a benchmark like this. They probably coord…

True; to err is human. But it is weird that they didn’t just fire up a standard RDS instance of one or more sizes and test those. After all, it’s already automated; two clicks on the website gets you a standard configuration and a couple more get you a 96c graviton cpu. I just wonder how the mistake happened.

You're assuming that they ran the workload with huge-pages disabled unintentionally.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#159

Post by menaerus » Mon, Apr 06, 2026, 9:25 AM UTC

anarazel wrote: post #118

anarazel wrote: post #113

Earlier quoted context omitted.

I really dislike the use of spinlocks in postgres (and have been replacing a lot of uses over time), but it's not always easy to replace them from a performance angle. On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Turns out that that increase in memory barriers causes regr…

Addendum big enough to warrant a separate post: The fact the contention is a spinlock, rather than a futex is unrelated to the "regression". A quick hack shows the contended performance to be nearly indistinguishable with a futex based lock. Which makes sense, non-PI futexes don't transfer the scheduler slice the lock owner, because they don't know who the lock owner is. Postgres' spinlock use randomized exponential…

Contention doesn't exist in older kernel versions even with huge-pages disabled, no?

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#160

Post by db48x » Mon, Apr 06, 2026, 12:06 PM UTC

menaerus wrote: post #158

db48x wrote: post #154

Earlier quoted context omitted.

True; to err is human. But it is weird that they didn’t just fire up a standard RDS instance of one or more sizes and test those. After all, it’s already automated; two clicks on the website gets you a standard configuration and a couple more get you a 96c graviton cpu. I just wonder how the mistake happened.

You're assuming that they ran the workload with huge-pages disabled unintentionally.

No… I’m assuming that they didn’t use the same automation that creates RDS clusters for actual customers. No doubt that automation configures the EC2 nodes sanely, with hugepages turned on. Leaving them turned off in this benchmark could have been accidental, but some accident of that kind was bound to happen as soon as the tests use any kind of setup that is different from what customers actually get.