Live data from Hacker News

AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

phoronix.com

121–130 of 177 posts

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#121
post #102

Earlier quoted context omitted.

Performance regressions are different from ABI incompatibilities. If the kernel refused to do any work that slowed down any userspace program, the pace would go a lot slower.

Or be a lot uglier. See: Microsoft replacing its own API surfaces with binary-compatible representations to workaround companies like Adobe adding perf improvements like bypassing the kernel-provided kernel object constructors because it saved them a few cycles to just hard-code the objects they wanted and memcpy them into existence.

Microsoft's whole "Let's just ship all the dlls" attitude is a big part of the reason a windows install is like 300GB now.

Eventually you'd expect that something has to give.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#122

Earlier quoted context omitted.

Because such settings aren’t obvious to those not familiar with them. LLMs should make discoverability easier though

Honest question: what's the value of running the benchmark and reporting a performance regression if the author is not familiar with basic operation of the software? I'd argue that not understanding those settings disqualifies you from making statements about it.

The performance was reduced without a settings change. That is still a regression even if huge pages mitigates the problem.

I'd be curious to know if there's still a regression with hugepages turned on in older kernels.

If you are benchmarking something and the only changed variable between benchmarks is the kernel, that is useful information. Even if your environment isn't correctly setup.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#123
post #77

Earlier quoted context omitted.

Yes, and in the following messages the conclusion was that the regression is mitigated when using huge pages.

Which you always should use anyway if you can.

Hmmm, it's not always that clear cut.

For example, Redis officially advised people to disable it due to a latency impact:

https://redis.io/docs/latest/operate/oss_and_stack/managemen...

Pretty sure Redis even outputs a warning to the logs upon startup when it detects hugepages are enabled.

Note that I'm not a Redis expert, I just remember this from when I ran it as a dependency for other software I was using.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#124

Earlier quoted context omitted.

Because such settings aren’t obvious to those not familiar with them. LLMs should make discoverability easier though

Honest question: what's the value of running the benchmark and reporting a performance regression if the author is not familiar with basic operation of the software? I'd argue that not understanding those settings disqualifies you from making statements about it.

Some software clearly wants hugepages disabled, so it's not always the slam dunk people seem to be making it out to be.

ie Redis:

https://redis.io/docs/latest/operate/oss_and_stack/managemen...

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#125

Earlier quoted context omitted.

The following up mails conclude that the regression happens only when huge pages are not used. While using huge pages whenever possible is the right solution and this should be enough for PostgreSQL, perhaps there are applications that cannot use huge pages and which are affected by the regression. So I do not think that it is right to just ignore what happened.

> While using huge pages whenever possible is the right solution and this should be enough for PostgreSQL, perhaps there are applications that cannot use huge pages and which are affected by the regression. It will be more interesting to talk about those applications if and when they are found. And I wouldn't assume the solutions are limited to reverting this change, starting to use the new spinlock time-slice extens…

> It will be more interesting to talk about those applications if and when they are found.

Redis recommend disabling hugepages: https://redis.io/docs/latest/operate/oss_and_stack/managemen...

---

Actually, looks like they changed the log warning to be more specific, as it's just the "always" setting which seems to cause Redis grief?

https://github.com/redis/redis/issues/3895

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#126

I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations.

> I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations. Yeah, exactly. "Doctor, help, somebody replaced my wooden hammer with a metal one, and now I can't hit myself in the face with it as many times." If you use spinlocks in userspace, you're gonna have a bad time.

[deleted]

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#127

I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations.

I really dislike the use of spinlocks in postgres (and have been replacing a lot of uses over time), but it's not always easy to replace them from a performance angle. On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Turns out that that increase in memory barriers causes regr…

> On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake).

Now you've gotten me wondering. This issue is, in some sense, artificial: the actual conceptual futex unlock operation does not require sequential consistency. What's needed is (roughly, anyway) an release operation that synchronizes with whoever subsequently acquires the lock (on x86, any non-WC store is sufficient) along with a promise that the kernel will get notified eventually (and preferably fairly quickly) if there was a non-spinning sleeper. But there is no requirement that the notification occur in any particular order wrt anything else except that the unlock must be visible by the time the notification occurs [0]; there isn't even a requirement that the notification not occur if there is no futex waiter.

I think that, in common cache coherence protocols, this is kind of straightforward -- the unlock is a store-release, and as long as the cache line ends up being written locally, the hardware or ucode or whatever simply [1] needs to check whether a needs-notification flag is set in the same cacheline. Or the futex-wait operation needs to do a super-heavyweight barrier to synchronize with the releasing thread even though the releasing thread does not otherwise have any barrier that would do the job.

One nasty approach that might work is to use something like membarrier, but I'm guessing that membarrier is so outrageously expensive that this would be a huge performance loss.

But maybe there are sneaky tricks. I'm wondering whether CMPXCHG (no lock) is secretly good enough for this. Imagine a lock word where bit 0 set means locked and bit 1 set means that there is a waiter. The wait operation observes (via plain MOV?) that bit 0 is set and then sets bit 1 (let's say this is done with LOCK CMPXCHG for simplicity) and then calls futex_wait(), so it thinks the lock word has the value 3. The unlock operation does plain CMPXCHG to release the lock. The failure case would be that it reports success while changing the value from 1 to 0. I don't know whether this can happen on Intel or AMD architectures.

I do expect that it would be nearly impossible to convince an x86 CPU vendor to commit to an answer either way.

(Do other architectures, e.g. the most recent ARM variants, have an RMW release operation that naturally does this? I've tried, and entirely failed AFAICT, to convince x86 HW designers to add lighter weight atomics.)

[0] Visible to the remote thread, but the kernel can easily mediate this, effectively for free.

[1] Famous last words. At least in ossified microarchitectures, nothing is simple.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#129
post #59

Earlier quoted context omitted.

Ubuntu is used in many serious backend environments. Heroku runs tens of thousands (if not more) instances of Ubuntu on its fleet. Or at least it did through the teens and early 2020s. https://devcenter.heroku.com/articles/stack

There is serious as in "corporate-serious" and serious as in "engineer-serious".

I’ve seen more 5k+-core fleets running Ubuntu in prod than not, in my career. Industries include healthcare, US government, US government contractor, marketing, finance.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#130
post #96

Earlier quoted context omitted.

Yes, I had a good laugh at that. It might technically be a regression, but not one that most people will see in practice. Pretty weird that someone at Amazon is bothering to run those tests without hugepages.

I doubt they explicitly said "I'll run without huge pages, which is an important AWS configuration". They probably just forgot a step. And "someone at Amazon" describes a lot of people; multiply your mental probability tables accordingly.

The number of people at Amazon is pretty much irrelevant; the org is going to ensure that someone is keeping an eye on kernel performance, but also that the work isn’t duplicative.

Surely they would be testing the configuration(s) that they use in production? They’re not running RDS without hugepages turned on, right?

Post reply on HN