Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

phoronix.com

Post reply on HN
Print view View original

141–150 of 177 posts

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#141

Post by amluto »

anarazel wrote: post #132
amluto wrote: post #127

Earlier quoted context omitted.

> On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Now you've gotten me wondering. This issue is, in some sense, artificial: the actual conceptual futex unlock operation does not require sequential consistency. What's needed is (roughly, anyway) an release operation that sync…

> > On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). > Now you've gotten me wondering. This issue is, in some sense, artificial: the actual conceptual futex unlock operation does not require sequential consistency. What's needed is (roughly, anyway) an release operation that…

> I suspect the problem isn't so much the lock prefix, but that the non-futex spinlock release just is a store, whereas a futex release has to be a RMW operation.

> I'm talking out of my ass here, but my guess is that the reason for the performance gain of the plain-store-is-a-spinlock-release on x86 comes from being able to do the release via the store buffer, without having to wait for exclusive ownership of the cache line.

I don’t think so. The CPU is pretty good about hiding that kind of latency — reading a contended cache line and doing a correctly predicted branch shouldn’t stall anything after it.

But LOCK and MFENCE are quite expensive.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#142

Post by anarazel »

jcalvinowens wrote: post #136
anarazel wrote: post #113

Earlier quoted context omitted.

I really dislike the use of spinlocks in postgres (and have been replacing a lot of uses over time), but it's not always easy to replace them from a performance angle. On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Turns out that that increase in memory barriers causes regr…

That 64-bit atomic in the buffer head with flags, a spinlock, and refcounts all jammed into it is nasty . And there are like ten open coded spin waits around the uses... you certainly have my empathy :) This got me thinking about 64-bit futexes again. Obviously that can't work with PI... but for just FUTEX_WAIT/FUTEX_WAKE, why not? Somebody tried a long time ago, it got dropped but I didn't actually see any major obj…

> That 64-bit atomic in the buffer head with flags, a spinlock, and refcounts all jammed into it is nasty.

Turns out to be pretty crucial for performance though... Not manipulating them with a single atomic leads to way way worse performance.

For quite a while it was a 32bit atomic, but I recently made it a 64bit one, to allow the content lock (i.e. protecting the buffer contents, rather than the buffer header) to be in the same atomic var. That's for one nice for performance, it's e.g. very common to release a pin and a lock at the same time and there are more fun perf things we can do in the future. But the real motivation was work on adding support for async writes - an exclusive locker might need to consume an IO completion for a write that's in flight that is prevent it from acquiring the lock. And that was hard to do with a separate content lock and buffer state...

> And there are like ten open coded spin waits around the uses... you certainly have my empathy :)

Well, nearly all of those are all to avoid needing to hold a spinlock, which, as lamented a lot around this issue, don't perform that well when really contended :)

We're on our way to barely ever need the spinlock for the buffer header, which then should allow us to get rid of many of those loops.

> This got me thinking about 64-bit futexes again. Obviously that can't work with PI... but for just FUTEX_WAIT/FUTEX_WAKE, why not?

It'd be pretty nice to have. There are lot of cases where one needs more lock state than one can really encode into a 32bit lock state.

I'm quite keen to experiment with the rseq time slice extension stuff. Think it'll help with some important locks (which are not spinlocks...).

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#143

Post by jcalvinowens »

anarazel wrote: post #142
jcalvinowens wrote: post #136

Earlier quoted context omitted.

That 64-bit atomic in the buffer head with flags, a spinlock, and refcounts all jammed into it is nasty . And there are like ten open coded spin waits around the uses... you certainly have my empathy :) This got me thinking about 64-bit futexes again. Obviously that can't work with PI... but for just FUTEX_WAIT/FUTEX_WAKE, why not? Somebody tried a long time ago, it got dropped but I didn't actually see any major obj…

> That 64-bit atomic in the buffer head with flags, a spinlock, and refcounts all jammed into it is nasty. Turns out to be pretty crucial for performance though... Not manipulating them with a single atomic leads to way way worse performance. For quite a while it was a 32bit atomic, but I recently made it a 64bit one, to allow the content lock (i.e. protecting the buffer contents, rather than the buffer header) to be…

> Turns out to be pretty crucial for performance though...

I don't doubt it. I just meant nasty with respect to using futex() to sleep instead of spin, I was having some "fun" trying.

I can certainly see how pushing that state into one atomic would simplify things, I didn't really mean to question that.

> We're on our way to barely ever need the spinlock for the buffer header, which then should allow us to get rid of many of those loops.

I'm cheering you on, I hadn't looked at this code before and its been fun looking through some of the recent work on it.

> It'd be pretty nice to have. There are lot of cases where one needs more lock state than one can really encode into a 32bit lock state.

I've seen too much open coded spinning around 64-bit CAS in proprietary code, where it was a real demonstrable problem, and similar to here it was often not straightforward to avoid. I confess to some bias because of this experience ("not all spinlocks...") :)

I remember a lot of cases where FUTEX_WAIT64/FUTEX_WAKE64 would have been a drop-in solution, that seems compelling to me.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#144

Post by 999900000999 »

whilenot-dev wrote: post #54
999900000999 wrote: post #40

Earlier quoted context omitted.

Depends on your shop. As someone with a heavy QA/Dev Opps background I don't think we have enough details. Is it only ARM64 ? How many ARM64 PG DBs are running 96 cores? However... This is the most popular database in the world. Odds are this will effect a bunch of other lesser known applications.

Please follow the complete thread: https://lore.kernel.org/lkml/xxbnmxqhx4ntc4ztztllbhnral2adog... > [...] used huge_pages=on - as that is the only sane thing to do with 10s to 100s of GB of shared memory [...] if I disable huge pages, I actually can reproduce the contention [...]

Thank you for sharing.

So it looks like an edge case, as usually you need huge pages at scale ??

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#145

Post by rixed »

zbentley wrote: post #129
rixed wrote: post #59

Earlier quoted context omitted.

There is serious as in "corporate-serious" and serious as in "engineer-serious".

I’ve seen more 5k+-core fleets running Ubuntu in prod than not, in my career. Industries include healthcare, US government, US government contractor, marketing, finance.

In other words, those industries that used to run windows before ?

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#146

Post by mgaunard »

IshKebab wrote: post #62
mgaunard wrote: post #55

Earlier quoted context omitted.

Most people looking for performance will reach for the spinlock. The expectation is that the kernel should somehow detect applications that are spinning, and avoid preempting them early.

Well that seems like an unreasonable expectation no? Also isn't the point of spinlocks that they get released before the kernel does anything? Otherwise you could just use a futex... Which maybe you should do anyway... https://matklad.github.io/2020/01/04/mutexes-are-faster-than...

The scheduling is based on how much the LWP made use of its previous time slices. A spinning program clearly is using every cycle it's given without yielding, and so you can clearly tell preemption should be minimized.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#147

Post by mgaunard »

jcalvinowens wrote: post #139
silon42 wrote: post #137

Earlier quoted context omitted.

If you are spinning so long that it requires preemption, you're doing something wrong, no?

It doesn't matter, it's a long tail thing: on average user spinlocks can work, and even appear to be beneficial on benchmarks (for many reasons, Andy alludes to some above). But if you have enough users, some of them will experience the apocalyptic long tail, no matter what you do: that's why user spinlocks are unacceptable. RSEQ is the first real answer for this, but it's still not a guarantee: it is not possible to…

Well, you can always pin to a core and move other threads out of that core.

That's what you'd do if manually scheduling. Ideally the dynamic scheduler would do that on its own.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#148

Post by devchix »

adrian_b wrote: post #70
justinclift wrote: post #21

Earlier quoted context omitted.

Note that it's just not a single post, and there's additional further information in following the full thread. :)

Yes, and in the following messages the conclusion was that the regression is mitigated when using huge pages.

This seems bad, Splunk advises you to turn off THP due its small read/write characteristics: https://help.splunk.com/en/splunk-enterprise/release-notes-a...

Bad because as of Splunk 10.x, Splunk bundles postgres to integrate with their SOAR platform. Parenthetically, this practice of bundling stuff with Splunk is making vuln remediation a real pain. Splunk bundles its own python, mongod, and now postgres, instead of doing dependency checking. They're going to have to keep doing it as long as they release a .tgz and not just an RPM. The most recent postgres vuln is not fixed in Splunk.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#149

Post by jcalvinowens »

mgaunard wrote: post #147
jcalvinowens wrote: post #139

Earlier quoted context omitted.

It doesn't matter, it's a long tail thing: on average user spinlocks can work, and even appear to be beneficial on benchmarks (for many reasons, Andy alludes to some above). But if you have enough users, some of them will experience the apocalyptic long tail, no matter what you do: that's why user spinlocks are unacceptable. RSEQ is the first real answer for this, but it's still not a guarantee: it is not possible to…

Well, you can always pin to a core and move other threads out of that core. That's what you'd do if manually scheduling. Ideally the dynamic scheduler would do that on its own.

Sure. But if you squint even that isn't good enough, you'll still take interrupts on that core in the critical section sometimes when somebody else wants the lock.

The other problem with spin-wait is that it overshoots, especially with an increasing backoff. Part of the overhead of sleeping is paid back by being woken up immediately.

When it's made to work, the backoff is often "overfit" in that very slight random differences in kernel scheduler behavior can cause huge apparent regressions.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#150

Post by jeltz »

justinclift wrote: post #123
jeltz wrote: post #77

Earlier quoted context omitted.

Which you always should use anyway if you can.

Hmmm, it's not always that clear cut. For example, Redis officially advised people to disable it due to a latency impact: https://redis.io/docs/latest/operate/oss_and_stack/managemen... Pretty sure Redis even outputs a warning to the logs upon startup when it detects hugepages are enabled. Note that I'm not a Redis expert, I just remember this from when I ran it as a dependency for other software I was using.

1) That is about transparent huge pages which is a different thing and 2) it is always clear cut for PostgreSQL. If you can you should always use huge pages (the non-transparent kind).
Post reply on HN