Live data from Hacker News

AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

phoronix.com

161–170 of 177 posts

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#161

Earlier quoted context omitted.

Addendum big enough to warrant a separate post: The fact the contention is a spinlock, rather than a futex is unrelated to the "regression". A quick hack shows the contended performance to be nearly indistinguishable with a futex based lock. Which makes sense, non-PI futexes don't transfer the scheduler slice the lock owner, because they don't know who the lock owner is. Postgres' spinlock use randomized exponential…

Contention doesn't exist in older kernel versions even with huge-pages disabled, no?

The contention does exist in older kernels and is quite substantial.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#162
post #127

Earlier quoted context omitted.

> On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Now you've gotten me wondering. This issue is, in some sense, artificial: the actual conceptual futex unlock operation does not require sequential consistency. What's needed is (roughly, anyway) an release operation that sync…

Using LOCK CMPXCHG or even plain CMPXCHG does not make sense unless it is done in a loop, which tests the success of the operation. Implementing locks does not need this kind of loops, which may greatly increase the overhead, but only loops that do simple loads, for detecting changes, or the invocation of a FUTEX_WAIT, which is equivalent with that. Besides loops that wait for changes, any kind of lock may be impleme…

You are misunderstanding me, which is perhaps understandable, since I’m talking about the minutiae of x86, not locking in general.

When unlocking a futex-backed mutex, one needs to do two things. First, one needs to actually unlock it: this is a store-release in modern lingo, and on x86 almost any store instruction has the correct ordering semantics. Second, one needs to determine whether to call futex_wake, which is conceptually just reading a flag “is someone waiting” and then branching on the result. The problem is that the load needs to be ordered after (or at least not before) the store.

x86 provides two main ways to do this, MFENCE and LOCK. For whatever reason, at least Intel has tried pretty hard to optimize LOCK, and it’s often the case that LOCKed operations on a hot cache line is faster than MFENCE. (I have benchmarked this, and Linux uses this trick.)

My point is that the specific algorithm of unlocking a futex-backed mutex does not require the full ordering semantics of MFENCE or LOCK. And my secondary observation is that x86 has some non-LOCKed RMW instructions, one of which is plain CMPXCHG. Unlocked CMPXCHG is much faster than LOCK anything or MFENCE — I’ve benchmarked it. There are also the flag outputs from operations like ADD. And I’m speculating that maybe some of these instructions are secretly actually ordered strongly enough for futex unlock.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#163

Earlier quoted context omitted.

Contention doesn't exist in older kernel versions even with huge-pages disabled, no?

The contention does exist in older kernels and is quite substantial.

You said

> Maybe we should, but requiring the use of a new low level facility that was introduced in the 7.0 kernel, to address a regression that exists only in 7.0+, seems not great.

... so that leaves me confused. My understanding is that the regression is triggered with the 7.0+ kernel and can be mitigated with huge pages turned on.

My question therefore was how come this regression hasn't been visible with huge pages turned off with older kernel versions? You say that it was but I can't find this data point.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#164
post #160

Earlier quoted context omitted.

You're assuming that they ran the workload with huge-pages disabled unintentionally.

No… I’m assuming that they didn’t use the same automation that creates RDS clusters for actual customers. No doubt that automation configures the EC2 nodes sanely, with hugepages turned on. Leaving them turned off in this benchmark could have been accidental, but some accident of that kind was bound to happen as soon as the tests use any kind of setup that is different from what customers actually get.

You're again assuming that having huge pages turned on always brings the net benefit, which it doesn't. I have at least one example where it didn't bring any observable benefit while at the same time it incurred extra code complexity, server administration overhead, and necessitated extra documentation.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#165

Earlier quoted context omitted.

The contention does exist in older kernels and is quite substantial.

You said > Maybe we should, but requiring the use of a new low level facility that was introduced in the 7.0 kernel, to address a regression that exists only in 7.0+, seems not great. ... so that leaves me confused. My understanding is that the regression is triggered with the 7.0+ kernel and can be mitigated with huge pages turned on. My question therefore was how come this regression hasn't been visible with huge p…

> ... so that leaves me confused. My understanding is that the regression is triggered with the 7.0+ kernel and can be mitigated with huge pages turned on.

It gets a bit worse with preempt_lazy - for me just 15% percent or so - because the lock holder is scheduled out a bit more often. But it was bad before.

> My question therefore was how come this regression hasn't been visible with huge pages turned off with older kernel versions? You say that it was but I can't find this data point.

I mean it wasn't a regression before, because this is how it has behaved for a long time.

This workload is not a realistic thing that anybody would encounter in this form in the real world. Even without the contention - which only happens the first time the buffer pool is filled - you lose so much by not using huge pages with a 100gb buffer pool that you will have many other issues.

We (postgres and me personally) were concerned enough about potential contention in this path that we did get rid of that lock half a year ago (buffer replacement selection has been lock free for close to a decade, just unused buffers were found via a list protected by this lock).

But the performance gains we saw were relatively small, we didn't measure large buffer pools without huge pages though.

And at least I didn't test with this many connections doing small random reads into a cold buffer pool, just because it doesn't seem that interesting.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#166
post #145

Earlier quoted context omitted.

I’ve seen more 5k+-core fleets running Ubuntu in prod than not, in my career. Industries include healthcare, US government, US government contractor, marketing, finance.

In other words, those industries that used to run windows before ?

I'd say about 2/3 of the places I've worked started on Linux without a Windows precedent other than workstations. I can't speak for the experience of the founding staff, though; they might have preferred Ubuntu due to Windows experience--if so, I'm curious as to why/what those have to do with each other.

That said, Ubuntu in large production fleets isn't too bad. Sure, other distros are better, but Ubuntu's perfectly serviceable in that role. It needs talented SRE staff making sure automation, release engineering, monitoring, and de/provisioning behave well, but that's true of any you-run-the-underlying-VM large cloud deployment.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#167

Earlier quoted context omitted.

Hmmm, it's not always that clear cut. For example, Redis officially advised people to disable it due to a latency impact: https://redis.io/docs/latest/operate/oss_and_stack/managemen... Pretty sure Redis even outputs a warning to the logs upon startup when it detects hugepages are enabled. Note that I'm not a Redis expert, I just remember this from when I ran it as a dependency for other software I was using.

That's transparent huge pages, which are also not the setting recommended for PostgreSQL.

Java can work with transparent hugepages (in addition to preallocated hugepages), but you just use +AlwaysPreTouch to map them in during the startup so that at runtime there won't be any delays or jitter. Redis should add a similar option

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#169
post #102

Earlier quoted context omitted.

I for one liked the old and simple WE DO NOT BREAK USERSPACE attitude. https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE

Performance regressions are different from ABI incompatibilities. If the kernel refused to do any work that slowed down any userspace program, the pace would go a lot slower.

Slow pace is appropriate for a mature kernel that the entire world relies on.

Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy

#170
post #12

Earlier quoted context omitted.

I’m absolutely flabbergasted by the performance left on the table; even by myself - just yesterday I learned Gentoo’s emerge can use git and be a billion times faster.

The time spent by emerge is utterly dwarfed by the time spent to build the packages, so who cares? Maybe it's different if installing a binary system but don't think most people are doing that.

That's really not universally true. Building can be parallelized on modern multi-code CPUs (minus configure), emerge cannot and portage is really really slow.
Post reply on HN