Earlier quoted context omitted.
Addendum big enough to warrant a separate post: The fact the contention is a spinlock, rather than a futex is unrelated to the "regression". A quick hack shows the contended performance to be nearly indistinguishable with a futex based lock. Which makes sense, non-PI futexes don't transfer the scheduler slice the lock owner, because they don't know who the lock owner is. Postgres' spinlock use randomized exponential…
Contention doesn't exist in older kernel versions even with huge-pages disabled, no?
AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
161–170 of 177 posts
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#162Earlier quoted context omitted.
> On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Now you've gotten me wondering. This issue is, in some sense, artificial: the actual conceptual futex unlock operation does not require sequential consistency. What's needed is (roughly, anyway) an release operation that sync…
Using LOCK CMPXCHG or even plain CMPXCHG does not make sense unless it is done in a loop, which tests the success of the operation. Implementing locks does not need this kind of loops, which may greatly increase the overhead, but only loops that do simple loads, for detecting changes, or the invocation of a FUTEX_WAIT, which is equivalent with that. Besides loops that wait for changes, any kind of lock may be impleme…
When unlocking a futex-backed mutex, one needs to do two things. First, one needs to actually unlock it: this is a store-release in modern lingo, and on x86 almost any store instruction has the correct ordering semantics. Second, one needs to determine whether to call futex_wake, which is conceptually just reading a flag “is someone waiting” and then branching on the result. The problem is that the load needs to be ordered after (or at least not before) the store.
x86 provides two main ways to do this, MFENCE and LOCK. For whatever reason, at least Intel has tried pretty hard to optimize LOCK, and it’s often the case that LOCKed operations on a hot cache line is faster than MFENCE. (I have benchmarked this, and Linux uses this trick.)
My point is that the specific algorithm of unlocking a futex-backed mutex does not require the full ordering semantics of MFENCE or LOCK. And my secondary observation is that x86 has some non-LOCKed RMW instructions, one of which is plain CMPXCHG. Unlocked CMPXCHG is much faster than LOCK anything or MFENCE — I’ve benchmarked it. There are also the flag outputs from operations like ADD. And I’m speculating that maybe some of these instructions are secretly actually ordered strongly enough for futex unlock.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#163Earlier quoted context omitted.
Contention doesn't exist in older kernel versions even with huge-pages disabled, no?
The contention does exist in older kernels and is quite substantial.
> Maybe we should, but requiring the use of a new low level facility that was introduced in the 7.0 kernel, to address a regression that exists only in 7.0+, seems not great.
... so that leaves me confused. My understanding is that the regression is triggered with the 7.0+ kernel and can be mitigated with huge pages turned on.
My question therefore was how come this regression hasn't been visible with huge pages turned off with older kernel versions? You say that it was but I can't find this data point.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#164Earlier quoted context omitted.
You're assuming that they ran the workload with huge-pages disabled unintentionally.
No… I’m assuming that they didn’t use the same automation that creates RDS clusters for actual customers. No doubt that automation configures the EC2 nodes sanely, with hugepages turned on. Leaving them turned off in this benchmark could have been accidental, but some accident of that kind was bound to happen as soon as the tests use any kind of setup that is different from what customers actually get.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#165Earlier quoted context omitted.
The contention does exist in older kernels and is quite substantial.
You said > Maybe we should, but requiring the use of a new low level facility that was introduced in the 7.0 kernel, to address a regression that exists only in 7.0+, seems not great. ... so that leaves me confused. My understanding is that the regression is triggered with the 7.0+ kernel and can be mitigated with huge pages turned on. My question therefore was how come this regression hasn't been visible with huge p…
It gets a bit worse with preempt_lazy - for me just 15% percent or so - because the lock holder is scheduled out a bit more often. But it was bad before.
> My question therefore was how come this regression hasn't been visible with huge pages turned off with older kernel versions? You say that it was but I can't find this data point.
I mean it wasn't a regression before, because this is how it has behaved for a long time.
This workload is not a realistic thing that anybody would encounter in this form in the real world. Even without the contention - which only happens the first time the buffer pool is filled - you lose so much by not using huge pages with a 100gb buffer pool that you will have many other issues.
We (postgres and me personally) were concerned enough about potential contention in this path that we did get rid of that lock half a year ago (buffer replacement selection has been lock free for close to a decade, just unused buffers were found via a list protected by this lock).
But the performance gains we saw were relatively small, we didn't measure large buffer pools without huge pages though.
And at least I didn't test with this many connections doing small random reads into a cold buffer pool, just because it doesn't seem that interesting.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#166Earlier quoted context omitted.
I’ve seen more 5k+-core fleets running Ubuntu in prod than not, in my career. Industries include healthcare, US government, US government contractor, marketing, finance.
In other words, those industries that used to run windows before ?
That said, Ubuntu in large production fleets isn't too bad. Sure, other distros are better, but Ubuntu's perfectly serviceable in that role. It needs talented SRE staff making sure automation, release engineering, monitoring, and de/provisioning behave well, but that's true of any you-run-the-underlying-VM large cloud deployment.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#167Earlier quoted context omitted.
Hmmm, it's not always that clear cut. For example, Redis officially advised people to disable it due to a latency impact: https://redis.io/docs/latest/operate/oss_and_stack/managemen... Pretty sure Redis even outputs a warning to the logs upon startup when it detects hugepages are enabled. Note that I'm not a Redis expert, I just remember this from when I ran it as a dependency for other software I was using.
That's transparent huge pages, which are also not the setting recommended for PostgreSQL.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#168Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#169Earlier quoted context omitted.
I for one liked the old and simple WE DO NOT BREAK USERSPACE attitude. https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE
Performance regressions are different from ABI incompatibilities. If the kernel refused to do any work that slowed down any userspace program, the pace would go a lot slower.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#170Earlier quoted context omitted.
I’m absolutely flabbergasted by the performance left on the table; even by myself - just yesterday I learned Gentoo’s emerge can use git and be a billion times faster.
The time spent by emerge is utterly dwarfed by the time spent to build the packages, so who cares? Maybe it's different if installing a binary system but don't think most people are doing that.