Can someone explain to me what's the problem? I have very little knowledge of Linux kernel, but I'm curious. I've tried reading a little, but it's jargon over jargon.
AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
51–60 of 177 posts
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#52Anyone check to see if Jia Tan has submitted any kernel patches lately?
They don't need to, there's about a billion bugs they can exploit.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#53Not sure why people have to upgrade to the newest major kernel version as soon as it is released.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#54Earlier quoted context omitted.
> Nobody sensible runs the latest kernel From the article: "Linux 7.0 stable is due out in about two weeks. This is also the kernel version powering Ubuntu 26.04 LTS to be released later in April." Unfortunately, lots of people will be running it in less than a month. At the moment, it'll take a kernel patch (not a sysctl) to undo this-- hopefully something changes.
Depends on your shop. As someone with a heavy QA/Dev Opps background I don't think we have enough details. Is it only ARM64 ? How many ARM64 PG DBs are running 96 cores? However... This is the most popular database in the world. Odds are this will effect a bunch of other lesser known applications.
Please follow the complete thread: https://lore.kernel.org/lkml/xxbnmxqhx4ntc4ztztllbhnral2adog...
> [...] used huge_pages=on - as that is the only sane thing to do with 10s to 100s of GB of shared memory [...] if I disable huge pages, I actually can reproduce the contention [...]
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#55I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations.
> I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations. Yeah, exactly. "Doctor, help, somebody replaced my wooden hammer with a metal one, and now I can't hit myself in the face with it as many times." If you use spinlocks in userspace, you're gonna have a bad time.
Most people looking for performance will reach for the spinlock.
The expectation is that the kernel should somehow detect applications that are spinning, and avoid preempting them early.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#56Nobody sensible runs the latest kernel; nobody running PG in production should be afraid of setting a non-default at either boot time or as a sysctl. So this will, most likely, be another step in building a PG database server (turn off pre-emption if your kernel is 7.0 or later and PG is pre-whatever-version). At worst it might become a permanent part of building a PG server and a FAQ... but if it affects one thing t…
[dead]
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#57Earlier quoted context omitted.
Not historically.
and they are right, this is because a lot of junior sysadmins believe that newer = better. But the reality: a) may get irreversible upgrades (e.g. new underlying database structure) b) permanent worse performance / regression (e.g. iOS 26) c) added instability d) new security issues (litellm) e) time wasted migrating / debugging f) may need rewrite of consumers / users of APIs / sys calls g) potential new IP or licen…
On the other hand, I suspect LLMs will dramatically decrease the window between a vulnerability being discovered and that vulnerability being exploited in the wild, especially for open-source projects.
Even if the vulnerability itself is discovered through other means than by an LLM, it's trivial to ask a SOTA model to "monitor all new commits to project X and decide which ones are likely patching an exploitable vulnerability, and then write a PoC." That's a lot easier than finding the vulnerable itself.
I won't be surprised if update windows (for open source networked services) shrink to ~10 minutes within a year or two. It's going to be a brutal world.
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#58Not sure why people have to upgrade to the newest major kernel version as soon as it is released.
It's the performance team's job to test these things. Doesn't mean they're going to deploy it immediately.
Someone should be testing these things and reporting regressions
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#59Earlier quoted context omitted.
For production Postgres, i would assume it’s close to almost no effect? If someone is running postgres in a serious backend environment, i doubt they are using Ubuntu or even touching 7.x for months (or years). It’ll be some flavor of Debian or Red Hat still on 6.x (maybe even 5?). Those same users won’t touch 7.x until there has been months of testing by distros.
Ubuntu is used in many serious backend environments. Heroku runs tens of thousands (if not more) instances of Ubuntu on its fleet. Or at least it did through the teens and early 2020s. https://devcenter.heroku.com/articles/stack
There is serious as in "corporate-serious" and serious as in "engineer-serious".
Re: AWS engineer reports PostgreSQL perf halved by Linux 7.0, fix may not be easy
#60Once again phoronix shoot out an article without further researching nor letting the mail thread in question cool down.
The follow up mails make clear that the issue is more or less a non-issue since the benchmark is wrong.