Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

read.thecoder.cafe

Post reply on HN
Print view View original

31–40 of 73 posts

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#31

Post by jeltz »

Twirrim wrote: post #7
baq wrote: post #4

TLDR of the LMKL thread: 120GB RAM postgres with hugepages=off, lock contention went from terrible to abysmal. nothing to see here except that amazon for whatever reason runs DB tests with huge pages disabled. (hope I'm not paying for RDS and auroras like that in production!)

Huge pages has had a spotty history, that lead to people being paranoid about it, and no doubt a whole bunch of folks just disable it "because that's what we've always done". It has been stable and reliable for quite a while now, would really hope folks could move away from that perspective.

Are you sure you are not thinking of transparent huge pages? They have a spotty history but you are supposed to run big PostgreSQL instances with huge pages, not transparent huge pages.

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#32

Post by lstodd »

Twirrim wrote: post #7
baq wrote: post #4

TLDR of the LMKL thread: 120GB RAM postgres with hugepages=off, lock contention went from terrible to abysmal. nothing to see here except that amazon for whatever reason runs DB tests with huge pages disabled. (hope I'm not paying for RDS and auroras like that in production!)

Huge pages has had a spotty history, that lead to people being paranoid about it, and no doubt a whole bunch of folks just disable it "because that's what we've always done". It has been stable and reliable for quite a while now, would really hope folks could move away from that perspective.

I remember when support for them just appeared and you had to LD_PRELOAD a shim IIRC to make Postgres actually use them we jumped on it, enabled them immediately and got a pretty significant boost, around 15-20%, yes.

That was idk, 2008-9 -ish? I don't know what spotty history you are talking about, if you have multigigabyte address spaces floating on a machine it's stupid not to use hugepages.

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#33

Post by slackfan »

nobleach wrote: post #6
PunchyHamster wrote: post #3

Seems Linus needs to yell at someone again. Especially with containers around you might very well hit the case of running new kernel but older version of PostgreSQL with no code mitigation for the problem

I get that folks love a good Linus rant. But as someone who's been at the end of that style of "feedback", nothing can be more humiliating or demotivating. Certainly there are contributors that are making "rookie mistakes". There are folks that aren't willing to ingest the entire context of what was tried back in 2.0.36, 2.2, 2.4... etc. And perhaps it's wise to simply stay away until you're completely certain you've…

Code quality does not care about your feelings.

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#34

Post by matharmin »

nijave wrote: post #10

Right on the heels of 6.19 breaking tcmalloc and Mongo

Yup - interesting to see so much written about Postgres having a performance regression on Linux 7.0, in a scenario that affects almost no-one in practice. Meanwhile MongoDB refuses to run at all on Linux 7.0 due to some issue with tcmalloc.

https://jira.mongodb.org/browse/SERVER-121885

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#35

Post by andrewstuart »

ozgrakkurt wrote: post #20

It is a crime that postgres isn't able to allocate with 1GB huge pages by changing a config parameter in 2026 Also a crime that people are still running databases with 4kb pages. To put it in perspective, this means you will have more than 30 million pages on a server with 128GB RAM. As an example, if there is 16bytes of metadata for memory page. The metadata itself would take more than half a gigabyte.

Sensible defaults would be nice.

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#36

Post by fulafel »

> PREEMPT_NONE: The kernel almost never interrupts a running thread

This seems confused. These are options for preemptibility of the kernel, which is a relatively modern fearure. Userspace could always be preempted and these options do not change anything there. The kernel must in any case frequently interrupt threads and processes to implement preemptive multitasking which Linux of course had since the beginning.

Read more eg at https://lwn.net/Articles/944686/ or help texts at https://github.com/torvalds/linux/blob/master/kernel/Kconfig...

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#37

Post by teivah »

ameliaquining wrote: post #19

This post comes uncomfortably close to plagiarizing https://thebuild.com/blog/2026/04/23/preempt_none-is-dead-yo... , which it cites as a source; almost all the technical explanation is in there and some of the wording is extremely similar. Compare, e.g., "What Linux 7.0 actually changed" in Pettus's post to "What Is Preemption?" in this one. I think this link should have been to Pettus's post instead.

I used that post as a source yes, and it’s stated explicitly but it’s not the only one. One section in particular is similar since we both present the different preemption modes. However, both audiences are different. thebuild.com has an audience composed of PostgreSQL enthusiasts (if not experts) and I don’t. So a significant part of my post was about explaining things from first principles (what’s a page, a TLB, a spinlock, etc.). I explain way more “basic” things and he goes beyond me in terms of how to cope with the problem. I don’t think the posts are closed.

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#38

Post by AlienRobot »

ahartmetz wrote: post #22

PREEMPT_LAZY triggering on page faults seems like a bad idea in light of this. It is probably not a good idea to suspend processes right when they get unexpectedly bogged down. The logic makes a little more sense for syscalls that are expected to take long compared to a scheduling quantum (a few milliseconds). But page faults are mostly invisible and unplannable. It only took a few decades for Linux to get a good CPU…

One thing I miss from using Windows is that the desktop didn't just freeze completely if you ran out of RAM.

At first I thought that maybe Linux doesn't have ways to give priority to the desktop environment (a.k.a. "graphical shell") which is why running out of RAM means your cursor starts lagging, clicking on things stops working, etc.

But maybe Linux is just bad at that in general and a single process eating too much RAM can simply bring the whole system to a halt as it tries to move and compress RAM to a pagefile on an HDD (not SSD).

Every time it happens to me I just find it so incredible. Here I am with a PC with a multiple cores, multiple processors, and a single process eating all the RAM can bottleneck ALL of them at once? Am I misunderstanding something? Shouldn't it, ideally, work in such way that so long as one processor is free, the system can process mouse input and render the cursor and do all the desktop stuff no matter what I/O is happening in the background?

Since it's Linux maybe it's just my DE/distro (Cinnamon/Mint). Maybe it does allocations under the assumption there will always be a few free bytes in RAM available, so it halts if RAM runs out while some other DE wouldn't. But even then you'd think there would be a way to just reserve "premium" memory for critical processes so they never become unresponsive.

I wonder if other people have the same experience as me. This part of Linux just always felt fundamentally poor for me.

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#39

Post by fabian2k »

That regression is maybe most useful as a reminder to people to configure huge pages for PostgreSQL. That's the one recommended basic performance tuning that is just annoying enough to set up that I suspect many people with smaller DBs will skip it.

Though I actually don't know how large shared buffers has to be for huge pages to make a noticeable difference.

Re: Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained

#40

Post by jcgl »

AlienRobot wrote: post #38
ahartmetz wrote: post #22

PREEMPT_LAZY triggering on page faults seems like a bad idea in light of this. It is probably not a good idea to suspend processes right when they get unexpectedly bogged down. The logic makes a little more sense for syscalls that are expected to take long compared to a scheduling quantum (a few milliseconds). But page faults are mostly invisible and unplannable. It only took a few decades for Linux to get a good CPU…

One thing I miss from using Windows is that the desktop didn't just freeze completely if you ran out of RAM. At first I thought that maybe Linux doesn't have ways to give priority to the desktop environment (a.k.a. "graphical shell") which is why running out of RAM means your cursor starts lagging, clicking on things stops working, etc. But maybe Linux is just bad at that in general and a single process eating too mu…

Same experience here. Linux admin. I’d absolutely love to be told I’m holding it wrong, but all I can see is that there’s no way to hold it right.

Your consternation is seconded.

Post reply on HN