Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

thehftguy.com

Post reply on HN
Print view View original

41–50 of 58 posts

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#41

Post by masklinn »

throwbadubadu wrote: post #8
blkhawk wrote: post #4

Earlier quoted context omitted.

If i skimmed this correctly then its a malus on performance not a complete cliff. I guess people just thought "hohum - there gotta be some overhead in scheduling".

Yes, when you run in parallel, and e.g. see all >8 cores nicely nagging up to 100% why assume something wrong? Still don't get after rereading the article, what is the malus, it must be small by that? Because you definitely saw linear scaling with parallelizable problems on >8 cores, otherwise people would have noticed?

What happens is that the min_slice stops scaling up above 8 cores. The article misrepresents that as “limit to 8 cores”, but my admittedly shallow understanding is that min_slice is a preemption protection: if the kernel has tasks to schedule and no free core, it will try to preempt an existing task, a process within its min_slice is protected from that preemption.

So this is only relevant for an overloaded system, and furthermore just means that processes may be preempted after 3ms (instead of that protection delay keeping on increasing), ignoring all other tunables e.g. priority and stuff.

Not only that but it’s a log2, so if this was relaxed on a 128 cores system you’d have a preemption delay of 7ms instead. I don’t think that would save you if you’re overloading a 128 cores system honestly, although it does beg the question as to why the kernel devs felt the need to cap the scaling. Even assuming it scales per thread, and you have a dual socket epyc, log2(512) = 9.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#42

Post by nathants »

archi42 wrote: post #37
nathants wrote: post #31

Earlier quoted context omitted.

yeah? a xeon with 16 cores might have to be my next pc. pricey though.

You can try to make your code aware of the situation and distribute the tasks accordingly. IIRC that's what people do on NUMA systems.

afaik not for my use case. i need low latency with no variability. you only get that staying within a single cpu cluster.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#43

Post by masklinn »

globular-toast wrote: post #39

I used to work on a supercomputer with 128 cores on it that ran Linux (I actually seem to remember it had 256 cores, but someone said the kernel had a limit of 128). This was less than 15 years ago. There were surely many systems just like that one. Does that mean the kernel had been patched? But nobody thought to push that patch upstream? Reading the other comments here it seems the title is stupid and wrong and my…

There’s probably nothing to patch: from my understanding this does not actually affect the scaling of the OS, it affects the throughput on heavily loaded systems, as min_slice is the delay before which a task can’t be preempted.

So it’s only relevant if the system has more tasks than cores, and if you ignore priorities and pinning. I assume these are the sort of mistakes people working with supercomputers would not be making, and residency would be a very carefully monitored to ensure the system is not thrashing.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#44

Post by jamesnorden »

raverbashing wrote: post #14

Wow big oooff One thing I noticed is that min/max functions are tricky to use and it's easily to accidentally do the wrong thing, like in this case Because you think "you want the minimum (that is, the number should be at least 8) of those numbers to be 8" then you slap min(). And you got it wrong. You should have used max()

It's not wrong, the cap is 8, if it was max() it would use 8 on a single core system.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#45

Post by Fnoord »

0xDEADFED5 wrote: post #20

i'm curious if this matters a whole lot. definitely not my area, but it would be interesting to see a benchmark that shows this is a problem.

I'm also curious about the impact. Perhaps we'll see a meaningful Phoronix test on this one?

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#46

Post by archi42 »

nathants wrote: post #42
archi42 wrote: post #37

Earlier quoted context omitted.

You can try to make your code aware of the situation and distribute the tasks accordingly. IIRC that's what people do on NUMA systems.

afaik not for my use case. i need low latency with no variability. you only get that staying within a single cpu cluster.

It's not possible for every workload, and sometimes the necessary effort makes it infeasible. Getting a Xeon might be the cheaper option then ;-)

Edit: Though I'd still recommend heeding menaerus excellent sibling answer. Maybe not for this project, but it is great knowledge to have in your domain and I'd expect it to be relevant for the future.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#47

Post by masklinn »

PhilipRoman wrote: post #40
raverbashing wrote: post #14

Wow big oooff One thing I noticed is that min/max functions are tricky to use and it's easily to accidentally do the wrong thing, like in this case Because you think "you want the minimum (that is, the number should be at least 8) of those numbers to be 8" then you slap min(). And you got it wrong. You should have used max()

I don't think that's the case here, but yeah min and max can be a bit confusing to read. If your language allows adding methods or infix operators without performance overhead, I like to make something like this: x.atLeast(10).atMost(100) Very easy to read in my opinion

I find the word “clamp” extremely satisfying for this concept, especially with optional named parameters in order to leave off one bound e.g. `x.clamp(low=10)` but it still works fine with both required and anonymous, it’s just a bit less convenient.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#48

Post by menaerus »

nathants wrote: post #42
archi42 wrote: post #37

Earlier quoted context omitted.

You can try to make your code aware of the situation and distribute the tasks accordingly. IIRC that's what people do on NUMA systems.

afaik not for my use case. i need low latency with no variability. you only get that staying within a single cpu cluster.

This all comes down to the cache coherency protocols. And it's not surprising that you see increased latency with zen4 microarchitecture because, as one of the parent commenters already said, it's almost as if you're running a NUMA architecture within a single physical chip.

When dealing with NUMA we know that cross-socket, or in this case cross-CCD, latencies are always higher than the ones within the same socket or same CCD. Usually multi-fold.

This article nicely lists the core-to-core latencies between the Intel and AMD microarchitectures: https://chipsandcheese.com/2023/07/17/genoa-x-server-v-cache...

So, if you're able to somehow take advantage of this knowledge in your code (e.g. by scheduling less latency-sensitive tasks to the other CCD), you may be able to improve your overall performance.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#49

Post by nottheengineer »

piggycurse wrote: post #18

> It’s problematic that the kernel was hardcoded to a maximum of 8 cores Why? The article show no evidence of this being problematic. > It can’t be good to reschedule hundreds of tasks every few milliseconds, maybe on a different core, maybe on a different die. It can’t be good for performance and cache locality. Make a PoC and prove it.

I'm pretty sure schedulers are just black magic and anyone who doesn't work directly with them is unqualified to make statements like these.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#50

Post by throwbadubadu »

masklinn wrote: post #41
throwbadubadu wrote: post #8

Earlier quoted context omitted.

Yes, when you run in parallel, and e.g. see all >8 cores nicely nagging up to 100% why assume something wrong? Still don't get after rereading the article, what is the malus, it must be small by that? Because you definitely saw linear scaling with parallelizable problems on >8 cores, otherwise people would have noticed?

What happens is that the min_slice stops scaling up above 8 cores. The article misrepresents that as “limit to 8 cores”, but my admittedly shallow understanding is that min_slice is a preemption protection: if the kernel has tasks to schedule and no free core, it will try to preempt an existing task, a process within its min_slice is protected from that preemption. So this is only relevant for an overloaded system, a…

Ok, thanks a lot. Agreed, one might even argue if it is better to scale this number with the amount of cores, or in what relationship, or if the initial values chosen are the optimal ones, certainly not for every situation. With the right comment one might have even claimed it intentional, lol.

Clickbaity headline from a technical person makes me sad :(

Post reply on HN