Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

thehftguy.com

Post reply on HN
Print view View original

31–40 of 58 posts

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#31

Post by nathants »

dannyw wrote: post #27
nathants wrote: post #16

this may not be an error. i’m doing high performance gamedev on a 7950x with smt off. 16 cores. i can only use 8 to run the game. mutex and other synchronization gets highly variable latency if the first 8 cores try to talk to the second 8. i hadn’t heard of this before i started this game, but apparently it’s well known, and nobody makes a chip with more than 8 cores that can have low variability synchronization. li…

You’re describing behaviour with AMD’s Infinity Fabric, which groups cores into CCDs of 8 on 7000 series chips. Essentially, a 16 core AMD CPU is actually two separate 8 core CPUs glued together with a link. Intel does not have this limitation.

yeah? a xeon with 16 cores might have to be my next pc. pricey though.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#32

Post by 8fingerlouie »

I'm not well enough versed in the Linux kernel to comment on the post, but as a funny anecdote, 15 years ago, we were developing a system that relied heavily on using as many cores as possible simultaneously.

Performance was critical, and we had beefy developer machines (for the time), all with 8 cores. Development was done in C++, and as the project progressed the system performed very well, and more than exceeded our performance goals.

Fast forward a couple of years and it became time to put the thing into production on a massive 128 core Windows server. Much to our surprise the performance completely tanked when scaled to all 128 cores.

After much debugging, it turned out that the system spent most of its time context switching instead of doing actual work, and because we used atomic operations for message queue functionality (compare & swap), it effectively meant clearing the cache for every core working with/on that piece of heap memory, so every time a task passed a message to something else, it effectively reset the CPU cache, which would then have to be refetched from RAM. This was not (as big) a problem on the developer machines, as there were fewer cores and each task had more work queued up for it, but with 16 as many cores to work with, it simply ran out of tasks to do faster.

The "cure" was to limit the system to run on 8 cores just like the developer machines, and AFAIK it still runs in that way all these years later.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#33

Post by xanathar »

raverbashing wrote: post #14

Wow big oooff One thing I noticed is that min/max functions are tricky to use and it's easily to accidentally do the wrong thing, like in this case Because you think "you want the minimum (that is, the number should be at least 8) of those numbers to be 8" then you slap min(). And you got it wrong. You should have used max()

After reading the article quickly, I'm not even sure there's a bug or not, the only thing that seems obviously wrong to me is the headline.

But that aside, I feel your pain around min() and max(). In my case, I feel the problem comes from language (I mean, the real life one): I speak Italian and in Italian the words for "at most" are "al massimo" (i.e. "at max") and the words for "at least" are "come minimo" (i.e. "as min") that is, the exact opposite of their mathematical meaning. I've taken an habit of reviewing 3 times any piece of code where I write "min" or "max" for this reason.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#34

Post by richrichardsson »

raverbashing wrote: post #14

Wow big oooff One thing I noticed is that min/max functions are tricky to use and it's easily to accidentally do the wrong thing, like in this case Because you think "you want the minimum (that is, the number should be at least 8) of those numbers to be 8" then you slap min(). And you got it wrong. You should have used max()

It would have been more of a bug to set this value to minimum 8 on a system with fewer than 8 cores.

I appears the author has misunderstood exactly what was its purpose, and that setting this scaling factor unbound beyond 8 cores would have detrimental effects even if there are more cores available.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#35

Post by Rattled »

quietbritishjim wrote: post #25

If I've understood this article right (possibly not as it's not so clear): * On single core machines the scheduler interval is quite fine grained to ensure responsiveness. * As the number of cores grows, the scheduler interval increases (by default), presumably because there's greater cost to task switching across more cores and the greater number of cores inherently increases the responsiveness anyway. * BUT – and t…

Can the title in the HN post be changed? I agree it's totally misleading after reading the article.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#36

Post by sebstefan »

If this was true, many of us would have noticed including myself. Who hasn't run `make -j 32` and then `make -j 64` to see if it is faster? Many times on a 64 cores machine I've seen that increasing the core count for large compile tasks makes scale as expected up to 64 cores, then a bit faster up to 128 threads (because the threads are bottlenecking on I/O so there is some CPU leftover), and then it gets slower again past 128+.

If capped to 8 it would be very clear

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#37

Post by archi42 »

nathants wrote: post #31
dannyw wrote: post #27

Earlier quoted context omitted.

You’re describing behaviour with AMD’s Infinity Fabric, which groups cores into CCDs of 8 on 7000 series chips. Essentially, a 16 core AMD CPU is actually two separate 8 core CPUs glued together with a link. Intel does not have this limitation.

yeah? a xeon with 16 cores might have to be my next pc. pricey though.

You can try to make your code aware of the situation and distribute the tasks accordingly. IIRC that's what people do on NUMA systems.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#38

Post by l33tman »

I'm annoyed that the author knows perfectly well what the code does, and yet he multiple times draws the clickbaity wrong conclusion that "the kernel has been hardcoded to use only 8 cores for 15 years" which is just patently wrong as the discussion is just about pre-emption timing and NOTHING about how many cores are used.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#39

Post by globular-toast »

I used to work on a supercomputer with 128 cores on it that ran Linux (I actually seem to remember it had 256 cores, but someone said the kernel had a limit of 128). This was less than 15 years ago. There were surely many systems just like that one. Does that mean the kernel had been patched? But nobody thought to push that patch upstream?

Reading the other comments here it seems the title is stupid and wrong and my suspicion that this can't be right was correct.

Re: The Linux kernel has been accidentally hardcoded to a maximum of 8 cores

#40

Post by PhilipRoman »

raverbashing wrote: post #14

Wow big oooff One thing I noticed is that min/max functions are tricky to use and it's easily to accidentally do the wrong thing, like in this case Because you think "you want the minimum (that is, the number should be at least 8) of those numbers to be 8" then you slap min(). And you got it wrong. You should have used max()

I don't think that's the case here, but yeah min and max can be a bit confusing to read. If your language allows adding methods or infix operators without performance overhead, I like to make something like this:

    x.atLeast(10).atMost(100)
Very easy to read in my opinion
Post reply on HN