Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NUMA: Cores, memory, and the distance between them

edera.dev

Post reply on HN
Print view View original

21–30 of 36 posts

Re: NUMA: Cores, memory, and the distance between them

#21

Post by jeffbee »

toast0 wrote: post #20
jeffbee wrote: post #19

My question: why do mainstream users tolerate NUMA? 99% of you don't need to. Single-socket servers exist and they are not only tolerable but better in most ways. Dealing with NUMA in software consists of trying to logically partition the machine, but you can instead physically partition the machine. It's so much simpler! Amazon gets this. Except for the 4th generation their Graviton systems are not NUMA.

NUMA latencies across machines are way worse than across sockets or across core complexes. :p Single socket doesn't necessarily get you away from NUMA anyway, AMD server sockets are 4 way NUMA (you can set it for interleaving, but you could do better with NUMA-aware software), and I think Intel is doing NUMA on server socket as well. A lot of people like to take one big machine and partition it into several smaller v…

AMD's NPS4 mode isn't exactly user-friendly, I agree. But you can put it into NPS1 mode and relax. Graviton 5, as a counterpoint, doesn't give you the option. Physically there is a 2D mesh between the cores and the memory controller but the observable behavior is that every access gets the average mesh fabric latency. The efficiency you leave on the table isn't very large, whereas in multisocket NUMA you can't ignore the cost.

I think you can over-analyze this stuff and lose your sanity. On these multicore systems there are also hot cores in the center of the mesh and cold ones at the edges and theoretically you could be doing temperature-aware scheduling, gaining a bit more efficiency in doing so. But it's just easier to adopt the black box model of spherical frictionless CPUs.

Re: NUMA: Cores, memory, and the distance between them

#22

Post by gopalv »

lukax wrote: post #2

NUMA can cause really crappy performance. We deployed a Go based LLM gateway in Kubernetes deployed on a server with hundreds of CPU cores. We didn't explicitly set GOMAXPROCS so Go runtime scheduled goroutines over different CPUs and it constantly used 200% CPU and GC was causing latency spikes. Then we set GOMAXPROCS 8 and all performance issues went away. Until recently Kubernetes didn't work well with NUMA.

> Kubernetes deployed on a server with hundreds of CPU cores

Was that a Power9 or some sort of IBM machine?

Not all NUMA is the same, ccNUMA from the Intel is a different beast from the PPC version of the same.

Re: NUMA: Cores, memory, and the distance between them

#23

Post by strifey »

lukax wrote: post #2

NUMA can cause really crappy performance. We deployed a Go based LLM gateway in Kubernetes deployed on a server with hundreds of CPU cores. We didn't explicitly set GOMAXPROCS so Go runtime scheduled goroutines over different CPUs and it constantly used 200% CPU and GC was causing latency spikes. Then we set GOMAXPROCS 8 and all performance issues went away. Until recently Kubernetes didn't work well with NUMA.

Heck, we saw crazy performance degradation with redis when its memory usage exceeded a single NUMA block. Not much to be done about that at the k8s level when redis is single-threaded. Have to be super conscious of the underlying hardware at that point.

Re: NUMA: Cores, memory, and the distance between them

#24

Post by senderista »

Complete slop from start to finish. Some of the distinctions manufactured by the LLM author are nonsensical: “a thread may run on node 0 but access data on node 1. Conversely, it could also run on node 1 but access data on node 0.” These cases are different how? Or “there are 3 possible cases. A thread might run on the right node but access data on the wrong node. Or it might access data on the right node but run on the wrong node. Or both.” WTF could this possibly mean? This hallucinated nonsense logic wastes the time of the reader and whoever posted it, and whoever submitted it anyway, should be ashamed of themselves. The prevalence of this garbage just makes it harder to find accurate sources on topics of interest. They are certainly out there, but it’s getting harder and harder to find them.

Re: NUMA: Cores, memory, and the distance between them

#25

Post by sillywalk »

jpecar wrote: post #14

I'm baffled by the fact that NUMA is still an issue in 2026. My impression is that this was all solved back in dotcom era already on those big SUNs. At least in HPC we solved this already in mid 2000s. Why is supposedly modern world still wasting time on this? Kernel these days exposes just about everything you would ever want to know about a system topology and every runtime should be making use of that information.…

> those big SUNs

I'm pretty sure all the big Sun boxes were SMP, not NUMA, at least during the dot-com era. Not sure about later UltraSPARC T or M Series systems.

Re: NUMA: Cores, memory, and the distance between them

#26

Post by frollogaston »

treesknees wrote: post #8

Something I didn’t see mentioned was that this unequal memory access time also affects pcie I/O. If your thread on CPU A needs to get data in or out of a nic on CPU B, your throughput/latency will be impacted. We have to explain this to customers of our software all the time, it’s something that’s easy to miss.

Was going to mention this too, as it burned me once. Not cause I didn't know about it but because I was accidentally running stuff on the wrong node, and it wasn't obvious which slot was which node.

Re: NUMA: Cores, memory, and the distance between them

#27

Post by frollogaston »

toast0 wrote: post #20
jeffbee wrote: post #19

My question: why do mainstream users tolerate NUMA? 99% of you don't need to. Single-socket servers exist and they are not only tolerable but better in most ways. Dealing with NUMA in software consists of trying to logically partition the machine, but you can instead physically partition the machine. It's so much simpler! Amazon gets this. Except for the 4th generation their Graviton systems are not NUMA.

NUMA latencies across machines are way worse than across sockets or across core complexes. :p Single socket doesn't necessarily get you away from NUMA anyway, AMD server sockets are 4 way NUMA (you can set it for interleaving, but you could do better with NUMA-aware software), and I think Intel is doing NUMA on server socket as well. A lot of people like to take one big machine and partition it into several smaller v…

The NUMA use cases exist, but given the rarity, I'd rather the default behavior not prioritize them. Currently you have to explicitly tell the system not to let a task use multiple nodes' resources, not just CPU/memory but also PCIe. Maaybe it's smart enough to avoid some cross-node usage, but it's a soft heuristic rather than a clear rule, which actually makes it more of a footgun than being dumb about it.

I would rather it be the other way around, never allow a single process to do anything cross-NUMA unless it asks for that, maybe even be stricter and require a process to opt into using anything but node0. These machines are big enough that you're not going to saturate node0 with random tasks, and you're only going to saturate the whole machine with a more deliberate workload.

Re: NUMA: Cores, memory, and the distance between them

#28

Post by 0x457 »

strifey wrote: post #23
lukax wrote: post #2

NUMA can cause really crappy performance. We deployed a Go based LLM gateway in Kubernetes deployed on a server with hundreds of CPU cores. We didn't explicitly set GOMAXPROCS so Go runtime scheduled goroutines over different CPUs and it constantly used 200% CPU and GC was causing latency spikes. Then we set GOMAXPROCS 8 and all performance issues went away. Until recently Kubernetes didn't work well with NUMA.

Heck, we saw crazy performance degradation with redis when its memory usage exceeded a single NUMA block. Not much to be done about that at the k8s level when redis is single-threaded. Have to be super conscious of the underlying hardware at that point.

In that case I run one redis instance per NUMA domain. On my home server I essentially split machine in two and treat it as two distinct machines. PCIe devices attached to a proper domain etc.

Re: NUMA: Cores, memory, and the distance between them

#30

Post by nly »

jeffbee wrote: post #18
Twirrim wrote: post #11

NUMA is one of those amazing things that trip you up in all sorts of ways at unexpected times. The amazing "invisible" performance killler (invisible because unless you're already aware of NUMA, or remember to check, you won't know it's there potentially crippling you.) It has been a source of routine conversations with customers and engineers of all kinds, and often one of those things you don't know about until too…

It's unfortunately a hard message to sell, but your program itself should never be backed by a file, for this exact reason. All code should be remapped into anonymous memory that is 1) right where you want it; 2) backed by hugepages; and 3) not shared. You are gaining nothing by trying to share objects with other processes, and you are losing a great deal of performance. The tradeoff made a little sense in the 1980s…

If the code is mapped readable and not mutated how does it make a difference?

The code will be prefetched in to local (executing) CPU cache just the same. No invalidations will take place across NUMA zones.

Modern DLLs on e.g. Linux use PIC and not mutable relocations.

Post reply on HN