Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring CPU core-to-core latency

github.com

Post reply on HN
Print view View original

61–70 of 96 posts

Re: Measuring CPU core-to-core latency

#61

Post by nviennot »

mey wrote: post #58

Here is AMD Ryzen 9 5900x on Windows 11 https://gist.github.com/smarkwell/d72deee656341d53dff469df2b...

The data seems problematic, there's a few 0's here and there and some strange noise in the rest. Increase the number of samples / iterations with `core-to-core-latency 30000 1000 --csv > output.csv`

Re: Measuring CPU core-to-core latency

#63

Post by hot_gril »

I was wondering what real-life situations this benchmark matters the most in, then I remembered... A few years ago I was working on a uni research project trying to eek out the most performance possible in an x86 software-defined EPC, basically the gateway that sits between the LTE cell tower intranet and the rest of the Internet. The important part for me to optimize was the control plane, which handles handshakes between end users and the gateway (imagine everyone spam-toggling airplane mode when their LTE drops). Cache coherence latency was a bottleneck. The control plane I developed had diminishing returns in throughput up to like 8 cores on a 12-core CPU in our dual-socket test machine. Beyond that, adding cores actually slowed it down significantly*. Not a single-threaded task but not embarrassingly parallel either. The data plane was more parallel, and it ran on a separate NUMA node. Splitting either across NUMA nodes destroyed the performance.

* which in hindsight sounds like TurboBoost was enabled, but I vaguely remember it being disabled in tests

Re: Measuring CPU core-to-core latency

#64

Post by nviennot »

hot_gril wrote: post #63

I was wondering what real-life situations this benchmark matters the most in, then I remembered... A few years ago I was working on a uni research project trying to eek out the most performance possible in an x86 software-defined EPC, basically the gateway that sits between the LTE cell tower intranet and the rest of the Internet. The important part for me to optimize was the control plane, which handles handshakes b…

So what did you do to get the most performance out of the system? pin the threads to specific CPUs?

Re: Measuring CPU core-to-core latency

#66

Post by ambrosio »

Const-me wrote: post #17
sgtnoodle wrote: post #15

I've been doing some latency measurements like this, but between two processes using unix domain sockets. I'm measuring more on the order of 50uS on average, when using FIFO RT scheduling. I suspect the kernel is either letting processes linger for a little bit, or perhaps the "idle" threads tend to call into the kernel and let it do some non-preemptable book keeping. If I crank up the amount of traffic going through…

I only use AF_UNIX sockets when I need to pass open file handles between processes. I generally prefer message queues: https://linux.die.net/man/7/mq_overview I haven’t measured myself, but other people did, and they found the latency of message queues is substantially lower: https://github.com/goldsborough/ipc-bench

You might want to measure yourself, because the table of results there doesn't make a lot of sense to me. 4-5 of those methods should be dominated by context switch latency and thus clustered together tightly.

I looked at the sources. The pipe and fifo benchmarks also send a signal on every message, so they're not measuring what you'd expect.

Re: Measuring CPU core-to-core latency

#67

Post by monocasa »

bee_rider wrote: post #65

Does anyone know what is up with the 8275CL? It looks... almost periodic or something.

I think that's one of the Intel chips where the cores are connected on a 2D grid rather than the more hierarchical or single ring interconnects you see in the other systems.

Re: Measuring CPU core-to-core latency

#68

Post by mey »

nviennot wrote: post #61
mey wrote: post #58

Here is AMD Ryzen 9 5900x on Windows 11 https://gist.github.com/smarkwell/d72deee656341d53dff469df2b...

The data seems problematic, there's a few 0's here and there and some strange noise in the rest. Increase the number of samples / iterations with `core-to-core-latency 30000 1000 --csv > output.csv`

I've updated that Gist with a second run with your suggested settings.

Re: Measuring CPU core-to-core latency

#69

Post by amock »

rigtorp wrote: post #8

I have something similar but in C++: https://github.com/rigtorp/c2clat

This works much better on my POWER9 system. I have two CPUs in my machine with the same core count but different cache configurations according to hwloc and this showed that they really are different.

Re: Measuring CPU core-to-core latency

#70

Post by nviennot »

mey wrote: post #68
nviennot wrote: post #61

Earlier quoted context omitted.

The data seems problematic, there's a few 0's here and there and some strange noise in the rest. Increase the number of samples / iterations with `core-to-core-latency 30000 1000 --csv > output.csv`

I've updated that Gist with a second run with your suggested settings.

Added, thanks
Post reply on HN