When would cores talk to cores like this is measuring?
Would two cores reading and writing to the same memory have this contention?
Measuring CPU core-to-core latency
71–80 of 96 posts
Re: Measuring CPU core-to-core latency
#72I have something similar but in C++: https://github.com/rigtorp/c2clat
You need to add a Makefile there. It's just two lines.
Re: Measuring CPU core-to-core latency
#73It would be interesting to have a more detailed understanding of why these are the latencies, e.g. this repo has ‘clusters’ but there is surely some architectural reason for these clusters. Is it just physical distance on the chip or is there some other design constraint? I find it pretty interesting where the interface that cpu makers present (eg a bunch of equal cores) breaks down.
Rings are great for latency on low core count situations. The LCC Intel chips all have a massive 512-bit ring bus (2x256-bit in each direction) internally which delivers crazy fast core to core latency. However, this quickly starts to break down under higher core counts. Intel gets around this to some extent with its P and E cores with 4 E cores occupying the same slot on the ring bus as a P core. However, once you s…
I found that this is a very insightful overview of chip architectures today. Thank you for taking the time to spell this out!
I had no idea that there were 2x2 and 1x4 chips. Do you have a link that compares those in performance?
Re: Measuring CPU core-to-core latency
#74When would cores talk to cores like this is measuring? Would two cores reading and writing to the same memory have this contention?
They mention that they measure "cache coherence protocol" throughput. So yes, this happens when two or more cores need to work on the same memory and they need to synchronize/update what is in their L1 and L2 (and probably L3 as well?) caches.
Re: Measuring CPU core-to-core latency
#75When would cores talk to cores like this is measuring? Would two cores reading and writing to the same memory have this contention?
They mention that they measure "cache coherence protocol" throughput. So yes, this happens when two or more cores need to work on the same memory and they need to synchronize/update what is in their L1 and L2 (and probably L3 as well?) caches.
So cache invalidation? If core 1 writes to something core 2 has in L123? But who is responsible for that "correction", core 1 or core 2 or a combination?
Re: Measuring CPU core-to-core latency
#76I was wondering what real-life situations this benchmark matters the most in, then I remembered... A few years ago I was working on a uni research project trying to eek out the most performance possible in an x86 software-defined EPC, basically the gateway that sits between the LTE cell tower intranet and the rest of the Internet. The important part for me to optimize was the control plane, which handles handshakes b…
So what did you do to get the most performance out of the system? pin the threads to specific CPUs?
If you want to minimize latency - you got for busy waiting. It's a common thing for HFT
Re: Measuring CPU core-to-core latency
#77Earlier quoted context omitted.
They mention that they measure "cache coherence protocol" throughput. So yes, this happens when two or more cores need to work on the same memory and they need to synchronize/update what is in their L1 and L2 (and probably L3 as well?) caches.
So cache invalidation? If core 1 writes to something core 2 has in L123? But who is responsible for that "correction", core 1 or core 2 or a combination?
On the totally basic principle core 1 will tell core 2 (say non-dirty state) to 'invalidate' it. At the same time core 1' and '2' can read a different value for that memory address/cache line before '2' carries the task. The protocol is more complex than that, the cache lines have different states. The basic protocol is known as MESIF[0] for Intel.
Re: Measuring CPU core-to-core latency
#78Earlier quoted context omitted.
So what did you do to get the most performance out of the system? pin the threads to specific CPUs?
If you want to minimize latency - you got for busy waiting. It's a common thing for HFT
Not just while(true) busy wait, running the hot path continuously and keeping the 'actually send packet' flag unset. Gotta keep them caches hot.
Or, busy wait in a hardware state using MONITOR / MWAIT
Re: Measuring CPU core-to-core latency
#79Earlier quoted context omitted.
They mention that they measure "cache coherence protocol" throughput. So yes, this happens when two or more cores need to work on the same memory and they need to synchronize/update what is in their L1 and L2 (and probably L3 as well?) caches.
So cache invalidation? If core 1 writes to something core 2 has in L123? But who is responsible for that "correction", core 1 or core 2 or a combination?
Core1 would send a invalidation to get exclusive (write) access to that cache line. This would be routed to Core2. Which removes the cache line from its cache.
If core2 now accesses that cache line again, it would not find it in its L1, or L2, and at L3 there is the cacheline directory routing the request to core1.
Re: Measuring CPU core-to-core latency
#80Why does CPU=8 in Intel Core i9-12900K have fast access to all other cores? It is interesting.
Could it be because it's the newest CPU included in the test suite. Possibility related to what fab generation the CPU is on vs. other chips tested? EDIT: why the downvotes?
Each CPU in those graphs are a different core of a single physical CPU.