Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring CPU core-to-core latency

github.com

Post reply on HN
Print view View original

21–30 of 96 posts

Re: Measuring CPU core-to-core latency

#21

Post by bitcharmer »

ls65536 wrote: post #7
SkipperCat wrote: post #6

Earlier quoted context omitted.

In HFT, we typically pin processes to run on a single isolated core (on a multicore machine). That allows the process to avoid a lot of kernel and other interrupts which could cause the process to not operate in a low latency manner. If we have two of these processes, each on separate cores, and they occasionally need to talk to each other, then knowing the best choice of process/core location can keep the system ope…

In general this makes sense, but I think you need to be careful in some cases where the lowest latency between two logical "cores" is likely to be between those which are SMT siblings on the same physical core (assuming you have an SMT-enabled system). These logical "cores" will be sharing much of the same physical core's resources (such as the low-latency L1/L2 and micro-op caches), so depending on the particular wo…

No one in hft space runs with smt enabled

Re: Measuring CPU core-to-core latency

#22

Post by nextaccountic »

slabity wrote: post #9
ls65536 wrote: post #7

Earlier quoted context omitted.

In general this makes sense, but I think you need to be careful in some cases where the lowest latency between two logical "cores" is likely to be between those which are SMT siblings on the same physical core (assuming you have an SMT-enabled system). These logical "cores" will be sharing much of the same physical core's resources (such as the low-latency L1/L2 and micro-op caches), so depending on the particular wo…

SMT is usually disabled in these situations to prevent it from being a concern.

Doesn't this leave some performance on the table? Each core has more ports than a single thread could reasonably use, exactly because two threads can run on a single core

Re: Measuring CPU core-to-core latency

#23

Post by bitcharmer »

dan-robertson wrote: post #13

It would be interesting to have a more detailed understanding of why these are the latencies, e.g. this repo has ‘clusters’ but there is surely some architectural reason for these clusters. Is it just physical distance on the chip or is there some other design constraint? I find it pretty interesting where the interface that cpu makers present (eg a bunch of equal cores) breaks down.

Most of this cross-core overhead diversity is gone on skylake and newer chips because Intel moved from a ring topology to mesh design for their l3 caches.

Re: Measuring CPU core-to-core latency

#24

Post by haroldrijn »

crazytalk wrote: post #11
fideloper wrote: post #3

Because I’m ignorant: What are the practical take aways from this? When is a cpu core sending a message to another core?

It's mentioned in the readme - this is measuring the latency of cache coherence. Depending on architecture, some sets of cores will be organized with shared L2/L3 cache. In order to acquire exclusive access to a cache line (memory range of 64-128ish bytes), caches belonging to other sets of cores need to be waited on to release their own exclusive access, or to be informed they need to invalidate their caches. This i…

AMD processors also use a hierarchical coherence directory, where the global coherence directory on the IO die enforces coherence across chiplets and a local coherence directory on each chiplet enforces coherence on-die http://www.cs.cmu.edu/afs/cs/academic/class/15740-f03/www/le...

Re: Measuring CPU core-to-core latency

#25

Post by jeffbee »

Fails to build from source with Rust 1.59 so I tried the C++ `c2clat` from elsewhere in the thread. Quite interesting on Alder Lake, because the quartet of Atom cores has uniform latency (they share an L2 cache and other resources) while the core-to-core latency of the Core side of the CPU varies. Note that the way these are logically numbers is 0,1 are SMT threads of the first core and so forth through 14-15. 16-19 are Atom cores with 1 thread each.

  CPU   0    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17   18   19
   0    0   12   60   44   60   44   60   43   50   47   56   48   58   49   60   50   79   79   78   79
   1   12    0   45   45   44   44   60   43   51   49   55   47   57   49   56   51   76   76   76   76
   2   60   45    0   13   42   43   53   43   48   37   52   41   53   42   53   42   72   72   72   72
   3   44   45   13    0   42   43   53   42   47   37   51   40   53   41   53   42   72   72   72   72
   4   60   44   42   42    0   13   56   43   49   52   54   41   56   42   42   41   75   75   74   75
   5   44   44   43   43   13    0   56   43   51   54   55   41   56   42   56   42   77   77   77   77
   6   60   60   53   53   56   56    0   13   49   54   56   41   57   42   57   42   78   78   78   78
   7   43   43   43   42   43   43   13    0   46   47   54   41   41   41   55   41   72   71   71   71
   8   50   51   48   47   49   51   49   46    0   12   51   51   54   56   55   56   75   75   75   75
   9   47   49   37   37   52   54   54   47   12    0   49   53   54   56   55   54   74   69   67   68
  10   56   55   52   51   54   55   56   54   51   49    0   13   53   58   56   59   75   75   76   75
  11   48   47   41   40   41   41   41   41   51   53   13    0   51   52   55   59   75   75   75   75
  12   58   57   53   53   56   56   57   41   54   54   53   51    0   13   55   60   77   77   77   77
  13   49   49   42   41   42   42   42   41   56   56   58   52   13    0   55   54   77   77   77   77
  14   60   56   53   53   42   56   57   55   55   55   56   55   55   55    0   12   74   70   78   78
  15   50   51   42   42   41   42   42   41   56   54   59   59   60   54   12    0   75   74   74   77
  16   79   76   72   72   75   77   78   72   75   74   75   75   77   77   74   75    0   55   55   55
  17   79   76   72   72   75   77   78   71   75   69   75   75   77   77   70   74   55    0   55   55
  18   78   76   72   72   74   77   78   71   75   67   76   75   77   77   78   74   55   55    0   55
  19   79   76   72   72   75   77   78   71   75   68   75   75   77   77   78   77   55   55   55    0

Re: Measuring CPU core-to-core latency

#26

Post by slabity »

nextaccountic wrote: post #22
slabity wrote: post #9

Earlier quoted context omitted.

SMT is usually disabled in these situations to prevent it from being a concern.

Doesn't this leave some performance on the table? Each core has more ports than a single thread could reasonably use, exactly because two threads can run on a single core

In terms of throughput, technically yes, you are leaving performance on the table. However, in HFT the throughput is greatly limited by IO anyways, so you don't get much benefit with it enabled.

What you want is to minimize latency, which means you don't want to be waiting for anything before you start processing whatever information you need. To do this, you need to ensure that the correct things are cached where they need to be, and SMT means that you have multiple threads fighting each other for that precious cache space.

In non-FPGA systems I've worked with, I've seen dozens of microseconds of latency added with SMT enabled vs disabled.

Re: Measuring CPU core-to-core latency

#28

Post by jtorsella »

If anyone is interested, here are the results on my M1 Pro running Asahi Linux:

Min: 48.3 Max: 175.0 Mean: 133.0

I’ll try to copy the exact results once I have a browser on Asahi, but the general pattern is most pairs have >150ns and a few (0-1; 2-3,4,5; 3-4,5; 4-5; 6-7,8,9; 7-8,9; 8-9) are faster at about 50ns.

Edit: The results from c2clat (a little slower, but the format is nicer) are below.

CPU 0 1 2 3 4 5 6 7 8 9

   0    0   59  231  205  206  206  208  219  210  210

   1   59    0  205  215  207  207  209  209  210  210

   2  231  205    0   40   42   43  180  222  224  213

   3  205  215   40    0   43   43  212  222  213  213

   4  206  207   42   43    0   44  182  227  217  217

   5  206  207   43   43   44    0  215  215  217  217

   6  208  209  180  212  182  215    0   40   43   45

   7  219  209  222  222  227  215   40    0   43   43

   8  210  210  224  213  217  217   43   43    0   44

   9  210  210  213  213  217  217   45   43   44    0

Re: Measuring CPU core-to-core latency

#30

Post by nviennot »

jtorsella wrote: post #28

If anyone is interested, here are the results on my M1 Pro running Asahi Linux: Min: 48.3 Max: 175.0 Mean: 133.0 I’ll try to copy the exact results once I have a browser on Asahi, but the general pattern is most pairs have >150ns and a few (0-1; 2-3,4,5; 3-4,5; 4-5; 6-7,8,9; 7-8,9; 8-9) are faster at about 50ns. Edit: The results from c2clat (a little slower, but the format is nicer) are below. CPU 0 1 2 3 4 5 6 7 8…

It'd be great if you could provide the output.csv actually :D

I think there's a CLI tool for pastebin

Post reply on HN