Measuring CPU core-to-core latency

> So what did you do to get the most performance out of the system? pin the threads to specific CPUs?

No, they were already pinning to cores. Kernel was also in low-latency mode.

The project they handed to me was just the data plane, written in Rust. They also gave me a partial reference implementation of the control plane, meant for research purposes rather than performance. I had to add a lot of missing features to get it up to par with the supposedly industry-standard benchmark, which didn't exactly match spec so I had to reverse-engineer. Then I had to mate it with the data plane. (They originally suggested I build a control plane from scratch in Rust, but the lack of an ASN1 codegen lib for it made this infeasible within the time I had, considering also that I had 0 systems experience or familiarity with the protocols.) I don't remember all the optimizations, but the ones that still come to mind:

1. Fixing all their memory leaks, obviously. Kinda hard cause they were in C code auto-generated by a Python script. There was even a code comment // TODO free memory after use.

2. Improving the ASN1 en/decoding to take advantage of memcpy in some places. This is because asn1 has different alignment modes, some byte-aligned and some bit-aligned. The control plane used byte-aligned, but the standard ASN1.c lib understandably assumed bit-aligned in either case for simplicity's sake since it worked either way. So I added an optimized path for byte-aligned that used memcpy instead of inspecting each byte for the end markers. This was in a tight loop and made the biggest difference; basically every string copy got faster. The relevant function even knew it was in byte-aligned mode, so it was a simple fix once I figured it out; I tried to make a PR to improve this for everyone else, but forget why I couldn't.

3. Playing with different arrangements of passing messages between threads and different ways of locking. I forget all the ones we tried. Using "parking lot" locks instead of the default ones in the Rust portion helped, also more optimistic locks in other places instead of mutexes, I forget where and why. Since then I've come across the general concept of optimistic vs pessimistic locking a lot as something that makes or breaks performance, particularly in systems that handle money.

4. As I said, playing with the number of threads for each different setup in #3.

5. Playing with NIC settings. We were using Intel's DPDK library and optimized NIC drivers.

6. Making a custom `malloc` implementation that used a memory pool, was thread-scoped, and was optimized for repeated small allocs/deallocs specifically for a portion of the reused code that had a weird and inefficient pattern of memory access. I got it to be faster than the built-in malloc, BUT it was still break-even with DPDK's custom pooled malloc, so I gave up.

7. Branch hints. Tbh didn't make a big difference, even though this was pre Meltdown/Spectre.

8. Simplifying the telemetry. Idk if this helped performance, more of a rant... It's good enough to have some counters that you printf every 60s or something, then parse the logs with a Python script. That's very non-prone to bugs, anyone can understand it, and you can easily tell there's no significant impact on performance. It's overkill in this case to have a protobuf/HTTP client sending metrics to a custom sidecar process, complicating your builds, possibly impacting performance, and leaving no simple paper trail from each test. I respected the previous guy's engineering skills more than my own, but once I found a bug in that code, I took it out.

I suspect what you are seeing is the preferred cores. With TVB/OCTVB (thermal velocity boost/overclocking thermal velocity boost) those can boost higher than the other cores.

From my 12900ks (its a boosted 12900k) Core 6-7 is rated for 5.5ghz.

Freq MHz DTS C VID mv PC Eff Fr UCcode 0x1f VR Volt Limit 2500 ---------------------------------------------------- 800 24 1307 52 517 Uncore 3600 IA AC LL 0.5999 800 20 1282 52 47 Power 23.465 IA DC LL 1.0996 800 20 1314 52 70 Current Limit 0.0000 SA AC LL 0.0000 800 20 1308 52 69 iccmax 1023 SA DC LL 0.0000 800 20 1303 52 88 PL1 32760 iccmax dis True 800 18 1293 52 69 PL2 32760 TAU 33 800 20 1286 55 39 Memory 5200 PPP_OVR True 800 22 730 55 37 EE_Turbo_Dis True RTH_Dis True 800 22 1377 40 60 Dis_Ring_EE False HWGuidedSch True 800 22 1172 40 73 IA_CEP_Dis True Dynamic_Mem True Full_Range_Multi False SA_Freq_OVR False TSC_Dis_HW False Banding_Ratio 0 PVD_Ratio_thresh 0 SA_CEP_Dis True FLL OC Mode 3

---------------------------------------------------------------------------------------------------------------------------------

Core Voltage Adapt 0 Ring Voltage Adapt 0 Core PLL V 900 Core Voltage Offset 0 Ring Voltage Offset 0 Ring PLL V 900 L2 Voltage Adapt 0 L2 Voltage Offset 0 AVX512 Offset 0 SA Voltage Offset 0 Ring VID 0 MC PLL V 900 AVX Offset 0 AVX2 V Guardband 0 AVX512 V Guardband 0 SA Voltage Manual 1150

---------------------------------------------------------------------------------------------------------------------------------

Turbo Ratio Limit 55, 55, 52, 52, 52, 52, 52, 52 Core OCMB Max Ratio 0 Ring Min Ratio 8 Turbo Limit Cores 1, 2, 3, 4, 5, 6, 7, 8 Ring OCMB Max Ratio 0 Ring Max Ratio 47 Atom Ratio Limit 40, 40, 40, 40, 40, 40, 40, 40 Atom Limit Cores 255, 255, 255, 255, 255, 255, 255, 255Atom OCMB Max Ratio 0 OS Max Ratio 34 HWP Min Ratio 43 HWP Max Ratio 255 Max Possible Core 40 Max Possible Ring 47 UCLK 2600

Num cores: 24 Using RDTSC to measure time: false Num round trips per samples: 5000 Num samples: 300 Showing latency=round-trip-time/2 in nanoseconds:

       0       1       2       3       4       5       6       7       8       9      10      11      12      13      14      15      16      17      18      19      20      21      22      23
  0
  1    0±0
  2   37±6    36±6
  3   31±6    37±6     5±5
  4   26±6    26±6    26±6    26±6
  5   26±6    31±6    26±6    31±6     5±5
  6   31±6    31±6    36±6    31±6    26±6    21±6
  7   31±6    31±6    31±6    31±6    26±6    26±6     5±5
  8   31±6    31±6    31±6    31±6    26±6    26±6    31±6    26±6
  9   36±6    31±6    31±6    31±6    26±6    26±6    26±6    26±6     5±5
 10   31±6    36±6    31±6    26±6    26±6    21±6    26±6    31±6    26±6    31±6
 11   31±6    36±6    31±6    36±6    26±6    26±6    31±6    31±6    36±6    31±6     5±5
 12   31±6    36±6    31±6    42±6    26±6    31±6    31±6    37±6    31±6    37±6    31±6    36±6
 13   36±6    37±6    42±6    36±6    26±6    26±6    31±6    31±6    31±6    31±6    31±6    31±6     5±5
 14   31±6    36±6    37±6    36±6    31±6    26±6    36±6    31±6    36±6    31±6    36±6    31±6    31±6    36±6
 15   31±6    31±6    31±6    31±6    26±6    26±6    36±6    31±6    31±6    31±6    31±6    31±6    31±6    31±6     0±0
 16   42±6    42±6    42±6    42±6    31±6    31±6    37±6    36±6    36±6    42±6    37±6    36±6    36±6    36±6    37±6    37±6
 17   42±6    42±6    36±6    42±6    31±6    36±6    37±6    36±6    36±6    37±6    36±6    37±6    36±6    36±6    36±6    37±6    47±6
 18   42±6    42±6    42±6    42±6    31±6    31±6    37±6    36±6    37±6    36±6    42±6    36±6    37±6    36±6    36±6    36±6    47±6    52±6
 19   42±6    42±6    42±6    37±6    31±6    31±6    36±6    37±6    36±6    36±6    37±6    37±6    36±6    36±6    42±6    37±6    52±6    47±6    47±6
 20   42±6    42±6    42±6    36±6    37±6    31±6    37±6    36±6    37±6    36±6    36±6    42±6    37±6    36±6    36±6    42±6    42±6    42±6    42±6    42±6
 21   42±6    37±6    42±6    42±6    31±6    31±6    37±6    36±6    42±6    36±6    37±6    36±6    37±6    37±6    36±6    36±6    42±6    42±6    42±6    42±6    47±6
 22   42±6    42±6    42±6    36±6    36±6    31±6    36±6    37±6    36±6    37±6    36±6    36±6    36±6    37±6    36±6    42±6    42±6    42±6    42±6    42±6    47±6    52±6
 23   36±6    42±6    42±6    42±6    31±6    31±6    37±6    36±6    37±6    42±6    36±6    36±6    37±6    42±6    36±6    36±6    42±6    42±6    42±6    42±6    52±6    47±6    47±6

Min latency: 0.0ns ±0.0 cores: (1,0) Max latency: 52.3ns ±16.3 cores: (23,20) Mean latency: 34.6ns

This benchmark reminds me of "ffwd: delegation is (much) faster than you think" https://www.seltzer.com/margo/teaching/CS508-generic/papers-....

This paper describes a mechanism for client threads pinned to a distinct cores to delegate a function call to distinguished server thread pinned to its own core all on the same socket.

This has a multitude of applications the most obvious one making a shared data structure MT safe through delegation rather than saddling it with mutexes or other synchronization points especially beneficial with small critical sections.

The paper's abstract concludes claiming "100% [improvement] over the next best solution tested (RCL), and multiple micro-benchmarks show improvements in the 5–10× range."

The code does delegation without CAS, locks, or atomics.

The efficacy of such a scheme rests on two facets, which the paper explains:

* Modern CPUs can move GBs/second between core L2/LLC caches

* The synchronization between requesting clients and responding servers depends on each side spinning on shared memory address looking for bit toggles. Briefly, servers only read client request memory which the client only writes. (Clients each have their own slot). And on the response side client's read the servers shared response memory, which only the server writes. This one-side read, one-side write is supposed to minimize the number of cache invalidations and MESI syncs.

I spent some time testing the author's code and went so far as writing my own version. I was never able to make it work with anywhere near the throughput claimed in the paper. There's also some funny "nop" assembler instructions within the code that I gather is a cheap form of thread yielding.

In fact this relatively simple SPCP MT ring buffer which has but a fraction of the code:

https://rigtorp.se/ringbuffer/

did far, far better.

In my experiments then CPU spun too quickly so that core-to-core bandwidth was quickly squandered before the server could signal response or the client could signal request. I wonder if adding select atomic reads as with the SPSC ring might help.