Core to core latency data on large systems
chipsandcheese.com
Core to core latency data on large systems
1–10 of 32 posts
Re: Core to core latency data on large systems
#2It'll be interesting to see how CXL shakes out. It might end up being not much more than cross socket access! 150ns to go between sockets is about what we see here & is in the realm of what CXL had been promising.
Having a super short lightweight protocol like CXL.mem to talk over such fast fabric has so much killer potential.
These graphs are always such a delight to see. It's a network map, of how well connected cores are, and they reveal so many particular advantages and diaadvantages of the greater systems architecture.
Re: Core to core latency data on large systems
#3It's almost poetic to have those mid-1990s Pentiums there, with about 2-3x the inter-socket latency of the current state-of-the-art, 30 years later.
Re: Core to core latency data on large systems
#4Very interesting. Now do bandwidth next!
Re: Core to core latency data on large systems
#5The NUMA nature of recent* chips has made me wonder if there’s ever going to be a movement to start using message passing libraries (like MPI) on shared memory machines.
* actually, not even that recent, Zen planted this hope in my brain.
Re: Core to core latency data on large systems
#6The NUMA nature of recent* chips has made me wonder if there’s ever going to be a movement to start using message passing libraries (like MPI) on shared memory machines. * actually, not even that recent, Zen planted this hope in my brain.
Thread-per-core software architectures are doing this https://penberg.org/papers/tpc-ancs19.pdf
Real world examples are scylladb and Redpanda, both built on the seastar framework (C++ https://seastar.io/message-passing/).
And for rust there is glommio https://www.datadoghq.com/blog/engineering/introducing-glomm...
Re: Core to core latency data on large systems
#7The NUMA nature of recent* chips has made me wonder if there’s ever going to be a movement to start using message passing libraries (like MPI) on shared memory machines. * actually, not even that recent, Zen planted this hope in my brain.
A good recent paper on implementing message passing over shared memory:
"Message Passing or Shared Memory: Evaluating the Delegation Abstraction for Multicores"
Re: Core to core latency data on large systems
#8The NUMA nature of recent* chips has made me wonder if there’s ever going to be a movement to start using message passing libraries (like MPI) on shared memory machines. * actually, not even that recent, Zen planted this hope in my brain.
In HPC it's common to do a mix of MPI (message-passing / distributed memory) and OpenMP (shared memory) parallelism when running on big multicore (and obviously multi-node) machines. It helps with locality, among other things.
Re: Core to core latency data on large systems
#9The NUMA nature of recent* chips has made me wonder if there’s ever going to be a movement to start using message passing libraries (like MPI) on shared memory machines. * actually, not even that recent, Zen planted this hope in my brain.
IMO, MPI is the wrong level to do this on. Most apps should either be using some form of mapreduce or not using parallelism beyond the numa node.
Re: Core to core latency data on large systems
#10It'll be interesting to see how CXL shakes out. It might end up being not much more than cross socket access! 150ns to go between sockets is about what we see here & is in the realm of what CXL had been promising. Having a super short lightweight protocol like CXL.mem to talk over such fast fabric has so much killer potential. These graphs are always such a delight to see. It's a network map, of how well connected co…
Back in the days before Oracle, Sun would sell you a dual socket Opteron desktop and you could add your own FPGA right on the hypertransport in the second socket.
Exciting to see that capability becoming more standardized with CXL.
Edit: phrasing.