Earlier quoted context omitted.
LLM inference is mostly memory bound. An 12-channel Epyc Genoa with 4800MT/s DDR5 ram clocks at 460.8 GB/sec. It's more than the 400GB/s of the M3 Max, only part of that accessible for the CPU. And on the Epyc System you can plug much more memory for when you need larger memory and PCI-E gpus, for when you need less faster memory. Threadripper PRO is only 8-channel, but with memory overclocking it might reach numbers…
That's interesting. It's about the same speed as the M3 Max then. Have you tested it yourself?
Nope, but this guy has a similar build: https://www.reddit.com/r/LocalLLaMA/comments/1bt8kc9/compari...
It seems to reach only a little above half the theoretical speed, and scale only up to 32 threads for some reason. Might be a temporary software limitation or something more fundamental.