1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

For RDMA you'd want Infiniband not Ethernet.

The thermal design is irrelevant, and people saying they want insane power density are, in my personal view, deluded ridiculous individuals who understand very very little.

Overclocking long ago was an amazing saintly act, milking a lot of extra performance that was just there waiting, without major downsides to take. But these days, chips are usually already well tuned. You can feed double or tripple the power into the chip with adequate cooling, but the gain is so unremarkable. +10% +15% +20% is almost never going to be a make or break difference for your work, and doing so at double or triple the power budget is an egregious waste.

So many of the chips about are already delivered at way higher than optimum efficiency, largely for bragging rights. The exponential decay of efficiency you keep pushing for is an anti-quest, is against good. The absolute performance wins are ridiculous to seek. In almost all cases.

If your problem will not scale and dumping a ton of power into one GPU or one cpu socket is all you got, fine, your problem is bad and you have to deal with that. But for 90% of people, begging for more power proces you don't actually know jack & my personal recommendation is that all such points of view deserve massive down voting by anyone with half a brain.

Go back to 2018 and look at Matthew Dillon on DragobflyBSD underpowering the heck out of their 2990wx ThreadRipper. Efficiency just soars as you tell the chip to take less power. The situation has not improved! Efficiency skyrockets today at least as much as it did then by telling chips not to go all out. Good chips behave & reward. I believe Apple competent enough to thoroughly disabuse this position that this chip would be far better if we could dump 2x 3x more power into it. Just a fools position, beyond a joke, imo. https://apollo.backplane.com/DFlyMisc/threadripper.txt

RDMA for new AI/HPC clusters is moving toward ethernet (the keyword to look for is RoCE). Ethernet gear is so much cheaper that you can massively over-provision to make up for some of the disadvantages of asynchronous networking, and it lets your run jobs on hyperscalers (only Azure ever supported actual IB). Most HPC is not latency-sensitive enough that it needs Infiniband’s lower jitter/median, and vendors have mostly caught up on the hardware acceleration front.

> 3090 would be nice

They would need 3x speedup over the current generation to approach 3090. A100 that has +- the 3090 compute but 80GB VRAM (so fits LLaMA 70B) does prefill at 550tok/s on a single GPU: https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacp...

I really hope AMD or Intel can get on the clue train and respond.

Intel in particular has half a decade of having extremely amazing Thunderbolt ports on their mobile chips, built in (alas not present on desktop chips, for shame). There's been not bad but not great thunderbolt host-to-host networking, that TCP can go over, but the system to system connectivity had been a total afterthought, not at all tuned for obvious smart readily available options like RDMA here. But nothing stops anyone from having better host-to-host protocols.

There are also so many smart good excellent next steps competitors could go for. CXL is showing up on server systems as a much lighter weight much lower latency transport that is PCIe PHY compatible but lighter weight. Adding this to consumer chips and giving even a third of a shit could blow what we see here out of the water. It could probably be done over USB4 & radically blast this bespoke RDMA capability.

Connectivity had been a bespoke special capability for too long. Intel did amazing with Xeon having integrated OmniPath 100Gb a long time ago, that was amazing, for barely any extra bucks. But the market didn't reward them kicking total ass and everyone gave up on connecting chips together. Today we are hostage to fantastically expensive shitty inefficient NIC that cost a crap ton of money to do a worse job, paying enormous penalty for not having the capability on chip, making at best asmedia io hubs do the USB4 dance a hip away from the CPU.

I really hope Intel can appreciate how good they were, see the threat of Apple kicking as here doing what Intel uniquely has been offering for half a decade with incredible Thunderbolt offerings on-chip (limited alas only to mobile chips). I hope AMD feels the heat and gets some god dMned religion and sees the pressure and thread: man they delivered so strong on PCIe lane counts but man they have been so so so slacking on io capabilities for so long, especially on consumer platforms, and Apple is using both their awesome awesome awesome on-chip memory here and their fan-tastic exceptional ability to care just even the tiniest bit about using the consumer interconnect (that already exists in hardware).

I really really really hope someone else other than Apple can ante up and care. There are so many wins to be had, so close. These companies feel so distracted from the plot. Fucking shame. Good on Apple for being the only mofos to a Tually seize the obvious that was just sitting here, they took no effort nor innovation. What a shame no other players are trying at all.