Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

jeffgeerling.com

Post reply on HN
Print view View original

51–60 of 236 posts

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#51

Post by angoragoats »

nicky_nickell wrote: post #48
angoragoats wrote: post #40

Earlier quoted context omitted.

Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.

how significant is the latency hit?

The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt.

Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect that to happen with 200GbE also.

And with a switch, it would likely be even worse, since the hop to the switch adds additional latency that isn’t a factor in the TB5 setup.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#52

Post by sgjohnson »

angoragoats wrote: post #49
wtallis wrote: post #46

Earlier quoted context omitted.

That switch appears to have 2x 400G ports, 2x 200G ports, 8x 50G ports, and a pair of 10G ports. So unless it allows bonding together the 50G ports (which the switch silicon probably supports at some level), it's not going to get you more than four machines connected at 200+ Gbps.

As with most 40+GbE ports, the 400Gbit ports can be split into 2x200Gbit ports with the use of special cables. So you can connect a total of 6 machines at 200Gbit.

Breakout cables typically split to 4.

e.g. QSFP28 (100GbE) splits into 4x SFP28s (25GbE each), because QSFP28 is just 4 lanes of SFP28.

Same goes for QSFP112 (400GbE). Splits into SFP112s.

It’s OSFP that can be split in half, i.e. into QSFPs.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#53

Post by wtallis »

angoragoats wrote: post #49
wtallis wrote: post #46

Earlier quoted context omitted.

That switch appears to have 2x 400G ports, 2x 200G ports, 8x 50G ports, and a pair of 10G ports. So unless it allows bonding together the 50G ports (which the switch silicon probably supports at some level), it's not going to get you more than four machines connected at 200+ Gbps.

As with most 40+GbE ports, the 400Gbit ports can be split into 2x200Gbit ports with the use of special cables. So you can connect a total of 6 machines at 200Gbit.

Ah, good point. Though if splitter cables are an option, then it seems more likely that the 50G ports could be combined into a 200G cable. Marvell's product brief for that switch chip does say it's capable of operating as an 8x 200G or 4x 400G switch, but Mikrotik may need to do something on their end to enable that configuration.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#54

Post by angoragoats »

sgjohnson wrote: post #52
angoragoats wrote: post #49

Earlier quoted context omitted.

As with most 40+GbE ports, the 400Gbit ports can be split into 2x200Gbit ports with the use of special cables. So you can connect a total of 6 machines at 200Gbit.

Breakout cables typically split to 4. e.g. QSFP28 (100GbE) splits into 4x SFP28s (25GbE each), because QSFP28 is just 4 lanes of SFP28. Same goes for QSFP112 (400GbE). Splits into SFP112s. It’s OSFP that can be split in half, i.e. into QSFPs.

Here’s an example of the cables I was referring to that can split a single 400Gbit QSFP56-DD port to two 200Gbit ports:

https://www.fs.com/products/101806.html

But all of this is pretty much irrelevant to my original point.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#55

Post by wtallis »

behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac

I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#56

Post by MBCook »

xmddmx wrote: post #38

I was impressed by the lack of dominance of Thunderbolt: "Next I tested llama.cpp running AI models over 2.5 gigabit Ethernet versus Thunderbolt 5" Results from that graph showed only a ~10% benefit from TB5 vs. Ethernet. Note: The M3 studios support 10Gbps ethernet, but that wasn't tested. Instead it was tested using 2.5Gbps ethernet. If 2.5G ethernet was only 10% slower than TB, how would 10G Ethernet have fared? A…

That’s llama, which didn’t scale nearly as well in the tests. Assumedly because it’s not optimized yet.

RDMA is always going to have lower overhead than Ethernet isn’t it?

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#57

Post by simonw »

I'd be interested in seeing numbers that split out the speed of reading input (aka prefill) and the speed of generating output (aka decode). Those numbers are usually different and I remember from this Exo article that they could be quite radically different on Mac hardware: https://blog.exolabs.net/nvidia-dgx-spark/

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#58

Post by geerlingguy »

xmddmx wrote: post #38

I was impressed by the lack of dominance of Thunderbolt: "Next I tested llama.cpp running AI models over 2.5 gigabit Ethernet versus Thunderbolt 5" Results from that graph showed only a ~10% benefit from TB5 vs. Ethernet. Note: The M3 studios support 10Gbps ethernet, but that wasn't tested. Instead it was tested using 2.5Gbps ethernet. If 2.5G ethernet was only 10% slower than TB, how would 10G Ethernet have fared? A…

10G Ethernet would only marginally speed things up based on past experience with llama RPC; latency is much more helpful but still, diminishing returns with that layer split.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#59

Post by Neywiny »

MBCook wrote: post #56
xmddmx wrote: post #38

I was impressed by the lack of dominance of Thunderbolt: "Next I tested llama.cpp running AI models over 2.5 gigabit Ethernet versus Thunderbolt 5" Results from that graph showed only a ~10% benefit from TB5 vs. Ethernet. Note: The M3 studios support 10Gbps ethernet, but that wasn't tested. Instead it was tested using 2.5Gbps ethernet. If 2.5G ethernet was only 10% slower than TB, how would 10G Ethernet have fared? A…

That’s llama, which didn’t scale nearly as well in the tests. Assumedly because it’s not optimized yet. RDMA is always going to have lower overhead than Ethernet isn’t it?

Possibly RDMA over thunderbolt. But for RoCE (RDMA over converged Ethernet) obviously not because it's sitting on top of Ethernet. Now that could still have a higher throughput when you factor in CPU time to run custom protocols that smart NICs could just DMA instead, but the overhead is still definitively higher

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#60

Post by almostgotcaught »

chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

I don't know what you're bemused by - there's no mystery here - you can read the release notes where it literally says this was added to support MLX:

https://developer.apple.com/documentation/macos-release-note...

Which I'm sure you saw in literally yesterday's thread about the exact same thing.

Post reply on HN