Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

jeffgeerling.com

Post reply on HN
Print view View original

141–150 of 236 posts

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#141

Post by rbanffy »

pdpi wrote: post #44
zozbot234 wrote: post #14

Earlier quoted context omitted.

> Neural accelerators to get prompt prefill time down. Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. > this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran…

> Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of? If you daisy chain four nodes, then traffic between nodes #1 and #4 eat up all of nodes #2 and #3's bandwidth, and you eat a big latency penalty. So, absent a switch, the fully connected mesh is the only way to have fast access to all the memory.

Can’t you make bandwidth reservations and optimise data location to prefer comms between directly connected nodes over one or two-hop paths?

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#142

Post by rbanffy »

checker659 wrote: post #87
behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

For a company that has repeatedly ignored macOS, your wishlist seems anything but a pipe dream. QSFP on a mac. Yeah right. If anything, they’ll double down on TB or some nonstandard interconnect. What is a computer? (Although, I do hope with the new work on supporting RDMA, the MLX5 driver shipped with macOS will finally support RDMA for ConnectX NICs ) https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/

QSFP makes sense on a MacPro platform - and might be where Apple chooses to differentiate (one could dream of an M5 Mega, with four chiplets). The Mac Studio is a general purpose compact workstation that doesn’t need ludicrously fast networking beyond what 10Gbe and TB5 offer. It’s already overkill for the vast majority of users. Top configuration Studios are already a niche product.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#143

Post by ascagnel_ »

re-thc wrote: post #110
MBCook wrote: post #99

Earlier quoted context omitted.

That they sell to the public? No way. They’ve clearly given up on server stuff and it makes sense for them. That they use INTERNALLY for their servers? I could certainly see this being useful for that. Mostly I think this is just to get money from the AI boom. They already had TB5, it’s not like this was costing them additional hardware. Just some time that probably paid off on their internal model training anyway.

> That they sell to the public? No way. They’ve clearly given up on server stuff and it makes sense for them. Given up is not a given. A lot of the exec team has been changing.

And if the rumors are right -- that hardware SVP John Ternus is next in line for CEO -- I could see a world where the company doubles-down on their specialized hardware vs. services.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#144

Post by clan »

As Jeff states there are really no Thunderbolt switches which currently limits the size of the cluster.

But would it be possible to utilize RoCE with these boxes rather than RDMA over Thunderbolt? And what would the expected performance be? As I understand RDMA should be 7-10 times faster than via TCP. But if I understand it correctly RoCE is RDMA over Converged Ethernet. So using ethernet frames and lower layer rather than TCP.

10G Thunderbolt adapters are fairly common. But you can find 40G and 80G Thunderbolt ethernet adapters from Atto. Probably not cheap - but would be fun to test! But ieven if the bandwidth is there we might get killed with latency.

Imagine this hardware with a PCIe slot. The Infiniband hardware is there - then we "just" need the driver.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#145

Post by KeplerBoy »

clan wrote: post #144

As Jeff states there are really no Thunderbolt switches which currently limits the size of the cluster. But would it be possible to utilize RoCE with these boxes rather than RDMA over Thunderbolt? And what would the expected performance be? As I understand RDMA should be 7-10 times faster than via TCP. But if I understand it correctly RoCE is RDMA over Converged Ethernet. So using ethernet frames and lower layer rath…

At that point you could just breakout the thunderbolt to PCIe and use a regular NIC. Actually, I'm pretty sure that's all that to the Atto Thunderlink, a case around a broadcom nic.

Then you _just_ need the driver. Fascinating, Apple ships MLX5 drivers, that's crazy imo. I understand that's something they might need internally, but shipping that on ipadOs is wild. https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#146

Post by _zoltan_ »

CaliforniaKarl wrote: post #97

The "all nodes connecting to all other nodes" setup reminds me of NUMALink, the interconnect that SGI used on many (most? all?) of their supercomputers. In an ideal configuration, each 4-socket node has two NUMALink connections to every other node. As Jeff says, it's a ton of cables, and you don't have to think of framing or congestion in the same way as with RDMA over Ethernet.

just as an NVL72 rack today has 7271 links (18 probably) in the rack connecting all those GPUs together.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#147

Post by rbanffy »

doctorpangloss wrote: post #114
lostmsu wrote: post #88

Earlier quoted context omitted.

> 3090 would be nice They would need 3x speedup over the current generation to approach 3090. A100 that has +- the 3090 compute but 80GB VRAM (so fits LLaMA 70B) does prefill at 550tok/s on a single GPU: https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacp...

the GB10 is only the same performance as a 3090. gb10 uses way less power. i'm not sure why anyone would buy a mac studio instead of a gb10 machine for this use case.

> i'm not sure why anyone would buy a mac studio instead of a gb10

For an AI-only use case, the GB10s make sense, but they are only OK as desktop workstations, and I’m not sure for how long DGX OS will be updated, as dedicated AI machines have somewhat short lives. Apple computers, OTOH, have much longer lives, and desktops live the longest. I retired my Mac Mini a year after the machine was no longer getting OS updates, and it was still going strong.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#148

Post by KeplerBoy »

rbanffy wrote: post #141
pdpi wrote: post #44

Earlier quoted context omitted.

> Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of? If you daisy chain four nodes, then traffic between nodes #1 and #4 eat up all of nodes #2 and #3's bandwidth, and you eat a big latency penalty. So, absent a switch, the fully connected mesh is the only way to have fast access to all the memory.

Can’t you make bandwidth reservations and optimise data location to prefer comms between directly connected nodes over one or two-hop paths?

Sure, one could think of some kind of pipeline parallelism where you only need a fast transfer to the next step in the model and that would boost throughput but not increase model size.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#149

Post by jamesfmilne »

clan wrote: post #144

As Jeff states there are really no Thunderbolt switches which currently limits the size of the cluster. But would it be possible to utilize RoCE with these boxes rather than RDMA over Thunderbolt? And what would the expected performance be? As I understand RDMA should be 7-10 times faster than via TCP. But if I understand it correctly RoCE is RDMA over Converged Ethernet. So using ethernet frames and lower layer rath…

macOS ships with drivers for Mellanox ConnectX cards, but I have no idea if they will show up in `ibv_devices` or `ibv_devinfo`.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#150

Post by Neywiny »

_zoltan_ wrote: post #140
Neywiny wrote: post #59

Earlier quoted context omitted.

Possibly RDMA over thunderbolt. But for RoCE (RDMA over converged Ethernet) obviously not because it's sitting on top of Ethernet. Now that could still have a higher throughput when you factor in CPU time to run custom protocols that smart NICs could just DMA instead, but the overhead is still definitively higher

what do you think "ethernet's overhead" is?

Header and FCS, interpacket gap, and preamble. What do you think "Ethernet overhead" is?
Post reply on HN