Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

jeffgeerling.com

Post reply on HN
Print view View original

21–30 of 236 posts

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#21

Post by vsgherzi »

chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

last I heard for the private compute features they were racking and stacking m2 mac pros

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#22

Post by xienze »

chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

> rack mount hardware

I guess they prefer that third parties deal with that. There’s rack mount shelves for Mac Minis and Studios.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#23

Post by GeekyBear »

andy99 wrote: post #16

Very cool, I’m probably thinking too much but why are they seemingly hyping this now (I’ve seen a bunch of this recently) with no M5 Max/Ultra machines in sight. Is it because their release is imminent (I have heard Q1 2026) or is it to try and stretch out demand for M4 Max / M3 Ultra. I plan to buy one (not four) but would feel like I’m buying something that’s going to be immediately out of date if I don’t wait for…

I imagine that they want to give developers time to get their RDMA support stabilized, so third party software will be ready to take advantage of RDMA when the M5 Ultra lands.

I definitely would not be buying an M3 Ultra right now on my own dime.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#24

Post by zozbot234 »

rahimnathwani wrote: post #8

The largest nodes in his cluster each have 512GB RAM. DeepSeek V3.1 is a 671B parameter model whose weights take up 700GB RAM: https://huggingface.co/deepseek-ai/DeepSeek-V3.1 I would have expected that going from one node (which can't hold the weights in RAM) to two nodes would have increased inference speed by more than the measured 32% (21.1t/s -> 27.8t/s). With no constraint on RAM (4 nodes) the inference speed i…

Weights are read-only data so they can just be memory mapped and reside on SSD (only a small fraction will be needed in VRAM at any given time), the real constraint is activations. MoE architecture should help quite a bit here.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#25

Post by jeffbee »

chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

thunderbolt rdma is quite clearly the nuclear option for remote management.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#26

Post by rsync »

chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

These are my own questions - asked since the first mac mini was introduced:

- Why is the tooling so lame ?

- What do they, themselves, use internally ?

Stringing together mac minis (or a "Studio", whatever) with thunderbolt cables ... Christ.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#27

Post by hamdingers »

chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

> I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like this are downstream of that?

Or do they have some real server-grade product coming down the line, and are releasing this ahead of it so that 3rd party software supports it on launch day?

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#29

Post by fooblaster »

andy99 wrote: post #16

Very cool, I’m probably thinking too much but why are they seemingly hyping this now (I’ve seen a bunch of this recently) with no M5 Max/Ultra machines in sight. Is it because their release is imminent (I have heard Q1 2026) or is it to try and stretch out demand for M4 Max / M3 Ultra. I plan to buy one (not four) but would feel like I’m buying something that’s going to be immediately out of date if I don’t wait for…

Does it actually creates a unified memory pool? it looks more like an accelerated backend for a collective communications library like nccl, which is very much not unified memory.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#30

Post by fenced_load »

angoragoats wrote: post #13
behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP. This isn’t any different with QSFP unless you’re suggesting that one adds a 200GbE switch to the mix, which: * Adds thousands of dollars of cost, * Adds 150W or more of power usage and the accompanying loud fan noise that comes with…

Mikrotik has a switch that can do 6x200g for ~$1300 and https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...
Post reply on HN