Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

jeffgeerling.com

Post reply on HN
Print view View original

61–70 of 236 posts

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#61

Post by Terretta »

burnt-resistor wrote: post #9
behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

Apple has always sucked at properly embracing properly robust tech for high-end gear for markets outside of individual prosumer or creatives. When Xserves existed, they used commodity IDE drives without HA or replaceable PSUs that couldn't compete with contemporary enterprise servers (HP-Compaq/Dell/IBM/Fujitsu). Xserve RAID interconnection half-heartedly used fiber channel but couldn't touch a NetApp or EMC SAN/file…

> I'm disappointed Apple has a persistent blindspot preventing them from succeeding in ... things that would eventually find their way into my home lab after 5-7 years.

I can see the dollar signs in their eyes right now.

Aftermarkets are a nice reflection of durable value, and there's a massive one for iPhones and a smaller one for quick flameout startup servers, but not much money in 5 - 7 year old servers.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#62

Post by lvl155 »

geerlingguy wrote: post #28
lvl155 wrote: post #15

Seriously, Jeff has the best job. Him and STH Patrick.

I got to spend a day with Patrick this week, and try out his massive CyPerf testing rig with multiple 800 Gbps ConnectX-8 cards!

Patrick’s enthusiasm is so contagious and you perfected tech YouTube format. There’s not a dead spot in your video.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#64

Post by xienze »

mschuster91 wrote: post #43
xienze wrote: post #22

Earlier quoted context omitted.

> rack mount hardware I guess they prefer that third parties deal with that. There’s rack mount shelves for Mac Minis and Studios.

There's still a lot - particularly remote management, aka iLO in HP lingo - missing for an actual hands-off environment usable for hosters.

I know it’s not exactly IPMI, but don’t those little external IP KVM modules work well enough to do remote admin of Macs?

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#65

Post by dev_l1x_be »

behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

Mine is to remove the extreme Macos bloat.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#66

Post by wmf »

angoragoats wrote: post #51
nicky_nickell wrote: post #48

Earlier quoted context omitted.

how significant is the latency hit?

The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt. Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect…

You're ignoring RoCE which would have the same or lower latency than RoTB. And I think macOS already supports RoCE.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#67

Post by QuantumNomad_ »

wtallis wrote: post #55
behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/

Jeff mentioned in the video that only three of the ports can be used for RDMA. But it’s unclear where that limitation is coming from.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#68

Post by donavanm »

burnt-resistor wrote: post #9
behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

Apple has always sucked at properly embracing properly robust tech for high-end gear for markets outside of individual prosumer or creatives. When Xserves existed, they used commodity IDE drives without HA or replaceable PSUs that couldn't compete with contemporary enterprise servers (HP-Compaq/Dell/IBM/Fujitsu). Xserve RAID interconnection half-heartedly used fiber channel but couldn't touch a NetApp or EMC SAN/file…

Enterprise never ever mattered, and there arent enough digits available to show your “home lab” use case in the revenue numbers. Xserve, the RAID shelves, and the directory services were kinda there as a half hearted attempt for that late 90-00s AV setup. All of that fell on the cutting room floor once personal devices, esp iphone, was realized.

By the time I left in ‘10 the total revenue from mac hardware was like 15% of revenue. Im honestly surprised theres anyone who cared enough to package the business services for mac minis.

So if everything else is printing cash for a HUGE addressable consumer market at premium price points why would they try and compete with their own ODMs on more-or-less commodity enterprise gear?

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#69

Post by sneak »

rsync wrote: post #26
chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

These are my own questions - asked since the first mac mini was introduced: - Why is the tooling so lame ? - What do they, themselves, use internally ? Stringing together mac minis (or a "Studio", whatever) with thunderbolt cables ... Christ.

I assume a company like Apple either has custom server boards with tons of unified memory on M series with all the i/o they could want (that are ugly and thus not productized) or just use standard expensive nvidia stuff like everyone else.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#70

Post by jeffbee »

vsgherzi wrote: post #21
chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

last I heard for the private compute features they were racking and stacking m2 mac pros

I honestly forgot they still made the Mac Pro. Amazing that they have these ready to ship on their website. But at a 50% premium over similar but faster Mac Studio models, what is the point? You can't usefully put GPUs in them as far as I know. You'd have to have a different PCIe need to make it make sense.
Post reply on HN