Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

jeffgeerling.com

Post reply on HN
Print view View original

151–160 of 236 posts

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#151

Post by clan »

KeplerBoy wrote: post #145
clan wrote: post #144

As Jeff states there are really no Thunderbolt switches which currently limits the size of the cluster. But would it be possible to utilize RoCE with these boxes rather than RDMA over Thunderbolt? And what would the expected performance be? As I understand RDMA should be 7-10 times faster than via TCP. But if I understand it correctly RoCE is RDMA over Converged Ethernet. So using ethernet frames and lower layer rath…

At that point you could just breakout the thunderbolt to PCIe and use a regular NIC. Actually, I'm pretty sure that's all that to the Atto Thunderlink, a case around a broadcom nic. Then you _just_ need the driver. Fascinating, Apple ships MLX5 drivers, that's crazy imo. I understand that's something they might need internally, but shipping that on ipadOs is wild. https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-mac…

That is what I am suggesting with the Atto adapter.

Infiniband is way faster and lower latency than a NIC. These days NIC==Ethernet.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#152

Post by KeplerBoy »

clan wrote: post #151
KeplerBoy wrote: post #145

Earlier quoted context omitted.

At that point you could just breakout the thunderbolt to PCIe and use a regular NIC. Actually, I'm pretty sure that's all that to the Atto Thunderlink, a case around a broadcom nic. Then you _just_ need the driver. Fascinating, Apple ships MLX5 drivers, that's crazy imo. I understand that's something they might need internally, but shipping that on ipadOs is wild. https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-mac…

That is what I am suggesting with the Atto adapter. Infiniband is way faster and lower latency than a NIC. These days NIC==Ethernet.

What makes you think Infiniband is faster than Ethernet? Aren't they pretty much equal these days with RDMA and kernel bypass?

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#153

Post by re-thc »

kappuchino wrote: post #121
wtallis wrote: post #55

Earlier quoted context omitted.

> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/

He corrected that in the comment section of the youtube video. Six is actually the maximum amount. He just didn't want to buy another one. He also published the Benchmarks in Detail and with two/four Macs in Comparison: https://github.com/geerlingguy/beowulf-ai-cluster/issues/17

> He just didn't want to buy another one.

Wasn’t it loaned ie didn’t buy any at all?

Apple should have loaned enough to flex.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#154

Post by stetrain »

Melatonic wrote: post #105
chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

Do they run any of their own datacenter stuff ? I thought they just outsourced to GCP

All of the Private Cloud Compute stuff they are working on runs on their own Apple Silicon server hardware.

https://security.apple.com/blog/private-cloud-compute/

https://www.apple.com/newsroom/in-the-loop/2025/10/shipping-...

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#155

Post by gwehrli »

xmddmx wrote: post #38

I was impressed by the lack of dominance of Thunderbolt: "Next I tested llama.cpp running AI models over 2.5 gigabit Ethernet versus Thunderbolt 5" Results from that graph showed only a ~10% benefit from TB5 vs. Ethernet. Note: The M3 studios support 10Gbps ethernet, but that wasn't tested. Instead it was tested using 2.5Gbps ethernet. If 2.5G ethernet was only 10% slower than TB, how would 10G Ethernet have fared? A…

This Video tests the setup using 10Gbps ethernet: https://www.youtube.com/watch?v=4l4UWZGxvoc

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#156

Post by jermaustin1 »

ssl-3 wrote: post #112
jauntywundrkind wrote: post #84

Earlier quoted context omitted.

The thermal design is irrelevant, and people saying they want insane power density are, in my personal view, deluded ridiculous individuals who understand very very little. Overclocking long ago was an amazing saintly act, milking a lot of extra performance that was just there waiting, without major downsides to take. But these days, chips are usually already well tuned. You can feed double or tripple the power into…

I learned a lot about underclocking, undervolting, and computational power efficiency during my brief time in the ethereum mining[1] shenanigans. The best ROI was with the most-numerous stable computations at the lowest energy expense. I'd tweak individual GPUs' various clocks and volts to optimize this. I'd even go so far as to tweak fan speed ramps on the cards themselves (those fans don't power themselves! There's…

I did Crypto Mining as an alternative to heating. In my centrally cool apartment my office was the den which had the air return. So my mining rig ran RIGHT in front of that, it sucked the heat out and pushed it all over the house. Then summer came, and in Texas the AC can barely keep up to begin with. So then my GPUs became part of a render farm instead.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#157

Post by mcculley »

spacedcowboy wrote: post #124
PunchyHamster wrote: post #95

Earlier quoted context omitted.

For Apple, datacenter stuff is low margin business

Considering that Apple is moving away from Linux in the datacenter to its own devices, I'm not sure that's the case. The apple machines aren't available to the consumer (they're rack-mounted, dozens of chips per PCB board, custom-made machines) but they're much less power-hungry, just as fast (or more so), much cheaper for them to make rather than buy, and natively support their own ecosystem. Some of the machine-des…

> dozens of chips per PCB board

Have there been leaks or something about these internal machines? I am curious to know more.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#158

Post by justincormack »

rbanffy wrote: post #142
checker659 wrote: post #87

Earlier quoted context omitted.

For a company that has repeatedly ignored macOS, your wishlist seems anything but a pipe dream. QSFP on a mac. Yeah right. If anything, they’ll double down on TB or some nonstandard interconnect. What is a computer? (Although, I do hope with the new work on supporting RDMA, the MLX5 driver shipped with macOS will finally support RDMA for ConnectX NICs ) https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/

QSFP makes sense on a MacPro platform - and might be where Apple chooses to differentiate (one could dream of an M5 Mega, with four chiplets). The Mac Studio is a general purpose compact workstation that doesn’t need ludicrously fast networking beyond what 10Gbe and TB5 offer. It’s already overkill for the vast majority of users. Top configuration Studios are already a niche product.

Also given that Apple are using these in their datacenters I think they will ship much more server like hw.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#159

Post by RunSet »

wartywhoa23 wrote: post #118
behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

Would you please mind leaving some RAM to remain available for purchase at an affordable price for us mere mortals ? 1Tb for what, like, "Come on AI, make the humankind happy now"? /"s"

AI bubble will do wonders for used RAM prices when it pops.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#160

Post by nottorp »

jauntywundrkind wrote: post #84
Marsymars wrote: post #72

Earlier quoted context omitted.

> - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W. I don't think the Mac Studio has a thermal design capable of dissipating 650W of heat for anything other than bursty workloads. Need to look at the Mac Pro design for that.

The thermal design is irrelevant, and people saying they want insane power density are, in my personal view, deluded ridiculous individuals who understand very very little. Overclocking long ago was an amazing saintly act, milking a lot of extra performance that was just there waiting, without major downsides to take. But these days, chips are usually already well tuned. You can feed double or tripple the power into…

> Overclocking long ago was an amazing saintly act, milking a lot of extra performance that was just there waiting, without major downsides to take.

Back when you bought a 233 Mhz chip with ram at 66 Mhz, ran the bus at 100 Mhz which also increased your ram speed if it could handle it, and everything was faster.

> But these days, chips are usually already well tuned. You can feed double or tripple the power into the chip with adequate cooling, but the gain is so unremarkable. +10% +15% +20% is almost never going to be a make or break difference for your work

20% in synthetic benchmarks maybe, or very particular loads. Because you only overclock the CPU these days so anything hitting the ram won't even go to 20%.

Post reply on HN