Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

developer.apple.com

Post reply on HN
Print view View original

91–100 of 304 posts

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#91

Post by kmeisthax »

huslage wrote: post #34
PunchyHamster wrote: post #19

Earlier quoted context omitted.

I'm sure someone will figure out how to make thunderbolt switch/router

I don't believe the standard supports such a thing. But I wonder if TB6 will.

RDMA is a networking standard, it's supposed to be switched. The reason why it's being done over Thunderbolt is that it's the only cheap/prosumer I/O standard with enough bandwidth to make this work. Like, 100Gbit Ethernet cards are several hundred dollars minimum, for two ports, and you have to deal with SFP+ cabling. Thunderbolt is just way nicer[0].

The way this capability is exposed in the OS is that the computers negotiate an Ethernet bridge on top of the TB link. I suspect they're actually exposing PCIe Ethernet NICs to each other, but I'm not sure. But either way, a "Thunderbolt router" would just be a computer with a shitton of USB-C ports (in the same way that an "Ethernet router" is just a computer with a shitton of Ethernet ports). I suspect the biggest hurdle would actually just be sourcing an SoC with a lot of switching fabric but not a lot of compute. Like, you'd need Threadripper levels of connectivity but with like, one or two actual CPU cores.

[0] Like, last time I had to swap work laptops, I just plugged a TB cable between them and did an `rsync`.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#93

Post by dghlsakjg »

teeray wrote: post #66
mschuster91 wrote: post #32

Earlier quoted context omitted.

it's not like regular people can afford this kind of Apple machine anyway.

It’s just depressing that the “PC in every home” era is being rapidly pulled out from under our feet by all these supply shocks.

Huh?

Home PCs are as cheap as they’ve ever been. Adjusted for inflation the same can be said about “home use” Macs. The list price of an entry level MacBook Air has been pretty much the same for more than a decade. Adjust for inflation, and you get a MacBook air for less than half the real cost of the launch model that is massively better in every way.

A blip in high end RAM prices has no bearing on affordable home computing. Look at the last year or two and the proliferation of cheap, moderately to highly speced mini desktops.

I can get a Ryzen 7 system with 32gb of ddr5, and a 1tb drive delivered to my house before dinner tomorrow for $500 + tax.

That’s not depressing, that’s amazing!

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#94

Post by int32_64 »

Apple should setup their own giant cloud of M chips with tons of vram, make Metal as good as possible for AI purposes, then market the cloud as allowing self-hosted models for companies and individuals that care about privacy. They would clean up in all kinds of sectors whose data can't touch the big LLM companies.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#95

Post by icedchai »

sspiff wrote: post #87
icedchai wrote: post #65

Earlier quoted context omitted.

For $50K, you could buy 25 Framework desktop motherboards (128G VRAM each w/Strix Halo, so over 3TB total) Not sure how you'll cluster all of them but it might be fun to try. ;)

There is no way to achieve a high throughput low latency connection between 25 Strix Halo systems. After accounting for storage and network, there are barely any PCIe lanes left to link two of them together. You might be able to use USB4 but unsure how the latency is for that.

I figured, but it's good to have confirmation.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#96

Post by gumboshoes »

api wrote: post #78
andy99 wrote: post #30

Earlier quoted context omitted.

I’m hoping this isn’t as attractive as it sounds for non-hobbyists because the performance won’t scale well to parallel workloads or even context processing, where parallelism can be better used. Hopefully this makes it really nice for people that want the experiment with LLMs and have a local model but means well funded companies won’t have any reason to grab them all vs GPUs.

No way buying a bunch of minis could be as efficient as much denser GPU racks. You have to consider all the logistics and power draw, and high end nVidia stuff and probably even AMD stuff is faster than M series GPUs. What this does offer is a good alternative to GPUs for smaller scale use and research. At small scale it’s probably competitive. Apple wants to dominate the pro and serious amateur niches. Feels like th…

Exactly: The AI appliance market. A new kind of home or small-business server.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#97

Post by stego-tech »

fwip wrote: post #63
stego-tech wrote: post #54

This doesn’t remotely surprise me, and I can guess Apple’s AI endgame: * They already cleared the first hurdle to adoption by shoving inference accelerators into their chip designs by default. It’s why Apple is so far ahead of their peers in local device AI compute, and will be for some time. * I suspect this introduction isn’t just for large clusters, but also a testing ground of sorts to see where the bottlenecks l…

The bandwidth of rdma over thunderbolt is so much faster (and lower latency) than Apple's system of mostly-wireless devices, I can't see how any learnings here would transfer.

You're thinking, "You can't put modern models on that sort of distributed compute network", which is technically correct.

I was thinking, "How could we package or run these kinds of large models or workloads across a consumer's distributed compute?" The Engineer in me got as far as "Enumerate devices on network via mDNS or Bonjour, compare keys against iCloud device keys or otherwise perform authentication, share utilization telemetry and permit workload scheduling/balance" before I realized that's probably what they're testing here to a degree, even if they're using RDMA.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#98

Post by throawayonthe »

piskov wrote: post #68

George Hotz made nvidia running on macs with his tinygrad via usb4 https://x.com/__tinygrad__/status/1980082660920918045

https://social.treehouse.systems/@janne/115509948515319437 nvidia on a 2023 Mac Pro running linux :p

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#100

Post by heavyset_go »

dghlsakjg wrote: post #93
teeray wrote: post #66

Earlier quoted context omitted.

It’s just depressing that the “PC in every home” era is being rapidly pulled out from under our feet by all these supply shocks.

Huh? Home PCs are as cheap as they’ve ever been. Adjusted for inflation the same can be said about “home use” Macs. The list price of an entry level MacBook Air has been pretty much the same for more than a decade. Adjust for inflation, and you get a MacBook air for less than half the real cost of the launch model that is massively better in every way. A blip in high end RAM prices has no bearing on affordable home c…

Home calculators are cheap as they've ever been, but this era of computing is out of reach for the majority of people.

The analogous PC for this era requires a large amount of high speed memory and specialized inference hardware.

Post reply on HN