Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

developer.apple.com

Post reply on HN
Print view View original

211–220 of 304 posts

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#211

Post by embedding-shape »

oidar wrote: post #167
Waterluvian wrote: post #136

Earlier quoted context omitted.

I wonder what the actual lifetime amortized cost will be.

Every time I'm tempted to get one of these beefy mac studios, I just calculate how much inference I can buy for that amount and it's never a good deal.

Every time someone brings up that, it brings me back memories of trying to frantically finish stuff as quickly as possible as either my quota slowly go down with each API request, or the pay-as-you-go bill is increasing 0.1% for each request.

Nowadays I fire off async jobs that involve 1000s of requests, billion of tokens, yet it costs basically the same as if I didn't.

Maybe it takes a different type of person, than the one I am, but all these "pay-as-you-go"/tokens/credits platforms make me nervous to use, and I end up not using it or spending time trying to "optimize", while investing in hardware and infrastructure I can run at home and use that seems to be no problem for my head to just roll with.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#212

Post by asimovDev »

oidar wrote: post #167
Waterluvian wrote: post #136

Earlier quoted context omitted.

I wonder what the actual lifetime amortized cost will be.

Every time I'm tempted to get one of these beefy mac studios, I just calculate how much inference I can buy for that amount and it's never a good deal.

anyone buying these is usually more concerned with just being able to run stuff on their own terms without handing their data off. otherwise it's probably always cheaper to rent compute for intense stuff like this

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#213

Post by aurareturn »

nickysielicki wrote: post #210
aurareturn wrote: post #208

Earlier quoted context omitted.

The end goal is that Macs become good local LLM inference machines and for AI devs to keep using Macs.

The former will never happen and the latter is a certainty.

The former is already true and will become even more true when M5 Pro/Max/Ultra release.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#214

Post by jamiek88 »

wtallis wrote: post #117
wlesieutre wrote: post #58

Earlier quoted context omitted.

Looks like it! Thanks for pointing this out, I had no idea it was a standard. Apparently since 2016 https://www.usb.org/sites/default/files/documents/usb_type-c... So for any permanent Thunderbolt GPU setups, they should really be using this type of cable

Note that the locking connector OWC uses is a standard, not the standard. This is USB we're dealing with, so they made it messy: the spec defines two different mutually-incompatible locking mechanisms.

Of course they do.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#215

Post by zeristor »

Will Apple be able to ramp up M3 Ultra MacStudios if this becomes a big thing?

Is this part of Apple’s plan of building out server side AI support using their own hardware?

If so they would need more physical data centres.

I’m guessing they too would be constrained by RAM.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#216

Post by ChrisMarshallNY »

codazoda wrote: post #40
andy99 wrote: post #30

Earlier quoted context omitted.

I’m hoping this isn’t as attractive as it sounds for non-hobbyists because the performance won’t scale well to parallel workloads or even context processing, where parallelism can be better used. Hopefully this makes it really nice for people that want the experiment with LLMs and have a local model but means well funded companies won’t have any reason to grab them all vs GPUs.

I haven’t looked yet but I might be a candidate for something like this, maybe. I’m RAM constrained and, to a lesser extent, CPU constrained. It would be nice to offload some of that. That said, I don’t think I would buy a cluster of Macs for that. I’d probably buy a machine that can take a GPU.

I’m not particularly interested in training models, but it would be nice to have eGPUs again. When Apple Silicon came out, support for them dried up. I sold my old BlackMagic eGPU.

That said, the need for them also faded. The new chips have performance every bit as good as the eGPU-enhanced Intel chips.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#217

Post by dpe82 »

wmf wrote: post #146
dpe82 wrote: post #108

Earlier quoted context omitted.

> The main challenge is latency since you have to do much more frequent communication. Earlier this year I experimented with building a cluster to do tensor parallelism across large cache CPUs (AMD EPYC 7773X have 768mb of L3). My thought was to keep an entire model in SRAM and take advantage of the crazy memory bandwidth between CPU cores and their cache, and use Infiniband between nodes for the scatter/gather opera…

That's how Groq works. A cluster of LPUv2s would probably be faster and cheaper than an Infiniband cluster of Epycs.

Yeah I'm familiar; I was hoping I could do something related on previous generation commodity(ish) hardware. It didn't work but I learned a ton.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#218

Post by madeofpalk »

DrStartup wrote: post #118
icedchai wrote: post #70

Earlier quoted context omitted.

Outside of YouTube influencers, I doubt many home users are buying a 512G RAM Mac Studio.

I'm neither and have 2. 24/7 async inference against github issues. Free. (once you buy the macs that is)

I'm not sure who 'home users' are, but i doubt they're buying two $9,499 computers.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#219

Post by dontlaugh »

oidar wrote: post #167
Waterluvian wrote: post #136

Earlier quoted context omitted.

I wonder what the actual lifetime amortized cost will be.

Every time I'm tempted to get one of these beefy mac studios, I just calculate how much inference I can buy for that amount and it's never a good deal.

For now, while everything you can rent is sold at a loss.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#220

Post by kergonath »

Weryj wrote: post #127
mechagodzilla wrote: post #55

Earlier quoted context omitted.

You can keep scaling down! I spent $2k on an old dual-socket xeon workstation with 768GB of RAM - I can run Deepseek-R1 at ~1-2 tokens/sec.

Just keep going! 2TB of swap disk for 0.0000001 t/sec

Hang on, starting benchmarks on my Raspberry Pi.
Post reply on HN