Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

developer.apple.com

Post reply on HN
Print view View original

111–120 of 304 posts

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#111

Post by jabbywocker »

gumboshoes wrote: post #96
api wrote: post #78

Earlier quoted context omitted.

No way buying a bunch of minis could be as efficient as much denser GPU racks. You have to consider all the logistics and power draw, and high end nVidia stuff and probably even AMD stuff is faster than M series GPUs. What this does offer is a good alternative to GPUs for smaller scale use and research. At small scale it’s probably competitive. Apple wants to dominate the pro and serious amateur niches. Feels like th…

Exactly: The AI appliance market. A new kind of home or small-business server.

I’m expecting Apple to release a new Mac Pro in the next couple years who’s main marketing angle is exactly this

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#112

Post by awnihannun »

liuliu wrote: post #71
zackangelo wrote: post #56

Earlier quoted context omitted.

No you use tensor parallelism in both cases. The way it typically works in an attention block is: smaller portions of the Q, K and V linear layers are assigned to each node and are processed independently. Attention, rope norm etc is run on the node-specific output of that. Then, when the output linear layer is applied an "all reduce" is computed which combines the output of all the nodes. EDIT: just realized it wasn…

I usually call it "head parallelism" (which is a type of tensor parallelism, but paralllelize for small clusters, and specific to attention). That is what you described: sharding input tensor by number of heads and send to respective Q, K, V shard. They can do Q / K / V projection, rope, qk norm whatever and attention all inside that particular shard. The out projection will be done in that shard too but then need to…

Right, my comment was mostly about decoding speed. For prefill you can get a speed up but there you are less latency bound.

In our benchmarks with MLX / mlx-lm it's as much as 3.5x for token generation (decoding) at batch size 1 over 4 machines. In that case you are memory bandwidth bound so sharding the model and KV cache 4-ways means each machine only needs to access 1/4th as much memory.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#113

Post by ternus »

mechagodzilla wrote: post #55
reilly3000 wrote: post #48

dang I wish I could share md tables. Here’s a text edition: For $50k the inference hardware market forces a trade-off between capacity and throughput: * Apple M3 Ultra Cluster ($50k): Maximizes capacity (3TB). It is the only option in this price class capable of running 3T+ parameter models (e.g., Kimi k2), albeit at low speeds (~15 t/s). * NVIDIA RTX 6000 Workstation ($50k): Maximizes throughput (>80 t/s). It is sup…

You can keep scaling down! I spent $2k on an old dual-socket xeon workstation with 768GB of RAM - I can run Deepseek-R1 at ~1-2 tokens/sec.

And if you get bored of that, you can flip the RAM for more than you spent on the whole system!

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#114

Post by wtallis »

_zoltan_ wrote: post #106
conradev wrote: post #88

Earlier quoted context omitted.

Apple deploys LPDDR5X for the energy efficiency and cost (lower is better), whereas NVIDIA will always prefer GDDR and HBM for performance and cost (higher is better).

the GH/GB compute has LPDDR5X - a single or dual GPU shares 480GB, depending if it's GH or GB, in addition to the HBM memory, with NVLink C2C - it's not bad!

Essentially, the Grace CPU is a memory and IO expander that happens to have a bunch of ARM CPU cores filling in the interior of the die, while the perimeter is all PHYs for LPDDR5 and NVLink and PCIe.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#115

Post by bigyabai »

Eggpants wrote: post #103
bigyabai wrote: post #67

Earlier quoted context omitted.

The lack of official Linux/BSD support is enough to make it DOA for any serious large-scale deployment. Until Apple figures out what they're doing on that front, you've got nothing to worry about.

Not sure I understand, Mac OS is BSD based. https://en.wikipedia.org/wiki/Darwin_(operating_system)

macOS is XNU-based. There is BSD code that runs in the microkernel level and BSD tools in the userland, but the kernel does not resemble BSD's architecture or adopt BSD's license.

This is an issue for some industry-standard software like CUDA, which does provide BSD drivers with ARM support that just never get adopted by Apple: https://www.nvidia.com/en-us/drivers/unix/

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#116

Post by firecall »

jabbywocker wrote: post #111
gumboshoes wrote: post #96

Earlier quoted context omitted.

Exactly: The AI appliance market. A new kind of home or small-business server.

I’m expecting Apple to release a new Mac Pro in the next couple years who’s main marketing angle is exactly this

Seems like it could be a thing.

Also, I’m curious and in case anyone that knows reads this comment:

Apple say they can’t get the performance they want out of discreet GPUs.

Fair enough. But yet nVidia becomes the most valuable company in the world selling GPUs.

So…

Now I get that Apples use case is essentially sealed consumer devices built with power consumption and performance tradeoffs in mind.

But could Apple use its Apple Silicon tech to build a Mac Pro with its own expandable GPU options?

Or even other brand GPUs knowing they would be used for AI research etc…. If Apple ever make friends with nVidia again of course :-/

What we know of Tim Cooks Apple is that it doesn’t like to leave money on the table, and clearly they are right now!

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#117

Post by wtallis »

wlesieutre wrote: post #58
crote wrote: post #47

Earlier quoted context omitted.

That screw hole is just the regular locking USB-C variant, is it not? See for example: https://www.startech.com/en-jp/cables/usb31cctlkv50cm

Looks like it! Thanks for pointing this out, I had no idea it was a standard. Apparently since 2016 https://www.usb.org/sites/default/files/documents/usb_type-c... So for any permanent Thunderbolt GPU setups, they should really be using this type of cable

Note that the locking connector OWC uses is a standard, not the standard. This is USB we're dealing with, so they made it messy: the spec defines two different mutually-incompatible locking mechanisms.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#118

Post by DrStartup »

icedchai wrote: post #70
andy99 wrote: post #11

Earlier quoted context omitted.

It’s gonna suck if all the good Macs get gobbled up by commercial users.

Outside of YouTube influencers, I doubt many home users are buying a 512G RAM Mac Studio.

I'm neither and have 2. 24/7 async inference against github issues. Free. (once you buy the macs that is)

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#119

Post by behnamoh »

dghlsakjg wrote: post #93
teeray wrote: post #66

Earlier quoted context omitted.

It’s just depressing that the “PC in every home” era is being rapidly pulled out from under our feet by all these supply shocks.

Huh? Home PCs are as cheap as they’ve ever been. Adjusted for inflation the same can be said about “home use” Macs. The list price of an entry level MacBook Air has been pretty much the same for more than a decade. Adjust for inflation, and you get a MacBook air for less than half the real cost of the launch model that is massively better in every way. A blip in high end RAM prices has no bearing on affordable home c…

> Home PCs are as cheap as they’ve ever been.

just the 5090 GPU costs +$3k, what are you even talking about

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#120

Post by bigyabai »

650REDHAIR wrote: post #52
whimsicalism wrote: post #35

Earlier quoted context omitted.

I wasn't aware that M3 Ultra offered a half terabyte of unified memory, but an RTX5090 has double that bandwidth and that's before we even get into B200 (~8TB/s).

You could get x1 M3 Ultra w/ 512gb of unified ram for the price of x2 RTX 5090 totaling 64gb of vram not including the cost of a rig capable of utilizing x2 RTX 5090.

Which would almost be great, if the M3 Ultra's GPU wasn't ~3x weaker than a single 5090: https://browser.geekbench.com/opencl-benchmarks

I don't think I can recommend the Mac Studio for AI inference until the M5 comes out. And even then, it remains to be seen how fast those GPUs are or if we even get an Ultra chip at all.

Post reply on HN