Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

developer.apple.com

Post reply on HN
Print view View original

181–190 of 304 posts

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#181

Post by fooblaster »

wmf wrote: post #146
dpe82 wrote: post #108

Earlier quoted context omitted.

> The main challenge is latency since you have to do much more frequent communication. Earlier this year I experimented with building a cluster to do tensor parallelism across large cache CPUs (AMD EPYC 7773X have 768mb of L3). My thought was to keep an entire model in SRAM and take advantage of the crazy memory bandwidth between CPU cores and their cache, and use Infiniband between nodes for the scatter/gather opera…

That's how Groq works. A cluster of LPUv2s would probably be faster and cheaper than an Infiniband cluster of Epycs.

what is an lpuv2

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#182

Post by FuckButtons »

api wrote: post #78
andy99 wrote: post #30

Earlier quoted context omitted.

I’m hoping this isn’t as attractive as it sounds for non-hobbyists because the performance won’t scale well to parallel workloads or even context processing, where parallelism can be better used. Hopefully this makes it really nice for people that want the experiment with LLMs and have a local model but means well funded companies won’t have any reason to grab them all vs GPUs.

No way buying a bunch of minis could be as efficient as much denser GPU racks. You have to consider all the logistics and power draw, and high end nVidia stuff and probably even AMD stuff is faster than M series GPUs. What this does offer is a good alternative to GPUs for smaller scale use and research. At small scale it’s probably competitive. Apple wants to dominate the pro and serious amateur niches. Feels like th…

Power draw? A entire Mac Pro running flat out uses less power than 1 5090. If you have a workload that needs a huge memory footprint then the tco of the Macs, even with their markup may be lower.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#183

Post by bigyabai »

7e wrote: post #156
bigyabai wrote: post #115

Earlier quoted context omitted.

macOS is XNU-based. There is BSD code that runs in the microkernel level and BSD tools in the userland, but the kernel does not resemble BSD's architecture or adopt BSD's license. This is an issue for some industry-standard software like CUDA, which does provide BSD drivers with ARM support that just never get adopted by Apple: https://www.nvidia.com/en-us/drivers/unix/

If there were TCO advantages with this setup, CUDA would not be a blocker.

CUDA's just one example; there's a lot of hardware support on the BSDs that Apple doesn't want to inherit.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#184

Post by adastra22 »

bigyabai wrote: post #120
650REDHAIR wrote: post #52

Earlier quoted context omitted.

You could get x1 M3 Ultra w/ 512gb of unified ram for the price of x2 RTX 5090 totaling 64gb of vram not including the cost of a rig capable of utilizing x2 RTX 5090.

Which would almost be great, if the M3 Ultra's GPU wasn't ~3x weaker than a single 5090: https://browser.geekbench.com/opencl-benchmarks I don't think I can recommend the Mac Studio for AI inference until the M5 comes out. And even then, it remains to be seen how fast those GPUs are or if we even get an Ultra chip at all.

Again, memory bandwidth is pretty much all that matters here. During inference or training the CUDA cores of retail GPUs are like 15% utilized.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#185

Post by wtallis »

bleepblap wrote: post #161
wmf wrote: post #151

Earlier quoted context omitted.

Within a single node it's just called DMA. RDMA is DMA over a network and RoCE is RDMA over Ethernet.

Sorry, but it certainly isn't-- https://docs.nvidia.com/cuda/gpudirect-rdma/index.html The "R" in RDMA means there are multiple DMA controllers who can "transparently" share address spaces. You can certainly share address spaces across nodes with RoCE or Infiniband, but thats a layer on top

I don't know why that NVIDIA document is wrong, but the established term for doing DMA from eg. an NVMe SSD to a GPU within a single system without the CPU initiating the transfer is peer to peer DMA. RDMA is when your data leaves the local machine's PCIe fabric.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#186

Post by [deleted] »

adastra22 wrote: post #184
bigyabai wrote: post #120

Earlier quoted context omitted.

Which would almost be great, if the M3 Ultra's GPU wasn't ~3x weaker than a single 5090: https://browser.geekbench.com/opencl-benchmarks I don't think I can recommend the Mac Studio for AI inference until the M5 comes out. And even then, it remains to be seen how fast those GPUs are or if we even get an Ultra chip at all.

Again, memory bandwidth is pretty much all that matters here. During inference or training the CUDA cores of retail GPUs are like 15% utilized.

[deleted]

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#187

Post by wmf »

CamperBob2 wrote: post #176
wmf wrote: post #174

Earlier quoted context omitted.

Yeah, two Mac Studios is going to be ~400 W.

What am I missing? https://i.imgur.com/YpcnlCH.png (Edit: interesting, thanks. So the underlying OS APIs that supply the power-consumption figures reported by asitop are just outright broken. The discrepancy is far too large to chalk up to static power losses or die-specific calibration factors that the video talks about.)

https://www.youtube.com/watch?v=zCkbVLqUedg

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#189

Post by kridsdale1 »

icedchai wrote: post #70
andy99 wrote: post #11

Earlier quoted context omitted.

It’s gonna suck if all the good Macs get gobbled up by commercial users.

Outside of YouTube influencers, I doubt many home users are buying a 512G RAM Mac Studio.

I did. Admittedly it was for video processing at 8k which uses more than 128gb of ram, but I am NOT a YouTuber.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#190

Post by ThomasBb »

geerlingguy wrote: post #21

This implies you'd run more than one Mac Studio in a cluster, and I have a few concerns regarding Mac clustering (as someone who's managed a number of tiny clusters, with various hardware): 1. The power button is in an awkward location, meaning rackmounting them (either 10" or 19" rack) is a bit cumbersome (at best) 2. Thunderbolt is great for peripherals, but as a semi-permanent interconnect, I have worries over the…

With MDM solutions you can not only get software update management, but even full LOM for models that support this. There are free and open source MDM out there.
Post reply on HN