Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

developer.apple.com

Post reply on HN
Print view View original

81–90 of 304 posts

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#81

Post by FuckButtons »

reilly3000 wrote: post #48

dang I wish I could share md tables. Here’s a text edition: For $50k the inference hardware market forces a trade-off between capacity and throughput: * Apple M3 Ultra Cluster ($50k): Maximizes capacity (3TB). It is the only option in this price class capable of running 3T+ parameter models (e.g., Kimi k2), albeit at low speeds (~15 t/s). * NVIDIA RTX 6000 Workstation ($50k): Maximizes throughput (>80 t/s). It is sup…

Are you factoring in the above comment about as yet un-implemented parallel speed up in there? For on prem inference without any kind of asic this seems quite a bargain relatively speaking.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#82

Post by moralestapia »

londons_explore wrote: post #80

Nobodies gonna take them seriously till they make something rack mounted and that isn't made of titanium with pentalobe screws...

You might ignore this but, for a while, Mac Mini clusters were a thing and they were capex and opex effective. That same setup is kind of making a comeback.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#83

Post by crazygringo »

masspro wrote: post #13
Razengan wrote: post #6

Earlier quoted context omitted.

Really? I thought it's always been that HDR was notorious on Windows, hopeless on Linux, and only really worked in a plug-and-play manner on Mac, unless your display has an incorrect profile or something/ https://www.youtube.com/shorts/sx9TUNv80RE

MacOS does wash out SDR content in HDR mode specifically on non-Apple monitors. An HDR video playing in windowed mode will look fine but all the UI around it has black and white levels very close to grey. Edit: to be clear, macOS itself (Cocoa elements) is all SDR content and thus washed out.

Define "washed out"?

The white and black levels of the UX are supposed to stay in SDR. That's a feature not a bug.

If you mean the interface isn't bright enough, that's intended behavior.

If the black point is somehow raised, then that's bizarre and definitely unintended behavior. And I honestly can't even imagine what could be causing that to happen. It does seem like that it would have to be a serious macOS bug.

You should post a photo of your monitor, comparing a black #000 image in Preview with a pitch-black frame from a video. People edit HDR video on Macs, and I've never heard of this happening before.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#85

Post by 3abiton »

icedchai wrote: post #65
reilly3000 wrote: post #48

dang I wish I could share md tables. Here’s a text edition: For $50k the inference hardware market forces a trade-off between capacity and throughput: * Apple M3 Ultra Cluster ($50k): Maximizes capacity (3TB). It is the only option in this price class capable of running 3T+ parameter models (e.g., Kimi k2), albeit at low speeds (~15 t/s). * NVIDIA RTX 6000 Workstation ($50k): Maximizes throughput (>80 t/s). It is sup…

For $50K, you could buy 25 Framework desktop motherboards (128G VRAM each w/Strix Halo, so over 3TB total) Not sure how you'll cluster all of them but it might be fun to try. ;)

You could use llama.cpp rpc mode over "network" via usb4/thunderbolt connection

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#86

Post by londons_explore »

moralestapia wrote: post #82
londons_explore wrote: post #80

Nobodies gonna take them seriously till they make something rack mounted and that isn't made of titanium with pentalobe screws...

You might ignore this but, for a while, Mac Mini clusters were a thing and they were capex and opex effective. That same setup is kind of making a comeback.

It's in a similar vein to the PS2 linux cluster or someone trying to use vape CPU's as web servers...

It might be cost effective, but the supplier is still saying "you get no support, and in fact we might even put roadblocks in your way because you aren't the target customer".

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#87

Post by sspiff »

icedchai wrote: post #65
reilly3000 wrote: post #48

dang I wish I could share md tables. Here’s a text edition: For $50k the inference hardware market forces a trade-off between capacity and throughput: * Apple M3 Ultra Cluster ($50k): Maximizes capacity (3TB). It is the only option in this price class capable of running 3T+ parameter models (e.g., Kimi k2), albeit at low speeds (~15 t/s). * NVIDIA RTX 6000 Workstation ($50k): Maximizes throughput (>80 t/s). It is sup…

For $50K, you could buy 25 Framework desktop motherboards (128G VRAM each w/Strix Halo, so over 3TB total) Not sure how you'll cluster all of them but it might be fun to try. ;)

There is no way to achieve a high throughput low latency connection between 25 Strix Halo systems. After accounting for storage and network, there are barely any PCIe lanes left to link two of them together.

You might be able to use USB4 but unsure how the latency is for that.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#88

Post by conradev »

reilly3000 wrote: post #48

dang I wish I could share md tables. Here’s a text edition: For $50k the inference hardware market forces a trade-off between capacity and throughput: * Apple M3 Ultra Cluster ($50k): Maximizes capacity (3TB). It is the only option in this price class capable of running 3T+ parameter models (e.g., Kimi k2), albeit at low speeds (~15 t/s). * NVIDIA RTX 6000 Workstation ($50k): Maximizes throughput (>80 t/s). It is sup…

Apple deploys LPDDR5X for the energy efficiency and cost (lower is better), whereas NVIDIA will always prefer GDDR and HBM for performance and cost (higher is better).

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#89

Post by 3abiton »

reilly3000 wrote: post #48

dang I wish I could share md tables. Here’s a text edition: For $50k the inference hardware market forces a trade-off between capacity and throughput: * Apple M3 Ultra Cluster ($50k): Maximizes capacity (3TB). It is the only option in this price class capable of running 3T+ parameter models (e.g., Kimi k2), albeit at low speeds (~15 t/s). * NVIDIA RTX 6000 Workstation ($50k): Maximizes throughput (>80 t/s). It is sup…

What's the math on the $50k nvidia cluster? My understanding these things cost ~$8k and you can at least get 5 for $40k, that's around half a tb.

That being said, for inference mac still remain the best, and the M5 Ultra will even be a better value with its better PP.

Re: macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

#90

Post by stego-tech »

threecheese wrote: post #59
stego-tech wrote: post #54

This doesn’t remotely surprise me, and I can guess Apple’s AI endgame: * They already cleared the first hurdle to adoption by shoving inference accelerators into their chip designs by default. It’s why Apple is so far ahead of their peers in local device AI compute, and will be for some time. * I suspect this introduction isn’t just for large clusters, but also a testing ground of sorts to see where the bottlenecks l…

I think you are spot on, and this fits perfectly within my mental model of HomeKit; tasks are distributed to various devices within the network based on capabilities and authentication, and given a very fast bus Apple can scale the heck out of this.

Consumers generally have far more compute than they think; it's just all distributed across devices and hard to utilize effectively over unreliable interfaces (e.g. Wi-Fi). If Apple (or anyone, really) could figure out a way to utilize that at modern scales, I wager privacy-conscious consumers would gladly trade some latency in responses in favor of superior overall model performance - heck, branding it as "deep thinking" might even pull more customers in via marketing alone ("thinks longer, for better results" or some vaguely-not-suable marketing slogan). It could even be made into an API for things like batch image or video rendering, but without the hassle of setting up an app-specific render farm.

There's definitely something there, but Apple's really the only player setup to capitalize on it via their halo effect with devices and operating systems. Everyone else is too fragmented to make it happen.

Post reply on HN