191–200 of 304 posts
Remember when they enabled egpu over thunderbolt and no one cared because the thunderbolt housing cost almost as much as your macbook outright? Yeah. Thunderbolt is a racket. It’s a god damned cord. Why is it $50.
(The cord is $50 because it contains two active chips BTW.)
Earlier quoted context omitted.
Almost the most impressive thing about that is the power consumption. ~50 watts for both of them? Am I reading it wrong?
Yeah, two Mac Studios is going to be ~400 W.
Earlier quoted context omitted.
It’s just depressing that the “PC in every home” era is being rapidly pulled out from under our feet by all these supply shocks.
You can get a Mac Mini for $600 with 16GB of RAM and it will be more powerful than the "PC in every home" people would need for any common software. The personal computing situation is great right now. RAM is temporarily more expensive, but it's definitely not ending any eras.
Earlier quoted context omitted.
For $50K, you could buy 25 Framework desktop motherboards (128G VRAM each w/Strix Halo, so over 3TB total) Not sure how you'll cluster all of them but it might be fun to try. ;)
There is no way to achieve a high throughput low latency connection between 25 Strix Halo systems. After accounting for storage and network, there are barely any PCIe lanes left to link two of them together. You might be able to use USB4 but unsure how the latency is for that.
Strix Halo IO topology: https://www.techpowerup.com/cpu-specs/ryzen-ai-max-395.c3994
Frameworks mainboard implements 2 of those PCIe4x4 GPP interfaces as M.2 PHY's which you can use a passive adapter to connect a standard PCIe AIC (like a NIC or DPU) to, and also interestingly exposes that 3rd x4 GPP as a standard x4 length PCIe CEM slot, though the system/case isn't compatible with actually installing a standard PCIe add in card in there without getting hacky with it, especially as it's not an open-ended slot.
You absolutely could slap 1x SSD in there for local storage, and then attach up to 4x RDMA supporting NIC's to a RoCE enabled switch (or Infiniband if you're feeling special) to build out a Strix Halo cluster (and you could do similar with Mac Studio's to be fair). You could get really extra by using a DPU/SmartNIC that allows you to boot from a NVMeoF SAN to leverage all 5 sets of PCIe4x4 for connectivity without any local storage but we're hitting a complexity/cost threshold with that that I doubt most people want to cross. Or if they are willing to cross that threshold, they'd also be looking at other solutions better suited to that that don't require as many workarounds.
Apple's solution is better for a small cluster, both in pure connectivity terms and also with respect to it's memory advantages, but Strix Halo is doable. However, in both cases, scaling up beyond 3 or especially 4 nodes you rapidly enter complexity and cost territory that is better served by nodes that are less restrictive unless you have some very niche reason to use either Mac's (especially non-pro) or Strix Halo specifically.
dang I wish I could share md tables. Here’s a text edition: For $50k the inference hardware market forces a trade-off between capacity and throughput: * Apple M3 Ultra Cluster ($50k): Maximizes capacity (3TB). It is the only option in this price class capable of running 3T+ parameter models (e.g., Kimi k2), albeit at low speeds (~15 t/s). * NVIDIA RTX 6000 Workstation ($50k): Maximizes throughput (>80 t/s). It is sup…
You can keep scaling down! I spent $2k on an old dual-socket xeon workstation with 768GB of RAM - I can run Deepseek-R1 at ~1-2 tokens/sec.
Earlier quoted context omitted.
Outside of YouTube influencers, I doubt many home users are buying a 512G RAM Mac Studio.
I doubt many of them are, either. When the 2019 Mac Pro came out, it was "amazing" how many still photography YouTubers all got launch day deliveries of the same BTO Mac Pro, with exactly the same spec: 18 core CPU, 384GB memory, Vega II Duo GPU and an 8TB SSD. Or, more likely, Apple worked with them and made sure each of them had this Mac on launch day, while they waited for the model they actually ordered . Because…
Earlier quoted context omitted.
Home calculators are cheap as they've ever been, but this era of computing is out of reach for the majority of people. The analogous PC for this era requires a large amount of high speed memory and specialized inference hardware.
What regular home workload are you thinking of that the computer I described is incapable of? You can call a computer a calculator, but that doesn’t make it a calculator. Can they run SOTA LLMs? No. Can they run smaller, yet still capable LLMs? Yes. However, I don’t think that the ability to run SOTA LLMs is a reasonable expectation for “a computer in every home” just a few years into that software category even exis…
That’s great for AI people, but can we use this for other distributed workloads that aren’t ML?
See: https://ml-explore.github.io/mlx/build/html/usage/distribute...