BUILD AI has a post about this and in particular sharding k-v cache across GPUs, and how network is the new memory hierarchy:
https://buildai.substack.com/p/kv-cache-sharding-and-distrib...
1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
41–50 of 236 posts
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#42Earlier quoted context omitted.
Weights are read-only data so they can just be memory mapped and reside on SSD (only a small fraction will be needed in VRAM at any given time), the real constraint is activations. MoE architecture should help quite a bit here.
Even with MoE you still need enough memory to load all experts. For each token, only 8 experts (out of 256) are activated, but which experts are chosen changes dynamically based on the input. This means you'll be constantly loading and unloading experts from disk. MoEs is great for distributed deployments, because you can maintain a distribution of experts that matches your workload, and you can try to saturate each…
Loading and unloading data from disk is highly preferable to sending the same amount of data over a bottlenecked Thunderbolt 5 connection.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#43I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…
> rack mount hardware I guess they prefer that third parties deal with that. There’s rack mount shelves for Mac Minis and Studios.
There's still a lot - particularly remote management, aka iLO in HP lingo - missing for an actual hands-off environment usable for hosters.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#44My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…
> Neural accelerators to get prompt prefill time down. Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. > this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran…
> Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of?
If you daisy chain four nodes, then traffic between nodes #1 and #4 eat up all of nodes #2 and #3's bandwidth, and you eat a big latency penalty. So, absent a switch, the fully connected mesh is the only way to have fast access to all the memory.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#45Wow. $40k for a friendly chat(bot)...
Hey, at least this post allows us to feel as though we spent the money ourselves.
Bravo!
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#46Earlier quoted context omitted.
> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP. This isn’t any different with QSFP unless you’re suggesting that one adds a 200GbE switch to the mix, which: * Adds thousands of dollars of cost, * Adds 150W or more of power usage and the accompanying loud fan noise that comes with…
Mikrotik has a switch that can do 6x200g for ~$1300 and https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...
That switch appears to have 2x 400G ports, 2x 200G ports, 8x 50G ports, and a pair of 10G ports. So unless it allows bonding together the 50G ports (which the switch silicon probably supports at some level), it's not going to get you more than four machines connected at 200+ Gbps.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#47Earlier quoted context omitted.
Even with MoE you still need enough memory to load all experts. For each token, only 8 experts (out of 256) are activated, but which experts are chosen changes dynamically based on the input. This means you'll be constantly loading and unloading experts from disk. MoEs is great for distributed deployments, because you can maintain a distribution of experts that matches your workload, and you can try to saturate each…
Loading and unloading data from disk is highly preferable to sending the same amount of data over a bottlenecked Thunderbolt 5 connection.
No it's not.
With a cluster of two 512GB nodes, you have to send half the weights (350GB) over a TB5 connection. But you have to do this exactly once on startup.
With a single 512GB node, you'll be loading weights from disk each time you need a different expert, potentially for each token. Depending on how many experts you're loading, you might be loading 2GB to 20GB from disk each time.
Unless you're going to shut down your computer after generating a couple of hundred tokens, the cluster wins.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#48Earlier quoted context omitted.
Mikrotik has a switch that can do 6x200g for ~$1300 and https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...
Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.
how significant is the latency hit?
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#49Earlier quoted context omitted.
Mikrotik has a switch that can do 6x200g for ~$1300 and https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...
That switch appears to have 2x 400G ports, 2x 200G ports, 8x 50G ports, and a pair of 10G ports. So unless it allows bonding together the 50G ports (which the switch silicon probably supports at some level), it's not going to get you more than four machines connected at 200+ Gbps.
As with most 40+GbE ports, the 400Gbit ports can be split into 2x200Gbit ports with the use of special cables. So you can connect a total of 6 machines at 200Gbit.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#50Earlier quoted context omitted.
> Neural accelerators to get prompt prefill time down. Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. > this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran…
> Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. Or, Apple could pay for the engineers to add it.
Apple already paid software engineers to add Tensorflow support for the ANE hardware.