> That's definitely fast enough for vibe coding, if that's your thing, but it's not mine.
Why even…?
1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
71–80 of 236 posts
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#72My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…
> - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W.
I don't think the Mac Studio has a thermal design capable of dissipating 650W of heat for anything other than bursty workloads. Need to look at the Mac Pro design for that.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#73Earlier quoted context omitted.
last I heard for the private compute features they were racking and stacking m2 mac pros
I honestly forgot they still made the Mac Pro. Amazing that they have these ready to ship on their website. But at a 50% premium over similar but faster Mac Studio models, what is the point? You can't usefully put GPUs in them as far as I know. You'd have to have a different PCIe need to make it make sense.
[deleted]
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#74I wonder if there's any possibility that an RDMA expansion device could exist in the future - i.e. a box full of RAM on the other end of a thunderbolt cable. Although I guess such a device would cost almost as much as a mac mini in any case...
You still need an interface which does at least two things: handles incoming read/write requests using some kind of network protocol, and operates as a memory controller for the RAM.
Texas Memory Systems was in the business of making large 'RAM Drives'. They had a product line known as "RamSan" which made many gigabytes/terabytes of DDR available via a block storage interface over infiniband and fibre channel. The control layer was implemented via FPGA.
I recall a press release from 2004 which publicized the US govt purchase of a 2.5TB RamSan. They later expanded into SSDs and were acquired by IBM in 2012.
https://en.wikipedia.org/wiki/Texas_Memory_Systems
https://www.lhcomp.com/vendors/tms/TMS-RamSan300-DataSheet.p...
https://gizmodo.com/u-s-government-purchases-worlds-largest-...
https://www.lhcomp.com/vendors/tms/TMS-RamSan20-DataSheet.pd...
https://www.ibm.com/support/pages/ibm-plans-acquire-texas-me...
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#75> You have to click buttons in the UI.
I like doing development work on a Mac, but this has to be my biggest bugbear with the system.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#76Earlier quoted context omitted.
> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/
Jeff mentioned in the video that only three of the ports can be used for RDMA. But it’s unclear where that limitation is coming from.
From my brief discussion with Exo/Apple, it sounds like that is just a limitation of this initial rollout, but it's not a hardware limitation.
Though, I am always leery to recommend any decisions be made over something that's not already proven to work, so I would say don't bet on all ports being able to be used. They very well may be able to though.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#77I'd be interested in seeing numbers that split out the speed of reading input (aka prefill) and the speed of generating output (aka decode). Those numbers are usually different and I remember from this Exo article that they could be quite radically different on Mac hardware: https://blog.exolabs.net/nvidia-dgx-spark/
See https://github.com/geerlingguy/beowulf-ai-cluster/issues/17 for more data — I didn't save all the prompt processing times (Exo just outputs a time in ms, no other data for that), but will try to have another pass. Maybe also convince the Exo team to add a proper benchmarking capability ala `llama-bench` :)
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#78Earlier quoted context omitted.
There's still a lot - particularly remote management, aka iLO in HP lingo - missing for an actual hands-off environment usable for hosters.
I know it’s not exactly IPMI, but don’t those little external IP KVM modules work well enough to do remote admin of Macs?
The annoying thing is there's no ability to control power (or see system metrics) outside the chassis. With servers and desktop PCs, you can usually tap into power pins and such.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#79Earlier quoted context omitted.
> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/
Jeff mentioned in the video that only three of the ports can be used for RDMA. But it’s unclear where that limitation is coming from.
I bet there is one piece of silicon per two ports.
Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5
#80Earlier quoted context omitted.
> Neural accelerators to get prompt prefill time down. Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. > this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran…
Might be helpful if they actually provided a programming model for ANE that isn't onnx. ANE not having a native development model just means software support will not be great.
onnx supports CoreML, is that how?