Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

jeffgeerling.com

Post reply on HN
Print view View original

91–100 of 236 posts

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#91

Post by solarkraft »

zozbot234 wrote: post #14
behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

> Neural accelerators to get prompt prefill time down. Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. > this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran…

How much of an improvement can be expected here? It seems to me that in general most potential is pretty quickly realized on Apple platforms.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#92

Post by doctorpangloss »

sneak wrote: post #69
rsync wrote: post #26

Earlier quoted context omitted.

These are my own questions - asked since the first mac mini was introduced: - Why is the tooling so lame ? - What do they, themselves, use internally ? Stringing together mac minis (or a "Studio", whatever) with thunderbolt cables ... Christ.

I assume a company like Apple either has custom server boards with tons of unified memory on M series with all the i/o they could want (that are ugly and thus not productized) or just use standard expensive nvidia stuff like everyone else.

the answer is even more boring, they use GCP haha

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#93

Post by solarkraft »

almostgotcaught wrote: post #60
chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

I don't know what you're bemused by - there's no mystery here - you can read the release notes where it literally says this was added to support MLX: https://developer.apple.com/documentation/macos-release-note... Which I'm sure you saw in literally yesterday's thread about the exact same thing.

The comment is about the larger strategy surrounding that.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#94

Post by solarkraft »

doctorpangloss wrote: post #92
sneak wrote: post #69

Earlier quoted context omitted.

I assume a company like Apple either has custom server boards with tons of unified memory on M series with all the i/o they could want (that are ugly and thus not productized) or just use standard expensive nvidia stuff like everyone else.

the answer is even more boring, they use GCP haha

It’s quite interesting how „boring“ (traditionally enterprise?) their backend looks on the occasional peeks you get publicly. So much Apache stuff & XML.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#95

Post by PunchyHamster »

burnt-resistor wrote: post #9
behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

Apple has always sucked at properly embracing properly robust tech for high-end gear for markets outside of individual prosumer or creatives. When Xserves existed, they used commodity IDE drives without HA or replaceable PSUs that couldn't compete with contemporary enterprise servers (HP-Compaq/Dell/IBM/Fujitsu). Xserve RAID interconnection half-heartedly used fiber channel but couldn't touch a NetApp or EMC SAN/file…

For Apple, datacenter stuff is low margin business

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#96

Post by PunchyHamster »

jauntywundrkind wrote: post #90

I really hope AMD or Intel can get on the clue train and respond. Intel in particular has half a decade of having extremely amazing Thunderbolt ports on their mobile chips, built in (alas not present on desktop chips, for shame). There's been not bad but not great thunderbolt host-to-host networking, that TCP can go over, but the system to system connectivity had been a total afterthought, not at all tuned for obviou…

Intel is allergic for making consumer stuff good. Remember how in consumer range like half of the chips had fucking virtualisation disabled, long after competition had it on everything ?

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#97

Post by CaliforniaKarl »

The "all nodes connecting to all other nodes" setup reminds me of NUMALink, the interconnect that SGI used on many (most? all?) of their supercomputers. In an ideal configuration, each 4-socket node has two NUMALink connections to every other node. As Jeff says, it's a ton of cables, and you don't have to think of framing or congestion in the same way as with RDMA over Ethernet.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#98

Post by amluto »

Retr0id wrote: post #35

I wonder if there's any possibility that an RDMA expansion device could exist in the future - i.e. a box full of RAM on the other end of a thunderbolt cable. Although I guess such a device would cost almost as much as a mac mini in any case...

RDMA is not really intended for this. RDMA is really just a bunch of functionality of a PCIe device, and even PCIe isn’t really quite right to use like RAM because its cache semantics aren’t intended for this use case.

But the industry knows this, and there’s a technology that is electrically compatible with PCIe that is intended for use as RAM among other things: CXL. I wonder if a anyone will ever build CXL over USB-C.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#99

Post by MBCook »

hamdingers wrote: post #27
chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

> I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like this are downstream of that? Or do they have some real server-grade product coming down the line, and are releasing this ahead of it so that 3rd party software supports it on launch day?

That they sell to the public? No way. They’ve clearly given up on server stuff and it makes sense for them.

That they use INTERNALLY for their servers? I could certainly see this being useful for that.

Mostly I think this is just to get money from the AI boom. They already had TB5, it’s not like this was costing them additional hardware. Just some time that probably paid off on their internal model training anyway.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#100

Post by T3OU-736 »

CaliforniaKarl wrote: post #97

The "all nodes connecting to all other nodes" setup reminds me of NUMALink, the interconnect that SGI used on many (most? all?) of their supercomputers. In an ideal configuration, each 4-socket node has two NUMALink connections to every other node. As Jeff says, it's a ton of cables, and you don't have to think of framing or congestion in the same way as with RDMA over Ethernet.

SGI's HW also had ccNUMA (cache-coherent Non-Uniform Memory Access), which, given the latencies possible in systems _physically_ spanning entire rooms, was quite a feat.

The IRIX OS even had functionality to migrate kobs and theor working memory closer to each other to lower the latency of access.

We see echoes of this when companies like high-frequency traders pay attention to motherboard layouts and co-locate and pin the PTS (proprietary trading systems) processes to specific cores based on which DIMMs are on which side of the memory controller.

Post reply on HN