Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

jeffgeerling.com

Post reply on HN
Print view View original

121–130 of 236 posts

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#121

Post by kappuchino »

wtallis wrote: post #55
behnamoh wrote: post #2

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM t…

> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/

He corrected that in the comment section of the youtube video. Six is actually the maximum amount. He just didn't want to buy another one.

He also published the Benchmarks in Detail and with two/four Macs in Comparison: https://github.com/geerlingguy/beowulf-ai-cluster/issues/17

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#122

Post by throwaway2037 »

fenced_load wrote: post #30
angoragoats wrote: post #13

Earlier quoted context omitted.

> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP. This isn’t any different with QSFP unless you’re suggesting that one adds a 200GbE switch to the mix, which: * Adds thousands of dollars of cost, * Adds 150W or more of power usage and the accompanying loud fan noise that comes with…

Mikrotik has a switch that can do 6x200g for ~$1300 and https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...

Wow, this switch (MikroTik CRS812) is scary good for the price point. A quick Google search fails to find any online vendors with stock. I guess it is very popular! Retail price will be I did some digging to find the switching chip: Marvell 98DX7335

Seems confirmed here: https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS...

And here: https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS...

    > Switch chip model 98DX7335
From Marvell's specs: https://www.marvell.com/content/dam/marvell/en/public-collat...

    > Description: 32x50G / 16x100G-R2 / 8x100G-R4 / 8x200G-R4 / 4x400G-R8
    > Bandwidth: 1600Gbps
Again, those are some wild numbers if I have the correct model. Normally, Mikrotik includes switching bandwidth in their own specs, but not in this case.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#123

Post by throwaway2037 »

wtallis wrote: post #53
angoragoats wrote: post #49

Earlier quoted context omitted.

As with most 40+GbE ports, the 400Gbit ports can be split into 2x200Gbit ports with the use of special cables. So you can connect a total of 6 machines at 200Gbit.

Ah, good point. Though if splitter cables are an option, then it seems more likely that the 50G ports could be combined into a 200G cable. Marvell's product brief for that switch chip does say it's capable of operating as an 8x 200G or 4x 400G switch, but Mikrotik may need to do something on their end to enable that configuration.

I'm not trolling here: Do you think that Marvell sells the chips wholesale buy the vendor buys the feature set (IP/drivers/whatever)? That would allow Marvell to effectively sell the same silicon but segment the market depending upon what buyers needs. Example: A buyer might need a config that is just a bunch of 50GB/s ports and another 100GB/s ports and another a mix. (I'm thinking about blowing fuses in the manuf phase, similar to what AMD and Intel do.) I write this as a complete noob in switching hardware.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#124

Post by spacedcowboy »

PunchyHamster wrote: post #95
burnt-resistor wrote: post #9

Earlier quoted context omitted.

Apple has always sucked at properly embracing properly robust tech for high-end gear for markets outside of individual prosumer or creatives. When Xserves existed, they used commodity IDE drives without HA or replaceable PSUs that couldn't compete with contemporary enterprise servers (HP-Compaq/Dell/IBM/Fujitsu). Xserve RAID interconnection half-heartedly used fiber channel but couldn't touch a NetApp or EMC SAN/file…

For Apple, datacenter stuff is low margin business

Considering that Apple is moving away from Linux in the datacenter to its own devices, I'm not sure that's the case. The apple machines aren't available to the consumer (they're rack-mounted, dozens of chips per PCB board, custom-made machines) but they're much less power-hungry, just as fast (or more so), much cheaper for them to make rather than buy, and natively support their own ecosystem.

Some of the machine-designs that consumers are able to buy seem to have a marked resemblance to the feature-set that the datacenter people were clamouring for. Just saying...

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#125

Post by spacedcowboy »

hamdingers wrote: post #27
chis wrote: post #18

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like…

> I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like this are downstream of that? Or do they have some real server-grade product coming down the line, and are releasing this ahead of it so that 3rd party software supports it on launch day?

I worked on some of the internal server hardware. Yes they do have their own boards. Apple used to be all-in on Linux, but the newer chips are far and away more power-efficient, and power is one of the (if not the) major cost of outfitting a datacenter, at least over time.

These machines are very much internal - you can cram a lot of M-series (to use the public nomenclature) chips onto a rack-sized PCB. I was never under the impression they were destined for anything other than Apple datacenters though...

As I mentioned above, it seems to me there's a couple of feature that appeared on the customer-facing designs that were inspired by what the datacenter people wanted on their own PCB boards.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#126

Post by spacedcowboy »

re-thc wrote: post #111
Melatonic wrote: post #105

Earlier quoted context omitted.

Do they run any of their own datacenter stuff ? I thought they just outsourced to GCP

They have some of their own, also use AWS and others too.

AWS is just used for storage, because it's cheaper than Apple maintaining it, itself. Apple do have storage-datacenter at their campus at least (I've walked around one, it's many many racks of SSD's) but almost all the public stuff is on AWS (wrapped up in encryption) AFAIK.

Apple datacenters are mainly compute, other than the storage you need to run the compute efficiently.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#127

Post by spacedcowboy »

GeekyBear wrote: post #23
andy99 wrote: post #16

Very cool, I’m probably thinking too much but why are they seemingly hyping this now (I’ve seen a bunch of this recently) with no M5 Max/Ultra machines in sight. Is it because their release is imminent (I have heard Q1 2026) or is it to try and stretch out demand for M4 Max / M3 Ultra. I plan to buy one (not four) but would feel like I’m buying something that’s going to be immediately out of date if I don’t wait for…

I imagine that they want to give developers time to get their RDMA support stabilized, so third party software will be ready to take advantage of RDMA when the M5 Ultra lands. I definitely would not be buying an M3 Ultra right now on my own dime.

I am typing this on my own 512GB M3 Ultra. I've just put out some feelers for 2nd-hand sale price...

I have an M4 Max I can use to bridge any gap...

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#128

Post by Tepix »

Linux already has RDMA support but it cannot yet use Thunderbolt. It's probably quite a bit of work to add everything that's required. Is anyone working on it?

It would be great to have this for those cheap Strix Halo boxes with 128GB quad channel DDR5-8000 for using two or three of them with their 2 USB4 ports (which are Thunderbolt capable) to fit larger models.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#129

Post by cess11 »

throwaway2037 wrote: post #122
fenced_load wrote: post #30

Earlier quoted context omitted.

Mikrotik has a switch that can do 6x200g for ~$1300 and https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...

Wow, this switch (MikroTik CRS812) is scary good for the price point. A quick Google search fails to find any online vendors with stock. I guess it is very popular! Retail price will be I did some digging to find the switching chip: Marvell 98DX7335 Seems confirmed here: https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS... And here: https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS... > Switch c…

They are very popular and make quite good products, but as you noticed it can be tricky to find them in stock.

Besides stuff like this switch they've also produced pretty cool little micro-switches you can PoE and run as WLAN hotspots, e.g. to distance your mobile user device from some network you don't really trust, or more or less maliciously bridge a cable network through a wall because your access to the building is limited.

Post reply on HN