Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

jeffgeerling.com

Post reply on HN
Print view View original

211–220 of 236 posts

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#211

Post by rbanffy »

FuriouslyAdrift wrote: post #204
rbanffy wrote: post #147

Earlier quoted context omitted.

> i'm not sure why anyone would buy a mac studio instead of a gb10 For an AI-only use case, the GB10s make sense, but they are only OK as desktop workstations, and I’m not sure for how long DGX OS will be updated, as dedicated AI machines have somewhat short lives. Apple computers, OTOH, have much longer lives, and desktops live the longest. I retired my Mac Mini a year after the machine was no longer getting OS upda…

DGX OS (based on Ubuntu) is used for all of nVidias GPU compute systems so it's probably going to be around for a while.

But how long until hardware support is dropped?

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#212

Post by storus »

elorant wrote: post #20
rahimnathwani wrote: post #8

The largest nodes in his cluster each have 512GB RAM. DeepSeek V3.1 is a 671B parameter model whose weights take up 700GB RAM: https://huggingface.co/deepseek-ai/DeepSeek-V3.1 I would have expected that going from one node (which can't hold the weights in RAM) to two nodes would have increased inference speed by more than the measured 32% (21.1t/s -> 27.8t/s). With no constraint on RAM (4 nodes) the inference speed i…

You only get 80Gbps network bandwidth. There's your bottleneck right there. Infiniband in comparison can give you up to x10 times that.

I think the op meant pipeline parallelism where during inference you only transfer the activation between layers where you cut the model in two, which shouldn't be too large.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#213

Post by supermatt »

What is the max token throughput when batching. Lots of agentic workflows (not just vibe coding) are running many inferences in parallel.

It seems like every time someone does an AI hardware “review” we end up with figures for just a single instance, which simply isn’t how the target demographic for a 40k cluster are going to be using it.

Jeff, I love reading your reviews, but can’t help but feel this was a wasted opportunity for some serious benchmarking of LLM performance.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#214

Post by ssl-3 »

jermaustin1 wrote: post #156
ssl-3 wrote: post #112

Earlier quoted context omitted.

I learned a lot about underclocking, undervolting, and computational power efficiency during my brief time in the ethereum mining[1] shenanigans. The best ROI was with the most-numerous stable computations at the lowest energy expense. I'd tweak individual GPUs' various clocks and volts to optimize this. I'd even go so far as to tweak fan speed ramps on the cards themselves (those fans don't power themselves! There's…

I did Crypto Mining as an alternative to heating. In my centrally cool apartment my office was the den which had the air return. So my mining rig ran RIGHT in front of that, it sucked the heat out and pushed it all over the house. Then summer came, and in Texas the AC can barely keep up to begin with. So then my GPUs became part of a render farm instead.

I did that some after eth stopped mineable.

My office-room was heated mostly by resistance, plus whatever gas-fired heat trickled in through the doorway.

I didn't have as much power available there as I had in the basement, but I had enough to mine a bit of crypto to supplement the resistance heater. :)

From one perspective: It was never directly profitable to do this. Other than eth, nothing has ever been profitable-enough for me to care about.

From another perspective: I was going to burn the energy anyway. The Joules cost the same and add the same amount of warmth either way, so I might as well get them with a side dish of free crypto.

Good times.

(These days, I transcode videos with Tdarr during the winter.)

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#215

Post by keeganpoppen »

jauntywundrkind wrote: post #84
Marsymars wrote: post #72

Earlier quoted context omitted.

> - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W. I don't think the Mac Studio has a thermal design capable of dissipating 650W of heat for anything other than bursty workloads. Need to look at the Mac Pro design for that.

The thermal design is irrelevant, and people saying they want insane power density are, in my personal view, deluded ridiculous individuals who understand very very little. Overclocking long ago was an amazing saintly act, milking a lot of extra performance that was just there waiting, without major downsides to take. But these days, chips are usually already well tuned. You can feed double or tripple the power into…

this is all 100% true and yet the 12 year-old boy inside me still smiles smugly at how fucking cool my dual reservoir water-cooled setup is, and how there was a brief moment in time a couple years ago where i had arguably one of the fastest (consumer) setups in the entire world... was any part of that labor or money "worth" it? no, absolutely not. was the $1k power bill i had to pay PG&E one month worth it? even less so. but do i have any regrets? absolutely not! :)

anyone even remotely on the fence about whether or not they should bother with all this stuff, just read OP or read this tl;dr: the answer is no, it is not.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#216

Post by storus »

delaminator wrote: post #3

> Working with some of these huge models, I can see how AI has some use, especially if it's under my own local control. But it'll be a long time before I put much trust in what I get out of it—I treat it like I do Wikipedia. Maybe good for a jumping-off point, but don't ever let AI replace your ability to think critically! It is a little sad that they gave someone an uber machine and this was the best he could come u…

M3 Ultra has a crappy GPU, somewhere around 3060Ti-3070. Its only benefit is the memory throughput that makes LLM token generation fast, at around 3080 level. But token prefill that determines time-to-first-token is extremely slow, and coincidentally all those tasks you mentioned above would be around 3060Ti level. That's why Exo coupled DGX Spark (5090 performance for FP4) with MacStudio and sped it up 4x. M5 Ultra is supposed to be as fast as DGX Spark at FP4 due to new neural cores.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#217

Post by burnte »

Dylan16807 wrote: post #198
burnte wrote: post #189

Earlier quoted context omitted.

I've been in this game so long, seen so many shortages that I'm not even worried. Right now prices are high, manufacturers are switching production, and in 6 months there's going to be a glut of supply. It's all a long game, folks. Play it long.

> manufacturers are switching production In what ways? The only switching I've seen is away from desktop memory.

You've answered the question! They're redirecting those chips to industrial use which makes desktop products more expensive and less available. Samsung is also extending DDR4 production, for example.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#218

Post by Dylan16807 »

burnte wrote: post #217
Dylan16807 wrote: post #198

Earlier quoted context omitted.

> manufacturers are switching production In what ways? The only switching I've seen is away from desktop memory.

You've answered the question! They're redirecting those chips to industrial use which makes desktop products more expensive and less available. Samsung is also extending DDR4 production, for example.

I thought you were listing a switch in production that would relieve the shortages after we wait a few months. Switching away from desktop memory makes the shortages worse. So why do you expect there to be a glut in 6 months?

If you meant glut of memory suitable for datacenter GPUs, I don't expect that nearly so soon. That market can absorb extra chips pretty easily unless we see a really harsh pop really soon.

Re: 1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

#220

Post by FuriouslyAdrift »

rbanffy wrote: post #211
FuriouslyAdrift wrote: post #204

Earlier quoted context omitted.

DGX OS (based on Ubuntu) is used for all of nVidias GPU compute systems so it's probably going to be around for a while.

But how long until hardware support is dropped?

I don't know... when is nVidia planning on getting out of the ai business?
Post reply on HN