Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thelio Mira AI Linux Workstation: 192 GB GPU Memory

system76.com

Post reply on HN
Print view View original

111–120 of 134 posts

Re: Thelio Mira AI Linux Workstation: 192 GB GPU Memory

#111

Post by embedding-shape »

swiftcoder wrote: post #109
embedding-shape wrote: post #108

Earlier quoted context omitted.

Right, performance is more than just the aggregated memory speed of the hardware you have. I'm fairly sure, at least last time I looked, maybe Apple launched something new in the last 2-3 months that has completely changed the picture?

> performance is more than just the aggregated memory speed of the hardware you have In other fields, sure, but for big LLMs it's a very significant part of the performance picture. That stack of Macs also is going to be able to natively run models 4-5x larger than the dual Blackwells can hold in memory - any model over about 128GB of weights isn't going to fit on the GPUs, and is going to be heavily performance cons…

> any model over about 128GB of weights isn't going to fit on the GPUs

Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM.

> Indeed. The M5 Ultra (currently up for pre-order) has 50% higher memory bandwidth than its predecessor, and a claimed 4x improvement in prompt prefill.

Exciting! Eagerly awaiting the benchmarks and comparisons then. Lets hope "4x improvement" had a good enough baseline so 4x actually ends up useful in practice compared to the current hardware they offer.

> 50% higher memory bandwidth

Seems this lands on ~1.2 TB/s if what Apple claims is correct. For reference, RTX Pro 6000 does 1.8 TB/s, so seems Apple is indeed getting closer incrementally.

Re: Thelio Mira AI Linux Workstation: 192 GB GPU Memory

#112

Post by swiftcoder »

embedding-shape wrote: post #111
swiftcoder wrote: post #109

Earlier quoted context omitted.

> performance is more than just the aggregated memory speed of the hardware you have In other fields, sure, but for big LLMs it's a very significant part of the performance picture. That stack of Macs also is going to be able to natively run models 4-5x larger than the dual Blackwells can hold in memory - any model over about 128GB of weights isn't going to fit on the GPUs, and is going to be heavily performance cons…

> any model over about 128GB of weights isn't going to fit on the GPUs Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM. > Indeed. The M5 Ultra (currently up for pre-order) has 50% higher memory bandwidth than its predecessor, and a claimed 4x improvement in prompt prefill. Exciting! Eagerly awaiting the benchmarks and comparisons then. Lets hope "4x improvement" had a go…

> Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM.

Right, but the top option here is a pair of RTX Pro 6000s, hence 192 GB of VRAM in total. Should be enough for a 128GB model plus context, cache, etc.

Re: Thelio Mira AI Linux Workstation: 192 GB GPU Memory

#113

Post by embedding-shape »

swiftcoder wrote: post #112
embedding-shape wrote: post #111

Earlier quoted context omitted.

> any model over about 128GB of weights isn't going to fit on the GPUs Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM. > Indeed. The M5 Ultra (currently up for pre-order) has 50% higher memory bandwidth than its predecessor, and a claimed 4x improvement in prompt prefill. Exciting! Eagerly awaiting the benchmarks and comparisons then. Lets hope "4x improvement" had a go…

> Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM. Right, but the top option here is a pair of RTX Pro 6000s, hence 192 GB of VRAM in total. Should be enough for a 128GB model plus context, cache, etc.

Yeah, that's why the "any model over about 128GB of weights isn't going to fit on the GPUs" part doesn't make sense, you'll easily be able to run weights in that weight class on two of them. Or did I misunderstand what "on the GPUs" you meant?

Re: Thelio Mira AI Linux Workstation: 192 GB GPU Memory

#114

Post by embedding-shape »

anonreplier wrote: post #103
embedding-shape wrote: post #88

Earlier quoted context omitted.

Which one is the fastest and how fast is it? Even the "fast" ones doesn't seem to even reach close to consumer NVIDIA GPUs released years ago when it comes to prompt processing.

large model and slow, or speed and a teeny model, make your choice

Or you get N of RTX Pro 6000 and get large and fast models :) Make your choice, and do it before the prices go up even more.

Re: Thelio Mira AI Linux Workstation: 192 GB GPU Memory

#115

Post by swiftcoder »

embedding-shape wrote: post #113
swiftcoder wrote: post #112

Earlier quoted context omitted.

> Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM. Right, but the top option here is a pair of RTX Pro 6000s, hence 192 GB of VRAM in total. Should be enough for a 128GB model plus context, cache, etc.

Yeah, that's why the "any model over about 128GB of weights isn't going to fit on the GPUs" part doesn't make sense, you'll easily be able to run weights in that weight class on two of them. Or did I misunderstand what "on the GPUs" you meant?

Right, easily run 128-160 GB models, yes. Not easily run models much larger than that. Anything that doesn't fit in the 192 GB (including context, cache, etc) is going to have to be sparse/MoE, and the rather anaemic PCIE bandwidth is going to hurt.

By comparison, the 5x Mac cluster should be able to run a dense ~800GB-1TB model without a drastic slowdown.

Re: Thelio Mira AI Linux Workstation: 192 GB GPU Memory

#116

Post by larodi »

embedding-shape wrote: post #114
anonreplier wrote: post #103

Earlier quoted context omitted.

large model and slow, or speed and a teeny model, make your choice

Or you get N of RTX Pro 6000 and get large and fast models :) Make your choice, and do it before the prices go up even more.

indeed. make up your mind, and also "slow/fast" mean very little given plethora of models to choose from. fast for what, slow for what. fast with which harness, etc...

Re: Thelio Mira AI Linux Workstation: 192 GB GPU Memory

#117

Post by slowmovintarget »

embedding-shape wrote: post #89
coffeebeqn wrote: post #61

Earlier quoted context omitted.

Really just the GPUs. PSUs aren’t that expensive maybe 100-200 for a good 750W-1000W one

Kind of feels like if you spend $40K on your GPU setup, you don't want to cheap out on the PSU. Still, you're right, I have a 1200W PSU for a single RTX Pro 6000, cost ~500 something, so still the PSU shouldn't be that large part of the total budget.

Funny thing is, you don't really want to just plug that into a wall socket. You want an uninterruptible power supply that can keep the machine running for the 5 to 8 minutes you need to get to it and shut it down safely during a power cut.

But you'll find most household power sockets aren't actually rated for the kind of power these things draw at peak. If you go for a dual GPU you're going to want to sustain a higher current. Think washer / drier / fridge kind of hookups for the UPS and that rig.

You spent $42K on the rig, you're also going to need to get an electrician in and have the electrical hookup seen to. The single 4090 I'm running put me right up to that edge where I could use a standard electrical hookup with a 1000w UPS.

So another $1K for the UPS, and $500 worth of electrical work (or more if they have to up the switch to 200 amps, for example).

Not for the faint of wallet.

Re: Thelio Mira AI Linux Workstation: 192 GB GPU Memory

#118

Post by slowmovintarget »

matheusmoreira wrote: post #81
mandeepj wrote: post #58

Earlier quoted context omitted.

Hopefully it will come down to that price point or cheaper again in the next 3-4 years?

Hopefully... If it doesn't, mere mortals like us consumers are going to be priced out of computers altogether. It'll be the end of the personal computing era and a return to big iron mainframes.

"All of this has happened before, and will happen again."
Post reply on HN