Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5.2 – How to Run Locally

unsloth.ai

Post reply on HN
Print view View original

221–230 of 328 posts

Re: GLM-5.2 – How to Run Locally

#221

Post by Gracana »

bbor wrote: post #205
xrd wrote: post #2

So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970

I’m kinda lost here… do y’all really have machines in your houses with hundreds of gigs of RAM?? Am I just behind the times? The page advertises the 8-bit quant as taking ~800GB, which seems like it would require at least 3 consumer motherboards fully stacked w/ 4x64GB cards each. Maybe “locally” has slowly come to imply “…on your homelab”?

You don't have to have a server, workstation motherboards support lots of memory channels.

I was lucky to buy a lot of RAM before prices skyrocketed. I knew I wanted to play with this stuff, so I spent what felt like a lot of money at the time to buy 8x96GB DDR5-6400 RDIMMs. Now the same RAM costs at least 6x more.

Re: GLM-5.2 – How to Run Locally

#223

Post by traceroute66 »

patates wrote: post #216
traceroute66 wrote: post #202

Earlier quoted context omitted.

> We do want privacy, and we also want to own the hardware so the US can't just turn it off whenever it feels like it. I agree and I prefer on-prem where possible. The Apple Mac Studios have been great for that although I don't have enough of them to run GLM-5.2 without heavy quantization. I'm also waiting for the Apple next product refresh which I hope will enable me to do more with less. Meanwhile there are hosted…

> Apple next product refresh I have this feeling that it'll be very expensive and still scarce. Normally I wouldn't say this about Apple, because their pricing is part of their brand, but this time the demand (both by data-centers and prosumers) is the force majeure.

> because their pricing is part of their brand

I know people usually say that about Apple, but to be fair to them on this occasion they have not hiked up their prices yet because they are clearly at present still under some old deals that they did a good job negotiating.

However, of course, at some point Apple will run out of both inventory and old-pricing manufacturing capacity. Yes, I am fully expecting some sort of price-hike like has been seen everywhere else. I am not naïve.

When that time comes it will remain a financial calculation, Apple boxes on one side versus hosted-option-costs on another, in relation to my specific use-cases.

Ultimately I still blame the chip-hoarding hyperscalers though. :)

Re: GLM-5.2 – How to Run Locally

#224

Post by nijave »

bbor wrote: post #205
xrd wrote: post #2

So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970

I’m kinda lost here… do y’all really have machines in your houses with hundreds of gigs of RAM?? Am I just behind the times? The page advertises the 8-bit quant as taking ~800GB, which seems like it would require at least 3 consumer motherboards fully stacked w/ 4x64GB cards each. Maybe “locally” has slowly come to imply “…on your homelab”?

I got a 2U rackmount with 192Gi DDR4 for $1.1k USD in 2023. Around 1.5 yrs ago, server RAM could be had pretty cheap--especially slower LRDIMMs (I wanna say 512Gi DDR4 was <$500 USD). I checked a couple old ServeTheHome threads and seeing maybe around $50/32GB RDIMM although thought it was cheaper than that for a little while

Re: GLM-5.2 – How to Run Locally

#225

Post by nijave »

uberex wrote: post #45
xrd wrote: post #2

So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970

Funny I casually asked Gemini and it said 500k for unquantized with decent throughput.

That's fair for new hardware. You probably want to prompt "homelab" or "used hardware" to compare what's in this thread.

Re: GLM-5.2 – How to Run Locally

#226

Post by npodbielski »

Tepix wrote: post #132
__m wrote: post #43

Earlier quoted context omitted.

How fast will the hardware become outdated? Are there big improvements expected in the next 3 years?

I think there is a gap right now for running large models such as GLM 5.2 in Q4 or Q8. My hope is on Intel Crescent Island 480GB cards. Let‘s see how expensive they‘ll be.

480GB? Probably like 100k$ each? :D

Re: GLM-5.2 – How to Run Locally

#228

Post by Tepix »

Frannky wrote: post #108

There is a push from multiple directions at the same time: - new AI desktops with GB10s. They are relatively cheap and you can cluster them and load 1TB of VRAM - Nvidia, amd, intel, Cerebras etc pushing new hardware - oss models getting crazy good, like glm 5.2 - flash models getting very good like deepseek V4 flash - quantizations - harnesses being able to use different models (big for difficult stuff, small for gr…

My hope is on Intel Crescent Island with 480GB. I don't need 8x H200 performance (and cost), but I would like to run GLM 5.2 Q8.

Re: GLM-5.2 – How to Run Locally

#229

Post by edg5000 »

nijave wrote: post #220
edg5000 wrote: post #112

Earlier quoted context omitted.

Very cool. So it's not just about GPU VRAM which I incorrectly thought. I though you'd need 512 GB GPU VRAM. I don't think it cost only 2400; 512GB ram would be more expensive though back in the day. But not mortgage-grade 200.000 which I estimated myself (which assumed running in 100% VRAM; overkill for a single user probably).

Well, it's about GPU VRAM if you want something competitive with cloud-hosted offerings at the performance levels showing in benchmarks. This is a heavy quant with quality degradation and significantly lower performance. Cloud offerings are 80-200tk/sec versus single digit tk/sec. That said, I'm also surprised it runs at all locally. I do think it'd be painfully slow for anything interactive so you're relying on anot…

I see. So not quite usable apart for specific use cases. Maybe in a few years we'll see new hardware players and better prices.

Re: GLM-5.2 – How to Run Locally

#230

Post by Tepix »

mannanj wrote: post #33
gpm wrote: post #32

Earlier quoted context omitted.

The ram/gpu shortage won't last forever though. Moreover we can be pretty confident that long-term the prices will obey wrights law and come down in cost significantly (from the pre-shortage prices) as we learn to produce them more efficiently. LLM companies are valued as if they're going to have some enduring monopoly that they can extract money from... GLM-5.2 and similar models make that valuation very very questi…

> The ram/gpu shortage won't last forever though Don't underestimate the markets ability to remain irrational

What's the irrational part? There's sky high demand.
Post reply on HN