Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5.2 – How to Run Locally

unsloth.ai

Post reply on HN
Print view View original

11–20 of 328 posts

Re: GLM-5.2 – How to Run Locally

#11

Post by cogman10 »

pheggs wrote: post #7

I feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I wrong about that?

I don't think so. I could easily see a company deciding to host and run these models for their own development. If you have a dev team of about 10 people, a one time $50k investment in an LLM server has to be pretty tempting. Unlimited tokens, decent performance, upgrade options, and potential product integrations.

For companies wanting LLMs in their products in general, I have to think going the local llm route is even more tempting. Somewhat dumb models are more than good enough for a lot of the things people are integrating LLMs into their products.

Re: GLM-5.2 – How to Run Locally

#12

Post by kccqzy »

zuzululu wrote: post #6

wonder if AMD's new ai chip can run this with ease? I'm seriously consider buying it. GLM 5.2 is just shy of GPT 5.4 so I would welcome offloading any grunt work locally I am very excited for local LLMs I think we may have GPT 5.5-xhigh level of performance for under 2000 EUR This should put more pressure on the frontier models to avoid sitting on any fancy stuff and lower token prices as a whole. Nothing beats a loc…

The AMD 395 supports up to 128GB unified RAM. So still not enough even at 1-bit quant unfortunately.

Re: GLM-5.2 – How to Run Locally

#13

Post by benjiro29 »

zuzululu wrote: post #6

wonder if AMD's new ai chip can run this with ease? I'm seriously consider buying it. GLM 5.2 is just shy of GPT 5.4 so I would welcome offloading any grunt work locally I am very excited for local LLMs I think we may have GPT 5.5-xhigh level of performance for under 2000 EUR This should put more pressure on the frontier models to avoid sitting on any fancy stuff and lower token prices as a whole. Nothing beats a loc…

"GLM 5.2 is just shy of GPT 5.4"... If your running the full model. As in have 750 (FP8) to 1.5TB(FP16) of memory available.

Do not mix the benchmark results of GLM 5.2 FP16/FP8 with FP4 or FP2.

* FP4 will mean a accuracy loss of about 3%. Not noticeable but more chance for mistakes.

* FP2 ... what is what most people are able to run at home, for a "reasonable" price. Your looking at over 17% loss in accuracy.

At that point, your running at less then claude-sonnet-4.6, as the issues compound with accuracy losses. And reasonable priced is still in the ~ $5000 range (192GB + GPU 32GB active/kv cache system).

For that price your using a Codex / Claude Pro subscription for the next 4+ years with better models (by default), let alone with a FP2 GLM 5.2 version. And your looking at Unfortunately the local hardware cost is a major issue for running large models like that.

Edit: Its funny whenever the issue of cost and what you need to give up vs the subscription services, there are always people who downvote in bad faith.

Re: GLM-5.2 – How to Run Locally

#14

Post by nh43215rgb »

zuzululu wrote: post #6

wonder if AMD's new ai chip can run this with ease? I'm seriously consider buying it. GLM 5.2 is just shy of GPT 5.4 so I would welcome offloading any grunt work locally I am very excited for local LLMs I think we may have GPT 5.5-xhigh level of performance for under 2000 EUR This should put more pressure on the frontier models to avoid sitting on any fancy stuff and lower token prices as a whole. Nothing beats a loc…

Even with upcoming AI Max+ PRO 495 we are capped with 192GB, so no...

Re: GLM-5.2 – How to Run Locally

#15

Post by yieldcrv »

fny wrote: post #8
pheggs wrote: post #7

I feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I wrong about that?

The RAM requirements are still pretty painful.

equilibrium in one or two more years on the consumer/prosumer side

think Apple M6 or M7 with a currently unforeseen denser memory style, 256gb RAM

a couple inference or cache improvements on the algorithmic side, using less ram for context windows and doubling token speed again

denser open source models, packing more experts for smaller active layers

it'll still be expensive but like $8,000 - $13,000 instead of $450,000 worth of B200s

Re: GLM-5.2 – How to Run Locally

#17

Post by elliotbnvl »

xrd wrote: post #2

So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970

$500k is a vast overestimation. For massive concurrency at FP8 or even BF16 maybe.

NVFP4 at reasonable speeds (~120 tok/s) and concurrency is possible at a $80/90k figure with today's prices, maybe even less. That buys you 6 RTX 6000 PRO Blackwells, a decent CPU and motherboard, power supply. 576gb of VRAM.

You could do it for under $50k if you're OK with 40 tok/s decode, ~1200 tok/s prefill.

Re: GLM-5.2 – How to Run Locally

#18

Post by eventualcomp »

cogman10 wrote: post #11
pheggs wrote: post #7

I feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I wrong about that?

I don't think so. I could easily see a company deciding to host and run these models for their own development. If you have a dev team of about 10 people, a one time $50k investment in an LLM server has to be pretty tempting. Unlimited tokens, decent performance, upgrade options, and potential product integrations. For companies wanting LLMs in their products in general, I have to think going the local llm route is e…

Where is $50k coming from again?

Re: GLM-5.2 – How to Run Locally

#19

Post by stingraycharles »

eventualcomp wrote: post #18
cogman10 wrote: post #11

Earlier quoted context omitted.

I don't think so. I could easily see a company deciding to host and run these models for their own development. If you have a dev team of about 10 people, a one time $50k investment in an LLM server has to be pretty tempting. Unlimited tokens, decent performance, upgrade options, and potential product integrations. For companies wanting LLMs in their products in general, I have to think going the local llm route is e…

Where is $50k coming from again?

That’s less than the monthly salary of 10 software engineers, and assuming they pay API prices, probably earns itself back in about a year.

Having said that, I don’t think it’s all that tempting for companies at all, considering this whole market is developing rapidly and it’s nearly impossible to predict where we’ll be at in a year or two.

Re: GLM-5.2 – How to Run Locally

#20

Post by stingraycharles »

yieldcrv wrote: post #15
fny wrote: post #8

Earlier quoted context omitted.

The RAM requirements are still pretty painful.

equilibrium in one or two more years on the consumer/prosumer side think Apple M6 or M7 with a currently unforeseen denser memory style, 256gb RAM a couple inference or cache improvements on the algorithmic side, using less ram for context windows and doubling token speed again denser open source models, packing more experts for smaller active layers it'll still be expensive but like $8,000 - $13,000 instead of $450,…

Fairly certain that model sizes and computational requirements will grow as the price for LLM compute drops.
Post reply on HN