Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5.2 – How to Run Locally

unsloth.ai

Post reply on HN
Print view View original

151–160 of 328 posts

Re: GLM-5.2 – How to Run Locally

#151

Post by discordance »

segmondy wrote: post #84

I run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will bring that speed up to about 9tk/sec. I also have ok 32core epyc CPU, a better 64core would bring it up to about 11tk/sec. I did a budget build before the crazy hardware cost and I regret it everyday. Nevertheless, it's fantastic being able to run this model at h…

Running that full load is at least 600 W, so in a day ~14 kWh. At $0.2 a kWH, that would be $2.80/day or $1k a year of op-ex in electricity.

Unless you really want privacy or the fuzzy feeling of owning your own, it’s cheaper, more convenient and has much faster tok/s if you pay a hyper scaler.

That said, I do like the direction we are heading and look forward to seeing what host your own hardware we get in 2 years.

Re: GLM-5.2 – How to Run Locally

#152

Post by idonotknowwhy »

c7b wrote: post #146

Can someone explain the math to me? Why is 1-bit only ten percent less memory than 2-bit?

2 reasons.

First, it's not really "1 bit", actually much closer to 2-bit. IQ1_M is actually 1.75bit and IQ2_XXS is 2.06bit This is from the ./llama-quantize --help with most of the quant types and their size in bpw: https://pastebin.com/bCUqGfeE

And to elaborate on the "dynamic" aspect inconito said in the other comment, if you click on one of the .gguf files in huggingface:

https://huggingface.co/unsloth/GLM-5.2-GGUF/blob/main/UD-IQ1...

There are a lot of Q5_K, Q6_K, etc tensors. Only the routed experts (ffn_gate_exps.weight, ffn_up_exps.weight, ffn_down_exps.weight) are heavily quantized, and it looks like the down_proj is actually iq3_xxs for this model.

Re: GLM-5.2 – How to Run Locally

#155

Post by ekidd »

cheema33 wrote: post #5
xrd wrote: post #2

So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970

I have the RAM, but not the VRAM. What kind of speed/tps could you expect from a 3090 with 24GBs of RAM? I am somewhat tempted to pick a GPU with 24GBs of RAM.

A GPU with 24GBs of RAM is mostly useful for running a very carefully squeezed Qwen3.6 27B (4-bit Unsloth quants, 8-bit K/V cache, possibly MTP, 128k context). This is a fun little model that's smart enough to do debugging, refactoring, and implementing "clean" specs that don't force it to make complicated design choices. I've seen it rip through a 9-year-old Terraform AWS config, and (without using the network) correctly identify nearly everything that would need to be upgraded or migrated for modern AWS. But if I give it some poorly conceived spec with lurking design headaches, then it goes on an endless thinking binge and ultimately fails.

Speed-wise, I don't have numbers, but it feels subjectively faster than Opus in Claude Code. YMMV.

Once you go above "a used 3090 at a decentish price", then I strongly recommend renting cloud GPUs or at least testing models using paid APIs. This allows testing your use case before spending piles of money.

Re: GLM-5.2 – How to Run Locally

#156

Post by uberex »

j45 wrote: post #74
uberex wrote: post #45

Earlier quoted context omitted.

Funny I casually asked Gemini and it said 500k for unquantized with decent throughput.

LLMs aren't discrete calcluators or estimators of things unless framed and guided to do so.

Good job I didn't use a vanilla LLM without tool use harness then.

Re: GLM-5.2 – How to Run Locally

#160

Post by walrus01 »

I really don't think anyone is going to have a good time trying to run it on anything with 256GB of RAM no matter what the post says. 512 is the much more realistic minimum. I'm fortunate enough to have two 512GB RAM dual xeon workstations in my home office that I bought cheap before the price rise to mess around with things...
Post reply on HN