Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5.2 – How to Run Locally

unsloth.ai

Post reply on HN
Print view View original

111–120 of 328 posts

Re: GLM-5.2 – How to Run Locally

#111

Post by jiqiren »

easygenes wrote: post #60
__m wrote: post #43

Earlier quoted context omitted.

How fast will the hardware become outdated? Are there big improvements expected in the next 3 years?

M5 Ultra will ship before end of year, likely. Though with current RAM shortage, likely max spec will be 256GB and in short supply. In late 2027 or early 2028, Nvidia will release Vera Rubin DGX Spark, likely with double or better the performance of current Blackwell, though unclear if memory capacity will go up much from current 128GB. Two to four of those will run models like this decently. In 2028 we should expect…

I hope all this speculation comes true. Right now this ram crunch is ridiculous.

Re: GLM-5.2 – How to Run Locally

#112

Post by edg5000 »

segmondy wrote: post #84

I run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will bring that speed up to about 9tk/sec. I also have ok 32core epyc CPU, a better 64core would bring it up to about 11tk/sec. I did a budget build before the crazy hardware cost and I regret it everyday. Nevertheless, it's fantastic being able to run this model at h…

Very cool. So it's not just about GPU VRAM which I incorrectly thought. I though you'd need 512 GB GPU VRAM. I don't think it cost only 2400; 512GB ram would be more expensive though back in the day. But not mortgage-grade 200.000 which I estimated myself (which assumed running in 100% VRAM; overkill for a single user probably).

Re: GLM-5.2 – How to Run Locally

#113

Post by digitaltrees »

colinsane wrote: post #77
uberex wrote: post #45

Earlier quoted context omitted.

Funny I casually asked Gemini and it said 500k for unquantized with decent throughput.

i asked gemini and it replied with "Error: 400 Your prompt was blocked by safety filters. Please revise and try again."

I asked and it said “403 forbidden - careful peon attempts to bypass the late stage capitalism api with your monetary offerings in exchange for you daily tokens will get you perma banned right to jail”.

Re: GLM-5.2 – How to Run Locally

#114

Post by nextaccountic »

segmondy wrote: post #84

I run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will bring that speed up to about 9tk/sec. I also have ok 32core epyc CPU, a better 64core would bring it up to about 11tk/sec. I did a budget build before the crazy hardware cost and I regret it everyday. Nevertheless, it's fantastic being able to run this model at h…

How can you combine CPU cores and multiple GPU? Are you running some layers in cpu, others in gpu #1, and others in gpu #2? What about the bandwidth and latency between them?

Or maybe the model itself only runs at gpus, and the cpu memory only store the weights for experts not corrently activated? If so, then what's the 32 or 64 cpu cores for?

I'm a big fan of fully utilizing one's hardware and it's kinda sad that it's not the norm to run things on either gpu, cpu or both, dynamically choosing at runtime, for everyday software

Re: GLM-5.2 – How to Run Locally

#115

Post by andy_ppp »

stymaar wrote: post #62
uberex wrote: post #45

Earlier quoted context omitted.

Funny I casually asked Gemini and it said 500k for unquantized with decent throughput.

This is why you shouldn't believe uncritically an answer from an LLM (neither should you do for any answer from a human either though).

But I did my research online and the sun cycle is every 11 years and something something global warming is a hoax every single year now.

Re: GLM-5.2 – How to Run Locally

#116

Post by throwdbaaway »

Wowfunhappy wrote: post #30

> The full model requires 1.51TB of disk space ...a bit of an odd question: how well do LLMs losslessly compress, as in for cold storage? I definitely don't have the hardware to run this model at any kind of reasonable speed (and I don't want to use a super aggressive quantization that would kill performance). Even so, I think it would be cool to retain an offline copy, in case... I don't really know, a solar flare d…

On ZFS with zstd compression, I am getting 1.34x compressratio for the BF16 weights (across multiple models).

Here's the du output for GLM-5.2:

    $ du -s -BG /cube/models/zai-org/GLM-5.2/
    1099G   /cube/models/zai-org/GLM-5.2/

Re: GLM-5.2 – How to Run Locally

#117

Post by drudolph914 »

GLM 5.2 is the first time I'm actually excited about AI! I'm not the most bullish on AI code for several few reasons, but the biggest reason is the ownership model. We all know we're near the tail end of the "subsidized pricing" window for AI, and I've been hoping for so long to get an open weight model that is _close enough_ to the SOTA before this window closes - and we actually got it! I'm excited to be able to in the near future run GLM locally, and use these things like a tool instead of living in this for-rent model for the rest of my life. I'm excited to actually enjoy programming again

Re: GLM-5.2 – How to Run Locally

#118

Post by BobbyTables2 »

ijidak wrote: post #68
xrd wrote: post #2

So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970

Crossing my fingers that this boom jumpstarts 90's like improvements in computing hardware. I feel like part of the reason for the relative stagnation in hardware over the last twenty years was simply the lack of use cases to justify hardware refreshes by businesses. Most of the money and energy went to mobile for the last fifteen years. Affordable local inference might be the gravy train the server, desktop, and lap…

Yeah, even Windows managed to not drive terribly dramatic upgrades in general computing (besides Windows’ absurd RAM usage and now requiring a TPM).

In the old days, Microsoft Entertainment Pack games were somewhat visibly taxing on some lower end systems.

Re: GLM-5.2 – How to Run Locally

#119

Post by SlavikCA »

phamilton wrote: post #51
cheema33 wrote: post #5

Earlier quoted context omitted.

I have the RAM, but not the VRAM. What kind of speed/tps could you expect from a 3090 with 24GBs of RAM? I am somewhat tempted to pick a GPU with 24GBs of RAM.

Generation is basically just memory bandwidth math. Each token has to read all the active weights. I think that's around 40B parameters active. At a 4-bit quant that's 20GB. With 100GB/s (replace with whatever your bandwidth is) and you get 5 tokens per second.

And with MTP (or other speculation techniques) you can ~double that.

Re: GLM-5.2 – How to Run Locally

#120

Post by uberex »

segmondy wrote: post #91
hxii wrote: post #50

Any time I see one of these posts about models of this size a quote comes to mind – "Your Scientists Were So Preoccupied With Whether Or Not They Could, They Didn’t Stop To Think If They Should". Only a select few have the hardware required to run this to begin with, and even then the forecasted performance makes me wonder if it’s worth it at all.

Completely worth it. At 6tk a second. If I can get 2 hrs of token generation. That's 2hrs * 3600secs * 6tk = 43200 tokens, at about 10tk to a line of code, that's about 4320 lines. Let's even trim it more and slice it by half. That's 2160 lines of code a day. Most professional programmers can't deliver that much consistently in a day. The key to a model this large is (1) Use it to plan, generate lots of plan and farm…

Thats not a complete reasoning. Even frontiers need to revisit and fix things. Add 10 loops to that and it is 20 hours. Still great compared to a 2023 human, but why am I not just paying pocket money for Claude Pro instead?
Post reply on HN