if sam altman didnt exist i could afford to run this
GLM-5.2 – How to Run Locally
61–70 of 328 posts
Re: GLM-5.2 – How to Run Locally
#62So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970
Funny I casually asked Gemini and it said 500k for unquantized with decent throughput.
This is why you shouldn't believe uncritically an answer from an LLM (neither should you do for any answer from a human either though).
Re: GLM-5.2 – How to Run Locally
#63wonder if AMD's new ai chip can run this with ease? I'm seriously consider buying it. GLM 5.2 is just shy of GPT 5.4 so I would welcome offloading any grunt work locally I am very excited for local LLMs I think we may have GPT 5.5-xhigh level of performance for under 2000 EUR This should put more pressure on the frontier models to avoid sitting on any fancy stuff and lower token prices as a whole. Nothing beats a loc…
"GLM 5.2 is just shy of GPT 5.4"... If your running the full model. As in have 750 (FP8) to 1.5TB(FP16) of memory available. Do not mix the benchmark results of GLM 5.2 FP16/FP8 with FP4 or FP2. * FP4 will mean a accuracy loss of about 3%. Not noticeable but more chance for mistakes. * FP2 ... what is what most people are able to run at home, for a "reasonable" price. Your looking at over 17% loss in accuracy. At tha…
you are right that means GLM is still quite far off from truly competitive
i think your answer was perfect not sure why you are being downvoted
Re: GLM-5.2 – How to Run Locally
#64wonder if AMD's new ai chip can run this with ease? I'm seriously consider buying it. GLM 5.2 is just shy of GPT 5.4 so I would welcome offloading any grunt work locally I am very excited for local LLMs I think we may have GPT 5.5-xhigh level of performance for under 2000 EUR This should put more pressure on the frontier models to avoid sitting on any fancy stuff and lower token prices as a whole. Nothing beats a loc…
Are you talking about Medusa Halo? It's going to support up to 256GB unified memory (up from 128GB for Strix Halo and 192GB for Gorgon Halo). That might just be barely enough to run a 2-bit quant GLM-5.2. It will expand memory bus to 384-bits, vs. 256-bits for Strix Halo which will help with bandwidth (projected to be around 500 GB/sec). But don't expect Madusa Halo-based machines to appear until sometime in 2028. Th…
yeah you are correct 2 bit quant won't be enough
guess we'll be paying $200/month for a while
Re: GLM-5.2 – How to Run Locally
#65Earlier quoted context omitted.
How fast will the hardware become outdated? Are there big improvements expected in the next 3 years?
M5 Ultra will ship before end of year, likely. Though with current RAM shortage, likely max spec will be 256GB and in short supply. In late 2027 or early 2028, Nvidia will release Vera Rubin DGX Spark, likely with double or better the performance of current Blackwell, though unclear if memory capacity will go up much from current 128GB. Two to four of those will run models like this decently. In 2028 we should expect…
[deleted]
Re: GLM-5.2 – How to Run Locally
#66I feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I wrong about that?
Honestly, Qwen3.6 is already what you need for the large majority of tasks.
(I only ask Opus every 5 to 10 requests, when my local Qwen fails or when I encounter a situation that is too world-knowledge specific to be worth asking, but that way you can live easily with Claude's cheapest plan without ever facing usage limit).
Re: GLM-5.2 – How to Run Locally
#67wonder if AMD's new ai chip can run this with ease? I'm seriously consider buying it. GLM 5.2 is just shy of GPT 5.4 so I would welcome offloading any grunt work locally I am very excited for local LLMs I think we may have GPT 5.5-xhigh level of performance for under 2000 EUR This should put more pressure on the frontier models to avoid sitting on any fancy stuff and lower token prices as a whole. Nothing beats a loc…
> I am very excited for local LLMs I think we may have GPT 5.5-xhigh level of performance for under 2000 EUR We are maybe 10 years off that. RAM prices are going to continue to increase for the next 2 years at least. Even putting that aside it's currently around 40-70,000 EUR to run this with a FP8 quantization (which you need to get close to maximum performance). To actually get GPT 5.5-xhigh performance in the real…
[dead]
Re: GLM-5.2 – How to Run Locally
#68So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970
Crossing my fingers that this boom jumpstarts 90's like improvements in computing hardware.
I feel like part of the reason for the relative stagnation in hardware over the last twenty years was simply the lack of use cases to justify hardware refreshes by businesses.
Most of the money and energy went to mobile for the last fifteen years.
Affordable local inference might be the gravy train the server, desktop, and laptop manufacturers need to get back in gear.
Re: GLM-5.2 – How to Run Locally
#69Earlier quoted context omitted.
The hardware requirements aren't evolving and the local models have only been improving. It's not like you'd lose capabilities, if anything this solution just gets better with time.
If the newer models require more/better hardware then you’ll lose capabilities. I think you’re better off renting GPU instances and running all the software on those. It’ll be cheaper than Anthropic and OpenRouter but slightly more expensive than electricity and depreciation of hardware.
The newer models don't require more/better hardware. There's a small army of local llm enthusiasts who are running LLMs using 3090s and H100s because they have lots of memory. Them being old isn't really that big of an issue as the compute power needed is relatively low all things considered.
The number of parameters needed for these open weight models has mostly stabilized so the actual memory requirements aren't likely to change all that much.
Re: GLM-5.2 – How to Run Locally
#70So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading. https://unsloth.ai/docs/models/glm-5.2#usage-guide In a prior thread, someone said it would take $500k in hardware: https://news.ycombinator.com/item?id=48629970
$500k is a vast overestimation. For massive concurrency at FP8 or even BF16 maybe. NVFP4 at reasonable speeds (~120 tok/s) and concurrency is possible at a $80/90k figure with today's prices, maybe even less. That buys you 6 RTX 6000 PRO Blackwells, a decent CPU and motherboard, power supply. 576gb of VRAM. You could do it for under $50k if you're OK with 40 tok/s decode, ~1200 tok/s prefill.
[dead]