Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5.2 – How to Run Locally

unsloth.ai

Post reply on HN
Print view View original

261–270 of 328 posts

Re: GLM-5.2 – How to Run Locally

#261

Post by nomel »

AbsurdCensor wrote: post #189
walrus01 wrote: post #178

Earlier quoted context omitted.

That's true, there's a lot of places where power is considerably more expensive than $0.20 USD/kWh. But also the 600W figure assumes that it's fully loaded 24x7x365. Running a system that will be 600W under max CPU usage on all cores and RAM and a few 3090-class GPUs, that same system might be only 90W or around there when idle at 0.00 unix load. If we say: (600 * 24 * 31)/1000 = 446kWh in a month at full load 24 hou…

I think that's the biggest difference for most. If you can amortize the hardware costs, then 'burst usage' is cheaper at home to a degree, because you are paying a fixed monthly rate elsewise. Overall thought for most, it is likely cheaper to use the cloud than at home, but really depends on what you want.

> because you are paying a fixed monthly rate elsewise

No, you would pay usage based rates with API, in this case. I have exactly one fixed monthly rate for the 6 AI models I have tokens available for.

Re: GLM-5.2 – How to Run Locally

#262

Post by nomel »

cultofmetatron wrote: post #183
walrus01 wrote: post #166

Earlier quoted context omitted.

Not everyone lives in a place where electricity is $0.20 a kWh. For instance BC Hydro residential rates are $0.11 (CAD) for the first tier and $0.14 for the second tier of consumption in a month. At current exchange rate $0.14 CAD is $0.099 USD a kWh. Hydro Quebec is even cheaper. At a theoretical 6 tok/s, 86400 seconds in a day, approx 500,000 tokens of GLM5.2 output for 2 bucks a day seems like a pretty good bargai…

if only there was a magical place where geothermal and hydroelectric is ubiquitous and the weather is cold enough that no one is going to be complaining about free heating.

The largest geothermal plant in the world is only 1.5GW, in the United States, which is over double all the plants combined in Iceland. The second largest is 1/3 that, in Mexico. [1]

There is no "ubiquitous" geothermal where there also high power usage. Data centers have to go where power is, not can be.

[1] https://en.wikipedia.org/wiki/List_of_geothermal_power_stati...

Re: GLM-5.2 – How to Run Locally

#263

Post by leansensei »

SalariedSlave wrote: post #258
trollbridge wrote: post #235

Earlier quoted context omitted.

Particularly DeepSeek 4.1, which they appear to be A/B testing on the API and which also seems available on the free chat interface. It also has an input image modality, which is a game changer. The cheap Sinofrontier models have generally been lacking in this regard. Basically, Chinese competition is fierce - DeepSeek set the pricing tier, and the question for each lab now is how to justify charging a little more. M…

Competing and innovating in the fast moving SOTA end of the llm space requires a ruthless disregard for copyright, IP, bureaucracies, formalities, risk assurances and other slowdowns. It requires a risk tolerant, quick and large flowing investment of capital. It requires a scoped focus that is pragmatic and sharp about key concerns, and efficiently dismissive of meaningless details. Europe can provide none of this. T…

Europe was never at the frontier of any tech? Huh what now?

Re: GLM-5.2 – How to Run Locally

#264

Post by brailsafe »

walrus01 wrote: post #184
cultofmetatron wrote: post #183

Earlier quoted context omitted.

if only there was a magical place where geothermal and hydroelectric is ubiquitous and the weather is cold enough that no one is going to be complaining about free heating.

To be fair, Vancouver is such a magical place in terms of electrical cost, but the cost of living and real estate are otherwise through the roof, with decrepit and nasty (would need $100k in renovations immediately if you're not treating it as a teardown) single family detached homes on the east side of the city selling for 3.2 million.

Shhh don't forget we have a water shortage. But it is nice to have electricity wrapped into my relatively cheap basement suite rent ;)

Re: GLM-5.2 – How to Run Locally

#265

Post by SalariedSlave »

leansensei wrote: post #263
SalariedSlave wrote: post #258

Earlier quoted context omitted.

Competing and innovating in the fast moving SOTA end of the llm space requires a ruthless disregard for copyright, IP, bureaucracies, formalities, risk assurances and other slowdowns. It requires a risk tolerant, quick and large flowing investment of capital. It requires a scoped focus that is pragmatic and sharp about key concerns, and efficiently dismissive of meaningless details. Europe can provide none of this. T…

Europe was never at the frontier of any tech? Huh what now?

A hyperbole born of frustration, I admit.

Qualify it to software, rather than all tech, if you will.

Re: GLM-5.2 – How to Run Locally

#268

Post by leansensei »

scosman wrote: post #78
pheggs wrote: post #7

I feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I wrong about that?

It's not economic to run them locally. It's amazing for privacy, and fun hobby. But you're either looking at super slow CPU builds with $10k in RAM, $90k worth of GPUs, or a really quantized model that doesn't compare in quality. I might build one for fun, but it's not going to change the economics alone. Still exciting it's possible.

Not really, you can do great things without them. I've been summarizing hundreds of documents. I've added MCP servers to my internal business tools (Elixir apps) and can chat with the Nous Hermes agent over Telegram about pending orders, inventory level, historical product prices, etc., without having to click/dick around with a web UI.

Sure, it cannot replace SOTA models for agentic coding, except for small, well-scoped refactorings. But even a model like ministral-3:8b or qwen3.5:9b is a boon for so many smaller use cases!

Re: GLM-5.2 – How to Run Locally

#269

Post by iaw »

numlock86 wrote: post #125

Is this really worth it, though? Throughout the years my experience with quantized models has been that they feel like a lobotomized version of the original. Doesn't matter if it's an LLM, dedicated diffusion model or some other dedicated task. Sure, they get the job done. But a lot worse. The only ones that can somewhat hold up are the ones provided by the vendor directly. Gemma4 comes to mind. However I suspect the…

On a model of this size quantization has much less impact on quality of output. I'm running a 3bit version and find it comparable to sonnet, almost opus.

Re: GLM-5.2 – How to Run Locally

#270

Post by iaw »

zkmon wrote: post #162

I have high respect for unsloth's work, helping millions to get started with local AI, but this post appears kind of download bait. Offloading too many layers to CPU is not going to work at all. I have tried this many times and had to rm -rf on those heavy hf cache folders. Also I doubt 1-bit or 2-bit quants of GLM 5.2, running mostly outside of VRAM can beat Q8_0 of Qwen3.6-27B fully loaded in VRAM - on usefulness.

I run 3bit GLM5.2 and full precision Qwen3.6-27B. GLM is much much closer to frontier models in it's breadth and ability to plan. If you just need to implement Python code from an existing plan Qwen is your choice but it has problem succeeding with more complex tasks that GLM5.2 does not.

As I type this my local GLM5.2 is troubleshooting bugs that Qwen would not be able to handle.

Post reply on HN