Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5.2 – How to Run Locally

unsloth.ai

Post reply on HN
Print view View original

251–260 of 328 posts

Re: GLM-5.2 – How to Run Locally

#251

Post by achrono »

Havoc wrote: post #154

I bet OpenAI and Anthropic hate the timing of glm 5.2. Kinda shows they have a headstart rather than a magic moat

Nope, GLM 5.2 is only the latest and greatest in a long line of open-weights models. There are even fully open source models that are comparable to o1-mini (OLMo), or almost-fully-open ones that are comparable to o3 (Nemotron).

I'm super grateful to the open labs (who, importantly, do not have the word 'Open' in their name), all the more so to the likes of Ai2.

There is no magic moat indeed. It is math, engineering and of course copious amounts of data (and the political maneuvering required to secure it, e.g. how most everyone has trained on Anna's Archive by this point).

Re: GLM-5.2 – How to Run Locally

#253

Post by [deleted] »

discordance wrote: post #151
segmondy wrote: post #84

I run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will bring that speed up to about 9tk/sec. I also have ok 32core epyc CPU, a better 64core would bring it up to about 11tk/sec. I did a budget build before the crazy hardware cost and I regret it everyday. Nevertheless, it's fantastic being able to run this model at h…

Running that full load is at least 600 W, so in a day ~14 kWh. At $0.2 a kWH, that would be $2.80/day or $1k a year of op-ex in electricity. Unless you really want privacy or the fuzzy feeling of owning your own, it’s cheaper, more convenient and has much faster tok/s if you pay a hyper scaler. That said, I do like the direction we are heading and look forward to seeing what host your own hardware we get in 2 years.

[deleted]

Re: GLM-5.2 – How to Run Locally

#254

Post by hajile »

easygenes wrote: post #60
__m wrote: post #43

Earlier quoted context omitted.

How fast will the hardware become outdated? Are there big improvements expected in the next 3 years?

M5 Ultra will ship before end of year, likely. Though with current RAM shortage, likely max spec will be 256GB and in short supply. In late 2027 or early 2028, Nvidia will release Vera Rubin DGX Spark, likely with double or better the performance of current Blackwell, though unclear if memory capacity will go up much from current 128GB. Two to four of those will run models like this decently. In 2028 we should expect…

It might be M6 Ultra and I think the real reason for stopping selling top-tier units was to avoid mid-generation price hikes and increasing demand for the more expensive next-gen systems that I assume will come with 512gb (maybe 1TB) of RAM and a massive markup to match.

Re: GLM-5.2 – How to Run Locally

#255

Post by fghorow »

cultofmetatron wrote: post #183
walrus01 wrote: post #166

Earlier quoted context omitted.

Not everyone lives in a place where electricity is $0.20 a kWh. For instance BC Hydro residential rates are $0.11 (CAD) for the first tier and $0.14 for the second tier of consumption in a month. At current exchange rate $0.14 CAD is $0.099 USD a kWh. Hydro Quebec is even cheaper. At a theoretical 6 tok/s, 86400 seconds in a day, approx 500,000 tokens of GLM5.2 output for 2 bucks a day seems like a pretty good bargai…

if only there was a magical place where geothermal and hydroelectric is ubiquitous and the weather is cold enough that no one is going to be complaining about free heating.

You aren't, perchance, from Iceland, are you?

Re: GLM-5.2 – How to Run Locally

#256

Post by hajile »

dannyw wrote: post #106
Computer0 wrote: post #47

Earlier quoted context omitted.

128 gb of much slower ram than Apple.

DGX Spark is ~273GB/s. That’s about M5 Pro territory, and twice as fast as the M5. You’d have to go to the M5 Max, or M3 Ultra, to get higher memory bandwidth than the Spark.

If you are trying to get more than 64gb of RAM or doing tons of inferencing, you're getting a Max or Ultra anyway.

Re: GLM-5.2 – How to Run Locally

#258

Post by SalariedSlave »

trollbridge wrote: post #235
zozbot234 wrote: post #138

Earlier quoted context omitted.

AIUI the llama.cpp implementation for this model is still quite half-baked due to missing the support for DSA sparse attention mechanism. This leads to running the model with a different mechanism that it has not been trained for, which has been shown to lead to lower quality and performance. Anyway, I think GLM 5.2 in many ways is not as interesting as DeepSeek V4 series, which uses an even more advanced attention m…

Particularly DeepSeek 4.1, which they appear to be A/B testing on the API and which also seems available on the free chat interface. It also has an input image modality, which is a game changer. The cheap Sinofrontier models have generally been lacking in this regard. Basically, Chinese competition is fierce - DeepSeek set the pricing tier, and the question for each lab now is how to justify charging a little more. M…

Competing and innovating in the fast moving SOTA end of the llm space requires a ruthless disregard for copyright, IP, bureaucracies, formalities, risk assurances and other slowdowns. It requires a risk tolerant, quick and large flowing investment of capital. It requires a scoped focus that is pragmatic and sharp about key concerns, and efficiently dismissive of meaningless details.

Europe can provide none of this. They will never be at the frontier of AI tech, for the same reason they were never at the frontier of any tech.

I say this as a software engineer from Europe.

Re: GLM-5.2 – How to Run Locally

#259

Post by reasonabl_human »

antirez wrote: post #234

DwarfStar work in progress numbers: I see 14 tokens/sec generation, that slopes to 10 t/s with longer 10k or more context size. Consider that the indexed attention requires evaluating 2048 selected rows, 2x DeepSeek and with less compression, so the performances with larger contexts here to south faster. Prefill can be 180 t/s on small contexts to 150 t/s and less with larger contexts. I used DeepSeek v4 PRO in this…

Thank you for your work on DwarfStar! It is truly helping democratize access to frontier tech.

Re: GLM-5.2 – How to Run Locally

#260

Post by hajile »

stingraycharles wrote: post #181
3stacks wrote: post #54

Earlier quoted context omitted.

Maybe there's a conversation to be had about how much is enough... Unless something beyond my imagination happened, I would be happy enough with Opus 4.5 levels of productivity

This really sounds like “640kb should be enough”. I’m sorry, but I just can’t imagine us running smaller models than we are using right now in 5-10 years from now.

We've already hit RAM power and size limits (about 40k electrons which is the limit before we get noise messing up the amplifier).

If a model needs 2x more memory, but serves the same number of customers, the cost is going to go up per customer to cover the increased hardware and power costs. Companies are starting to implement AI limits to keep costs under control.

Anthropic and OpenAI are rumored to be considering cutting inference prices trying to retain customers as LLMs commoditize and race to the bottom. It reminds me of the Chinese bike wars where bike-share companies were losing massive amounts of money, but kept running sales and lowering prices in an attempt to compete and drive out their competitors. The end of that story was a bunch of major bankruptcies and giant bike graveyards.

Nvidia's hard pivot to "in the near future, everyone will run their AI at home" seems to indicate that they also see the market shifting. We've already had AI ingest everything out there. The real challenge becomes how to better optimize their algorithm to get more useful data in less space.

Post reply on HN