Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5.3 is now open-weight

huggingface.co

Post reply on HN
Print view View original

251–260 of 296 posts

Re: GLM-5.3 is now open-weight

#251

Post by solarkraft »

frigidwalnut wrote: post #113
solarkraft wrote: post #93

Earlier quoted context omitted.

So far I don’t regret buying an M1 Max device with 32Gb of RAM. The models available for it keep getting better (running just about okay for interactive use) and 400 GB/s of bandwidth is still considered a lot. The models are currently improving much faster than the hardware and this doesn’t seem to have plateaued yet.

Cool! I'm thinking about a local set up. What's your usual tokens/second rate?

NGL: I don’t really have a good way to find out right now. It also doesn’t matter that much because the way the models use the tokes varies a lot. Qwen 3.8 is known for overthinking while Muse Glimmer may be a little slower per token, but it uses them very efficiently, caveman style.

Generation speed isn’t the bottleneck anyway, at least on pre M4/M5 devices (the newer chips got significant processing acceleration). It’s prompt processing time. OpenCode’s system prompt can take up to 3 minutes to process, which is why good prompt caching is essential.

For that I use omlx, which can persist the KV cache to disk, chunked so you can reuse parts. This helps with the usability a lot, when an agentic session is warm it runs pretty smoothly. New requests can take a couple seconds (sometimes many, which must be fixable somehow).

So: It’s not fast, but I also don’t find it awfully slow. My use is typically semi-interactive, for fully interactive use you have to wait a bit, but it’s possible. I personally am still regularly amazed that something even close to this is possible on completely local hardware.

Re: GLM-5.3 is now open-weight

#252

Post by pprotas »

revolvingthrow wrote: post #16

GLM 5.3 is probably the sweet spot open weights model if you want to go beyond deepseek flash or the new glm flash. I used it with pi and had a fairly good time, especially since it’s less touchy about cyber and whatnot than the US guys. It’s slightly behind Kimi in ability but it’s a lot easier to run it, I’d expect prices (and speed!) from third parties to be noticeably better. Assuming you’re willing to drop a fat…

”less touchy” is an understatement, it happily complies with running exploits, reverse engineering and decompiling. Asking Claude to do that will give you an error and make you feel like you’re about to get banned.

Re: GLM-5.3 is now open-weight

#254

Post by nullbio »

jtrn wrote: post #248

So basically, we have a clear path to local, viable LLMs in the same ballpark as current frontier LLMs. - Get an M5 Ultra with 512 GB of memory. - A couple of generations of improvements to the base model training. - A couple of generations of improvements to the architecture for running it as efficiently as possible. - A couple of generations of open coding agent harnesses like Pi and OpenCode. And given how fast ea…

The first company to release solid open-weight models on hardware LLMs to general consumers will change the world. I don't see a good reason to invest in something like an M5 ultra when it'll be so unbearably slow to generate tokens. Imagine getting stuck with something doing 50 TPS when everyone else is doing 2000-10,000 TPS (10k TPS is what the latest chips hit, I'm not pulling that number out of thin air).

Re: GLM-5.3 is now open-weight

#256

Post by alexpotato »

rkangel wrote: post #75
sickcodebruh wrote: post #73

Earlier quoted context omitted.

When we consider: * LLM usage is new for the world * Models are evolving quickly with high worldwide competition * Hardware is evolving despite RAM shortages Is investing a huge sum of money in equipment for local inference a wise use of money? Or are M5 Ultra and equivalently priced local inference hardware future-proof enough to be worth it relative to how the market is evolving? Maybe it’s all a question of what y…

It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years. However there are other reasons (e.g. privacy) that might make it worth running locally for some people.

> It is absolutely not worth buying hardware to run models for purely (long term) cost reasons

This is especially true when it's trivial to have the LLM itself write you a script/tool that can rent a GPU node for you (via API calls to providers) and then download and set up an open weight model for you.

Re: GLM-5.3 is now open-weight

#257

Post by afiori »

tough wrote: post #100
futureshock wrote: post #38

Earlier quoted context omitted.

I think it would be an important historical document as well. We are potentially looking at the dawn of AGI and one of the most important models ever created. Each model is also a kind of ultimate time capsule, containing a snapshot of the entire human collective mind. If you wanted to ask a 2002 person what they thought about future historical events you can just ask them directly.

> If you wanted to ask a 2002 person what they thought about future historical events you can just ask them directly. The weights arent the truth tho, maybe a timecapsule-vhs but i wouldnt trust llm weights more than more hardcore deterministic media that might get preserved to infer facts from an era. The companies doing the training are becoming the "winners" that are "rewriting history" as they train their models.

True, but in a sense a single frontier model of today would be of incredible historical significance in the far away future.

Re: GLM-5.3 is now open-weight

#258

Post by lostmsu »

nolist_policy wrote: post #240
lostmsu wrote: post #232

Earlier quoted context omitted.

> GLM 5.3 Flash at Q4, AA score 57 That AA score is for the original model only

Then take DeepSeek V4 flash with AA score 52. Runs unquantized on 2x DGX spark with 1M context.

5 points is a huge regression

Re: GLM-5.3 is now open-weight

#259

Post by onesandofgrain »

MaxikCZ wrote: post #255
onesandofgrain wrote: post #150

Man, imagine if we didn't have China.

Its more about "man, imagine if we didnt have competition", which is what the us is trying to eliminate by soon labeling all openweight llms as dangerous and illegal.

I dream of a future where I can run qwen26b or glm5.3 on my consumer gpu

Re: GLM-5.3 is now open-weight

#260

Post by somenameforme »

millicentricism wrote: post #246
somenameforme wrote: post #244

Earlier quoted context omitted.

I think that's overly pessimistic. Here's [1] a video of somebody running it on a ~$6000 rig and getting around 14T/s for complex prompts (about double that for simpler prompts). Payback time is going to depend on your electric cost/consumption. In most domains cloud providers end up charging a significant premium rather than a offering a scale enabled discount, relative to local at retail costs. That will almost cer…

With roughly 2.7 million seconds per month, times 14 tokens per second, you are getting 38.5 million tokens a month at most. That’s less than 164USD worth of GLM5.3 tokens on the inference market. So that 6000 USD rig will take 3 years to break even - and only if it runs continuously. And this is being generous, as it’s not even taking quantisation into account.

I think if you steel-man what I'm saying, what you're saying falls apart. 14 tokens per second was rare. It only dropped that low in one scenario where he had it single shot an entire game (flappy bird clone) from scratch, with different assets, all self created, and so on. It ended up resulting in the LLM doing stuff like plotting out a some odd 100 item long to-do list, requerying it repeatedly, and so on. And it succeeded.

Also as the video mentions, the guy wasn't very familiar with what he was doing, and so there are almost certainly various optimizations on the config side he could work out, especially as he was using a 5 GPU system, which default configs are probably not well optimized for.

But I think we've rapidly moving along the same path as image gen stuff. Local generation has gone from purely theoretic, to requiring supercomputers to run relatively incapable models, to where we are today - where with a fairly basic high end setup, he's comfortably running a frontier level model. There's definitely an argument for going local that's only growing stronger by the day.

Post reply on HN