Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

181–190 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#181

Post by swiftcoder »

bigiain wrote: post #72
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

> on hardware that ordinary people can afford These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd? sigh

> These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?

You can, right now, buy a brand new Mini-PC at or above this spec for $600 at retail [1]

Of course, if you want it in a desktop format with a much faster CPU, its going to cost you more.

[1]: https://www.amazon.com/GMKtec-M6-Ultra-Upgraded-Computers/dp...

Re: Show HN: Getting GLM 5.2 running on my slow computer

#182

Post by Roxxik »

Cieric wrote: post #26

I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet. To expand since I ju…

Not sure if mmapping is the right way. In my own tests I noticed that simple mmapping will produce many small reads and not keep the SSD queue saturated. So if RAM is large enough to cache most experts, that is a rounding error. But if the base weigths without experts fill more than half or RAM and you basically need to load in a few experts for each layer of each token, the latency gets important and mmap sadly blocks until the data is loaded. You can't do concurrent requests for multiple experts with mmap (but you know all the ones you need right after the router ran). And even going one step further, depending on the arch / the tensors you need, you could eagerly load some, start computing with them and load the rest of the expert tensors in the background (extra thread or async io) parallel to the compute. This is not really possible with mmap, even with madvise.

One further step is predicting which experts will be needed next token / next layer. LRU does this okish. But a learned projection from the hidden state can do better. Or even a simple correlation from past activated experts. Expert usage is heavily skewed.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#183

Post by FeepingCreature »

xyzsparetimexyz wrote: post #160
kristianp wrote: post #130

Earlier quoted context omitted.

I you look at https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.

For purely coding tasks, is every single expert required? 50%? 25%?

Yes, every expert. It's not like different personas, more like very coarse dropout training.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#185

Post by lukaslalinsky »

lopatin wrote: post #176

After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way. Some things that give it away to me: - "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)" - "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires." - "Honest caveat f…

That was their headline for Opus 4.8, I guess the invested into some post-training to get them to write this, and I like it, it's a great way to identify AI posts from Claude.

https://www.anthropic.com/news/claude-opus-4-8

Re: Show HN: Getting GLM 5.2 running on my slow computer

#187

Post by walrus01 »

swiftcoder wrote: post #177
sandworm101 wrote: post #166

Earlier quoted context omitted.

Not just controversial but also regulated areas. Virtually every law firm would be interested on locally-hosted AI at a reasonable price. So too ever medical research lab. Every CGI firm doing work for film/TV. And all the video game developers.

Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...

For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it.

The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload.

The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math on one of these and it costs $12 per hour, or $289 a day, or $8900+ for one month.

Also plenty of Europeans or people from other locations may consider it as an unacceptable risk factor to put their "off site" self hosted AI stuff with an American controlled company. Particularly if the servers are physically in the USA.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#188

Post by efficax »

sakesun wrote: post #77

I just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.

the math involved is not very hard to understand. it’s linear algebra. the transformer model is brilliant but simple, nobody even really realized the impact it would have until they started training it on massive datasets

Re: Show HN: Getting GLM 5.2 running on my slow computer

#189

Post by xyzsparetimexyz »

FeepingCreature wrote: post #183
xyzsparetimexyz wrote: post #160

Earlier quoted context omitted.

For purely coding tasks, is every single expert required? 50%? 25%?

Yes, every expert. It's not like different personas, more like very coarse dropout training.

Right. So it's not like human experts or even different brain regions

Re: Show HN: Getting GLM 5.2 running on my slow computer

#190

Post by swiftcoder »

walrus01 wrote: post #187
swiftcoder wrote: post #177

Earlier quoted context omitted.

Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...

For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it. The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload. The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math…

Hetzner will also rent you 768 GB of RAM with a Blackwell 6000 Max Q GPU for €2300/month [1].

Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs.

[1]: https://www.hetzner.com/dedicated-rootserver/gex131/

Post reply on HN