Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

211–220 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#211

Post by swiftcoder »

somenameforme wrote: post #206
swiftcoder wrote: post #190

Earlier quoted context omitted.

Hetzner will also rent you 768 GB of RAM with a Blackwell 6000 Max Q GPU for €2300/month [1]. Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs. [1]: https://www.hetzner.com/dedicated-rootserver/g…

Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me. And yeah it depreciates, but not to zero. So if you're speaking of the breakeven point after liquidation, you're probably there in well under a year at those rental prices.

> Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me

And yet basically all AWS customers are doing exactly that. Turns out that making CAPEX "someone else's problem" is worth quite a lot to many businesses

Re: Show HN: Getting GLM 5.2 running on my slow computer

#213

Post by jannyfer »

lukaslalinsky wrote: post #185
lopatin wrote: post #176

After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way. Some things that give it away to me: - "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)" - "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires." - "Honest caveat f…

That was their headline for Opus 4.8, I guess the invested into some post-training to get them to write this, and I like it, it's a great way to identify AI posts from Claude. https://www.anthropic.com/news/claude-opus-4-8

Yup, and I had seen it in the Opus 4.5 soul document as well: https://gist.github.com/Richard-Weiss/efe157692991535403bd7e...

In the future, I expect different models from different firms behaving differently will become as obviously normal as different humans behaving differently. Those of us that have used agents a lot have already noticed this, but the general public still seems to consider AI to be plug-and-play.

The Opus 4.8 infatuation with being honest genuinely drove me nuts (honest take). The constant need for me to decipher and decide on something after its final message was tiring too.

GPT-5.6-Sol is refreshing that its replies start with "understood." instead of flattering me for my steering prompt. Now ... after a few weeks, will I get suck of "understood"? I dunno :)

Re: Show HN: Getting GLM 5.2 running on my slow computer

#214

Post by culi »

lopatin wrote: post #176

After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way. Some things that give it away to me: - "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)" - "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires." - "Honest caveat f…

Claude is listed as a contributor right there

Re: Show HN: Getting GLM 5.2 running on my slow computer

#215

Post by Cieric »

Roxxik wrote: post #182
Cieric wrote: post #26

I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet. To expand since I ju…

Not sure if mmapping is the right way. In my own tests I noticed that simple mmapping will produce many small reads and not keep the SSD queue saturated. So if RAM is large enough to cache most experts, that is a rounding error. But if the base weigths without experts fill more than half or RAM and you basically need to load in a few experts for each layer of each token, the latency gets important and mmap sadly bloc…

On the technical details of mmap I agree (at least while single threaded which is how I believe I'm running it), but making it async does sound like an interesting method for speeding it up. My only goal with my testing was get as large of a model as possible running on a computer that "can't run it." I'm going to have to actually read through the code and figure out how it really works to really make any good optimizations, but as before this was just experiments with using and LLM (claude + codex) to just get it running. Since if they couldn't get it running I'm not sure if I would have wanted to spend time trying to get it working myself.

I also know I did some things that would actually make the perf worse to, like I believe I also had AI mmap the KV Cache to make sure to runs under any circumstance. For actual optimizations based on what I currently know, I'm probably going to try and get the llm running under my igpu on my laptop with persistent shader that has some kind of inbuilt request mechanism. That way the weights that are loaded can be used as fast as possible.

For the expert prediction, I assume I could use the medusa paper as kind of a kick off point for that since I'm already using it to try and predict the next 4 tokens. Doing verification on those 4 tokens is about as much as I can do though since it started to thrash on loading the experts. So some method of predicting even more tokens, but then batching together those with the same experts would probably yield slightly better results in this weird case.

Note: All of my tests have been around programming since that's the use case I'm interested in. I don't actually know if this would preform well in other cases (and anything more broad than that I assume would be slower.)

Re: Show HN: Getting GLM 5.2 running on my slow computer

#216

Post by Cieric »

dicroce wrote: post #88
Cieric wrote: post #26

I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet. To expand since I ju…

This is the approach I was wondering about.

Let me know if you want to hear anything specific about it. It kind of works, so it's not something I recommend doing if it can be avoided, but as Roxxik pointed out there is much room for improvement since this was just a naive just get it to run experiment.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#217

Post by arm32 »

Archit3ch wrote: post #52

Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.

Link it already!

Re: Show HN: Getting GLM 5.2 running on my slow computer

#219

Post by wincy »

codazoda wrote: post #74
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.

Time to make EmailGPT
Post reply on HN