Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

241–250 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#241

Post by vforno »

efficax wrote: post #194

Tried this out on my ThreadRipper: AMD Ryzen Threadripper PRO 5975WX — 32 cores / 64 threads, Zen3 (znver3), AVX2+FMA (no AVX-512/VNNI), 128GB RAM, Kingston SKC3000D 4TB NVMe (PCIe4). Disk gets around 7GB/s. It took a little tuning (for example pinning to 32 physical cores instead of the 64 threads), but with that and --topp 0.7, got 0.44 tok/s on a cold start. That's way below the estimates in the README, which I as…

Thanks for this benchmark have you used the last commit? We made some changes from mtp int 4 to much more! If you like, leave an issue so we can work on it! Thank you so much!

Re: Show HN: Getting GLM 5.2 running on my slow computer

#242

Post by fibonachos »

taneq wrote: post #234
fibonachos wrote: post #230

Earlier quoted context omitted.

I’ve started learning into the various AI-isms in my prompts. Most recently, em dashes to chain related information in a single sentence. “Something something main point — but also pay attention to this related thing — finish main point.” Anecdotally, it seems to produce more consistent results when I “speak its language”. On the output side, I’ve become much more aggressive about trimming and editing any prose it pr…

So you’re code switching to incorporate AI-specific ‘vocal fry’ in order to better communicate with it? That’s not just mimicry. It’s insightful. And it just might work. ;)

What’s interesting is that it maps well to how my brain actually works. My chain of thought is very non-linear. Maybe even a little chaotic, going off on tangents before returning to the main thread as it were. This has always made written and verbal communication a bit of a challenge, but the AI seems to parse my unedited stream-of-consciousness without issue.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#243

Post by inigyou »

walrus01 wrote: post #111
JumpCrisscross wrote: post #92

Earlier quoted context omitted.

The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028. (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)

For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise. Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.

This was the case a few years ago, but now the RAM costs twice that much by itself, and the server is sold without RAM.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#244

Post by arximboldi »

abe94 wrote: post #207
lopatin wrote: post #176

After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way. Some things that give it away to me: - "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)" - "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires." - "Honest caveat f…

I've been seeing a lot of usage of the word `real` from recent fable sessions: - That gap is the real story: 3,873 flows - the real conversion filter types are: - I'll update the breakdown query to include a column for each real type

also substantivizing everything and using "gate", "latch", at every turn. a normal person would say "add the pasta once the water boils" or "the water should be boiling when you add the pasta". Claude would say: water boiling state gates the pasta addition. it sounds obnoxious and engineery but it's so mentally tiring to read and decypher.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#245

Post by ukd1 »

xtracto wrote: post #69
Archit3ch wrote: post #52

Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.

I will be delighted to try. I have a 128gb macbookpro m4 waiting for this.

ds4 is pretty optimized for this btw - https://github.com/antirez/ds4

Re: Show HN: Getting GLM 5.2 running on my slow computer

#247

Post by kzrdude »

ykonstant wrote: post #236

I only have 16GB of RAM in my laptop, but I would love to run a powerful model like that even at 0.5 tokens/s. For the questions I am asking, such a speed is more than enough. If anyone has any suggestions, please let me know! I am not an expert on LLMs, just a humble mathematician.

It might be impossible to run such a large model on the laptop, if I were you I'd first start experimenting with running smaller models, using llama.cpp or lmstudio, just to start understanding how it works and learn about some of the concepts involved - quantization and all!

For example Qwen3.5-4B or Qwen3.5-9B could probably be run cpu-only on some quantizations at more reasonable speeds.

If you want a larger one, you could probably run Qwen3.6-27B or 35B-A3B decently slowly.

Now to be humble - maybe it's impossible to run colibri (this project in the OP) on your computer. Note the performance stories - 0.07 token/s on a 24 GB laptop with 24 CPU threads!

Re: Show HN: Getting GLM 5.2 running on my slow computer

#248

Post by kzrdude »

bahmboo wrote: post #46

Another recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn't use the gpu]. Not hijacking anyt…

How many tokens/s do people feel are enough? Even 5 tok/s would be hard to use in an interesting way, not sure what the use is.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#250

Post by embedding-shape »

Cieric wrote: post #227
BigTTYGothGF wrote: post #223

Earlier quoted context omitted.

> Currently at a stop light so I can't list everything and I didn't get to read your full post either yet. Posting on the Hacker News forum is not worth risking your, or anybody else's, life.

Being fully stopped at a stop light isn't putting my or anyone else's life in danger. If I have a phone in my hand the car is not moving. I also finished the comment before the light turned green so the phone also was no longer in my hand when I needed to start moving.

Paying attention to your surroundings tend to be a good idea always in traffic, even if you're standing still. More than once I've beeped at someone who didn't notice a red light turning green, and their first reaction is to accelerate, then look up. I'm guessing if they were actually still paying attention even though they were stopped, they would have noticed better what's going on around them.
Post reply on HN