Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

51–60 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#51

Post by vforno »

hoppp wrote: post #48
voidmain0001 wrote: post #43

The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD. [0] https://github.com/JustVugg/colibri#ssd-wear-warning

Yes, avoid. Laptops with soldered in SSDs should definitely monitor their usage and take care with this. This project seems more of an experiment than something everyone should run, but pretty cool nonetheless

Thanks We're working on it!

Re: Show HN: Getting GLM 5.2 running on my slow computer

#52

Post by Archit3ch »

Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#53

Post by tarpitt »

vforno wrote: post #38
tarpitt wrote: post #33

I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2 I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth. >SSD Wear Warning > Cold starts are heavy on random reads (~11 GB/token). Rea…

I have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!

I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused.

I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAMGPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.

If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.

I'm going to benchmark this right now with what I have and I'll get back to you on github.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#54

Post by VortexLain »

voidmain0001 wrote: post #43

The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD. [0] https://github.com/JustVugg/colibri#ssd-wear-warning

Probably yes, use an external drive for that sort of thing

Re: Show HN: Getting GLM 5.2 running on my slow computer

#55

Post by joshsantiago01 »

walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

[flagged]

Re: Show HN: Getting GLM 5.2 running on my slow computer

#57

Post by Archit3ch »

vforno wrote: post #8
Pragmata wrote: post #6

Would this cause issues with SSD lifespan?

What causes problems is the rewriting in this case are only read while writing is the cache! However, I'm working to improve more and more and make some parts lighter!

You can keep the KV cache in (possibly Unified) RAM to avoid SSD writes entirely. Not sure if it would fit on a 32GB laptop, though.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#58

Post by vforno »

tarpitt wrote: post #53
vforno wrote: post #38

Earlier quoted context omitted.

I have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!

I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused. I think another route mi…

Really thanks!!

Re: Show HN: Getting GLM 5.2 running on my slow computer

#59

Post by sebastianconcpt »

Archit3ch wrote: post #52

Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.

This is the way.
Post reply on HN