Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

41–50 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#41

Post by TacticalCoder »

valicord wrote: post #36
tarpitt wrote: post #33

I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2 I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth. >SSD Wear Warning > Cold starts are heavy on random reads (~11 GB/token). Rea…

> OS page cache can generate writes Is this a hallucination? What am I missing? Why would heavy reads generate writes?

> Is this a hallucination? What am I missing? Why would heavy reads generate writes?

I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?

I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?

Re: Show HN: Getting GLM 5.2 running on my slow computer

#42

Post by espetro »

kodablah wrote: post #27

I've taken a similar strategy w/ image/video gen at https://github.com/cretz/thinfer (see video branch for a ton of work). Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coup…

Wow, I see you managed to fit in so many models (krea, wan, hunyan, etc.). Did you get to build a common harness to run all of them? Which ones stay under your VRAM budget more consistently?

Re: Show HN: Getting GLM 5.2 running on my slow computer

#43

Post by voidmain0001 »

The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD.

[0] https://github.com/JustVugg/colibri#ssd-wear-warning

Re: Show HN: Getting GLM 5.2 running on my slow computer

#45

Post by fallingbananna »

valicord wrote: post #36
tarpitt wrote: post #33

I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2 I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth. >SSD Wear Warning > Cold starts are heavy on random reads (~11 GB/token). Rea…

> OS page cache can generate writes Is this a hallucination? What am I missing? Why would heavy reads generate writes?

Good catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#46

Post by bahmboo »

Another recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn't use the gpu].

Not hijacking anything as this project is amazing.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#48

Post by hoppp »

voidmain0001 wrote: post #43

The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD. [0] https://github.com/JustVugg/colibri#ssd-wear-warning

Yes, avoid.

Laptops with soldered in SSDs should definitely monitor their usage and take care with this.

This project seems more of an experiment than something everyone should run, but pretty cool nonetheless

Re: Show HN: Getting GLM 5.2 running on my slow computer

#49

Post by [deleted] »

valicord wrote: post #36
tarpitt wrote: post #33

I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2 I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth. >SSD Wear Warning > Cold starts are heavy on random reads (~11 GB/token). Rea…

> OS page cache can generate writes Is this a hallucination? What am I missing? Why would heavy reads generate writes?

[deleted]

Re: Show HN: Getting GLM 5.2 running on my slow computer

#50

Post by Cider9986 »

voidmain0001 wrote: post #43

The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD. [0] https://github.com/JustVugg/colibri#ssd-wear-warning

AppleCare.
Post reply on HN