Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

81–90 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#81

Post by fuzzfactor »

bigiain wrote: post #72
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

> on hardware that ordinary people can afford These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd? sigh

Ideally this engineer's approach will yield better performance on lesser equipment in the future, if they keep up the good work after they get more-capable gear to experiment with as time goes by :)

Re: Show HN: Getting GLM 5.2 running on my slow computer

#82

Post by fuzzfactor »

walrus01 wrote: post #79
bigiain wrote: post #72

Earlier quoted context omitted.

> on hardware that ordinary people can afford These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd? sigh

The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".

Maybe that's a measure of the self-fulfilling dollar incentive toward "renting" someone else's RAM in the future rather than trying to actually own such an outlandishly luxury item :\

Re: Show HN: Getting GLM 5.2 running on my slow computer

#83

Post by fuzzfactor »

vforno wrote: post #19
walrus01 wrote: post #17

Earlier quoted context omitted.

I have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).

Maybe we can see some integration!

If you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself.

On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :)

When pulling way more than your own weight happens like for few others.

There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste.

First Prize right before your eyes !

Grande praise !

And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#85

Post by vforno »

fuzzfactor wrote: post #83
vforno wrote: post #19

Earlier quoted context omitted.

Maybe we can see some integration!

If you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself. On a level playing field the expression of virtuosity can outshine those who hav…

Thanks for kind words!

Re: Show HN: Getting GLM 5.2 running on my slow computer

#86

Post by Cieric »

vforno wrote: post #30
Cieric wrote: post #26

I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet. To expand since I ju…

if you like, colibrì always needs to improve so if you have ideas or anything else you are welcome for pull request issues and also benchmarks!

Yeah I'll see what I can transfer over from my llama.cpp work. As before I'm not too experienced with llm work, but I have a lot of experiments I'm trying out. So I'll make a PR if I get any interesting results.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#88

Post by dicroce »

Cieric wrote: post #26

I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet. To expand since I ju…

This is the approach I was wondering about.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#89

Post by cjbprime »

voidmain0001 wrote: post #43

The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD. [0] https://github.com/JustVugg/colibri#ssd-wear-warning

It's a very conservative warning. The application does not perform writes, so the application doesn't actually wear your SSD at all. The rest is just application-independent general hygiene.
Post reply on HN