This is something that would benefit from Intel Optane memory. Too bad it was killed at the time.
Maybe some from intel can read and we can try? :)
Show HN: Getting GLM 5.2 running on my slow computer
71–80 of 269 posts
Re: Show HN: Getting GLM 5.2 running on my slow computer
#72My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…
> on hardware that ordinary people can afford
These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
sigh
Re: Show HN: Getting GLM 5.2 running on my slow computer
#73I wonder how would a RAID0 array of either disks or even nvme improve the performance of this.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#74My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…
I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#75Pretty cool! I've also been playing around with GLM 5.2 this week and was equally impressed. At work we're running it locally on some crazy expensive hardware as a test before starting another project so it's great to see people taking this massive FOSS model release and running it on an average machine, even if it's not terribly practical at this point. Nice work!
which hardware?
Re: Show HN: Getting GLM 5.2 running on my slow computer
#76This sort of thing is a lot of fun.
I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.
It's been interesting learning about the balance of factors for performant metal kernels on unified memory.
Should have a repo up on github in the next few weeks.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#77I just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#78[flagged]
Re: Show HN: Getting GLM 5.2 running on my slow computer
#79My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…
> on hardware that ordinary people can afford These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd? sigh
The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".
Re: Show HN: Getting GLM 5.2 running on my slow computer
#80How much time is spent interfacing between userland and the kernel? Can you try to get it to run as a kernel module? :)
Also in case your CPU is old enough, did you try disabling CPU bug mitigations?