Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

91–100 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#91

Post by alexhans »

codazoda wrote: post #74
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.

Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make.

The hard thing is always keeping complexity low and being ZeroOps.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#92

Post by JumpCrisscross »

walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028.

(I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)

Re: Show HN: Getting GLM 5.2 running on my slow computer

#94

Post by aliljet »

vikmals wrote: post #90
nogajun wrote: post #87

Is this similar to fastllm? https://github.com/ztxz16/fastllm

fastllm targets the GPU, while colibri uses CPU inference only

I'd be curious about an.option that would allow glm use with a low end GPU like a 2080 ti...

Re: Show HN: Getting GLM 5.2 running on my slow computer

#97

Post by paytonjjones »

JumpCrisscross wrote: post #92
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028. (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)

Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#98

Post by joerawr »

charcircuit wrote: post #67
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage.

But it's about the journey not the destination. My current running local LLMs train of thought...

Re: Show HN: Getting GLM 5.2 running on my slow computer

#100

Post by jnaina »

was lucky enough to snag the Olares One from Kickstarter just before this whole AI induced memory chip price gouging started.

specs are Intel Core Ultra 9 275HX (24 Cores, 5.4GHz),96GB of DDR5 5600MHz RAM, NVIDIA GeForce RTX 5090 Mobile GPU with 24GB of GDDR7 VRAM, 2TB NVMe PCIe 4.0 SSD.

going to see if I can wring at least 5 tok/s.

Post reply on HN