Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

151–160 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#151

Post by mrandish »

tharkun__ wrote: post #109
anon373839 wrote: post #102

Earlier quoted context omitted.

This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is…

Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me. But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.

> OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.

While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#152

Post by puzzlingcaptcha »

walrus01 wrote: post #140
Abishek_Muthian wrote: post #124

Earlier quoted context omitted.

Which CPU gen are you suggesting, is there any writeup on such setup where In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment. Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control. Kudos to OP thou…

I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB…

Xeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#154

Post by mft_ »

codazoda wrote: post #74
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.

I’ve been wondering about something similar - a system that enforces (or does the heavy lifting) of dividing a large task into smaller sub-tasks so that it’s easy to run/check/test each one independently - even on a fresh model instance if needed.

This is based on the observation that the medium-sized open weight models (~20-35b) are very able to one-shot smaller discrete tasks but seem to lose their way project managing themselves through larger tasks that have multiple steps.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#155

Post by zozbot234 »

walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the complete model at every token batch in order to get good reuse) and is ultimately limited by CPU/GPU thermals which are a tight constraint on typical inference platforms. It's also only really feasible with tiny KV caches, which requires either a very small context or sticking to KV-cache efficient models such as the DeepSeek V4 series. Still, this might be one way of making use of existing lower-end hardware for practical inference of non-tiny models.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#156

Post by Abishek_Muthian »

puzzlingcaptcha wrote: post #152
walrus01 wrote: post #140

Earlier quoted context omitted.

I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB…

Xeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.

Even though LLM benchmarks are very opinionated, I would really like to see some numbers for the setup parent suggested. From what I read elsewhere, anything below $40K in HW costs is not worth the effort for coding models locally.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#157

Post by mcv »

walrus01 wrote: post #140
Abishek_Muthian wrote: post #124

Earlier quoted context omitted.

Which CPU gen are you suggesting, is there any writeup on such setup where In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment. Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control. Kudos to OP thou…

I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB…

Would be nice if you could somehow connect GPU-levels of parallel floating point cores to that amount of memory. I guess that's what the big AI datacenters are doing, but how can we do that on a budget?

Re: Show HN: Getting GLM 5.2 running on my slow computer

#158

Post by stavros »

antondd wrote: post #108
alexhans wrote: post #91

Earlier quoted context omitted.

Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make. The hard thing is always keeping complexity low and being ZeroOps.

Are there any frameworks/scaffolding/harnesses or general resources on this you can share? I’d love to learn more

I no longer use a harness directly, instead I use Github issues/Linear to work on multiple tickets in parallel while the agents are doing work:

https://github.com/skorokithakis/symphony

Re: Show HN: Getting GLM 5.2 running on my slow computer

#159

Post by Roxxik »

I have prototyped something similar with ollama some months ago.

Do you mmap or issue reads on demand? Also do you use io_uring to interleave compute with io or do you spawn extra threads?

I also tried predicting which experts get reused and I managed to beat a simple LRU very slightly.

EDIT: That was on Kimi 2.5 but even worse quant than 4bit. IIRC it was 2.6 or so

Re: Show HN: Getting GLM 5.2 running on my slow computer

#160

Post by xyzsparetimexyz »

kristianp wrote: post #130
SubiculumCode wrote: post #101

Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?

I you look at https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.

For purely coding tasks, is every single expert required? 50%? 25%?
Post reply on HN