Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

111–120 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#111

Post by walrus01 »

JumpCrisscross wrote: post #92
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028. (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)

For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise.

Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#112

Post by walrus01 »

antondd wrote: post #108
alexhans wrote: post #91

Earlier quoted context omitted.

Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make. The hard thing is always keeping complexity low and being ZeroOps.

Are there any frameworks/scaffolding/harnesses or general resources on this you can share? I’d love to learn more

Most any ticketing system can integrate with ordinary IMAP and smtp email flow, so you can really use any agent that can "do" inbound and outbound email to talk to a self hosted ticket queue.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#113

Post by simpaticoder »

jnaina wrote: post #100

was lucky enough to snag the Olares One from Kickstarter just before this whole AI induced memory chip price gouging started. specs are Intel Core Ultra 9 275HX (24 Cores, 5.4GHz),96GB of DDR5 5600MHz RAM, NVIDIA GeForce RTX 5090 Mobile GPU with 24GB of GDDR7 VRAM, 2TB NVMe PCIe 4.0 SSD. going to see if I can wring at least 5 tok/s.

Let us know how it turns out.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#114

Post by p-e-w »

SubiculumCode wrote: post #101

Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?

You could (e.g. by replacing residual-dependent expert routing with hardcoded logic), but quality will suffer dramatically. It’s far better to use a similar-sized dense model then.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#115

Post by ProofHouse »

fuzzfactor wrote: post #83
vforno wrote: post #19

Earlier quoted context omitted.

Maybe we can see some integration!

If you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself. On a level playing field the expression of virtuosity can outshine those who hav…

Agreed!

Re: Show HN: Getting GLM 5.2 running on my slow computer

#116

Post by dyauspitr »

anon373839 wrote: post #102
paytonjjones wrote: post #97

Earlier quoted context omitted.

Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.

This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is…

> I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.

What could be more valuable than outputting the exact thing you asked for?

Re: Show HN: Getting GLM 5.2 running on my slow computer

#117

Post by SubiculumCode »

p-e-w wrote: post #114
SubiculumCode wrote: post #101

Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?

You could (e.g. by replacing residual-dependent expert routing with hardcoded logic), but quality will suffer dramatically. It’s far better to use a similar-sized dense model then.

Ahh, I remember hearing that before, and it makes general sense too.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#118

Post by bigiain »

walrus01 wrote: post #79
bigiain wrote: post #72

Earlier quoted context omitted.

> on hardware that ordinary people can afford These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd? sigh

The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".

I bought whole Intel N100 mini pc with 16GB of DDR5 in it in 2023 for $AUD289 (so about $US200). I got a 16GB (DDR4) SODIMM in 2022 for $AUD88 ($US60).

Re: Show HN: Getting GLM 5.2 running on my slow computer

#120

Post by ac29 »

llama.cpp supports a wide variety of 4-bit and smaller quants and mmap's models by default, so you dont need to be able to hold the weights in memory (the OS will handle bringing them in from storage as needed)

Its cool to see this implemented in a tiny amount of code without dependencies, but does it actually bring more performance?

Post reply on HN