Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

141–150 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#141

Post by ai_fry_ur_brain »

codazoda wrote: post #74
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.

Use the ticket system built into mininote.ink 's mcp server. Works perfect right out of the box. Also great notetaking app.

Docs:

https://mininote.ink/docs/mcp-docs

Re: Show HN: Getting GLM 5.2 running on my slow computer

#142

Post by kamranjon »

vforno wrote: post #5
nerder92 wrote: post #4

Is this inspired by antirez work on ds4? Amazing job!

Antirez is the number one!thanks really thanks!

Antirez has a GLM 5.2 branch now in dwarfstar: https://github.com/antirez/ds4/tree/glm5.2

It heavily utilizes ssd streaming from my understanding and I think he mentioned getting some semi usable speeds on a 128gb m5 mbp.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#143

Post by Rand01ph »

Abishek_Muthian wrote: post #124
walrus01 wrote: post #111

Earlier quoted context omitted.

For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise. Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.

Which CPU gen are you suggesting, is there any writeup on such setup where In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment. Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control. Kudos to OP thou…

[flagged]

Re: Show HN: Getting GLM 5.2 running on my slow computer

#144

Post by harrouet »

This is exactly the kind of technology that I expect Apple to ship anytime soon given the RAM prices and their HW/SW integration skills:

- ship super fast SSD (tbh they are already top notch)

- add a specific cache layer for tokens

- keep the amount of unified memory reasonable

Re: Show HN: Getting GLM 5.2 running on my slow computer

#146

Post by kmarc »

bigiain wrote: post #72
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

> on hardware that ordinary people can afford These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd? sigh

After 18y of thinkpads, this year I bouth a Lenovo yoga for... Cheap (1000€).

32G RAM, nvme 1TB, core ultra 258V.

Looking at the prices now... Wow, was I lucky.

Tried some of the 7b models locally, more than usable, around 30token/sec, not with the NPU, but using the ARC integrated GPU.

I am a noob for this, but I guess it's time to experiment more with this local setup

Re: Show HN: Getting GLM 5.2 running on my slow computer

#147

Post by 0xc133 »

codazoda wrote: post #74
walrus01 wrote: post #10

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for t…

I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.

So I’ve been thinking about this problem a lot, specifically as it relates to running LLMs at home, and I’ve been using GLM-5.2 to make an SMTP/IMAP-to-LLM gateway.

https://tangled.org/clee.sh/posthorn

Re: Show HN: Getting GLM 5.2 running on my slow computer

#148

Post by anon373839 »

3836293648 wrote: post #103
anon373839 wrote: post #102

Earlier quoted context omitted.

This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is…

Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.

Certainly. First of all, I am using OpenCode as the harness. (I have heard there are better harnesses such as little-coder for small open-weights models, but I haven't tried them yet.) Looking over some of my recent sessions, here are some examples:

- Asking Qwen to review project docs (requirements, user stories, etc) so that "we" can evaluate an iterate on an API design. Then back-and-forth chat about possible design directions. Then I ask for a rough-sketch plan of the one I'm interested in. I provide some tweaks to the plan and request a final plan in full detail. I switch to build mode and say go; everything is written to spec.

- Asking Qwen to write a suite of tests covering X, Y, Z issues with permutations A, B, C per issue.

- Asking Qwen to edit the shape of a CNN to insert auxiliary branches for intermediate supervision, and to extract out part of the network as a modular component with parameterized architecture.

I have less experience with the dense 27B because it's too slow to use on Apple Silicon. But regardless of which model you try, I would recommend trying a full-fat cloud hosted version of it first, so that you can get a sense of what it's capable of when the inference stack is correctly configured. LLMs are very sensitive to quantization formats, discrepancies in chat templates, etc. That kind of stuff is make-or-break.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#149

Post by anon373839 »

dyauspitr wrote: post #116
anon373839 wrote: post #102

Earlier quoted context omitted.

This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is…

> I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway. What could be more valuable than outputting the exact thing you asked for?

Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
Post reply on HN