[dead]
Show HN: Getting GLM 5.2 running on my slow computer
251–260 of 269 posts
Re: Show HN: Getting GLM 5.2 running on my slow computer
#252I'm running an archaic Dell PowerEdge T710 dual-Xeon E5690 (6/12 CPU, 3.73GHz turbo) with 192GB DDR3-800 (was 288GB but now PassMark 14,328). Token rate is 0.091 per second. Good for an overnight job.
that's like 6k tokens over 12 hours which doesn't seem like a lot for…any purpose i can think of
Re: Show HN: Getting GLM 5.2 running on my slow computer
#253Why did you use Claude to help you make this rather than GLM? (It’s listed as a contributor in your repo)
Re: Show HN: Getting GLM 5.2 running on my slow computer
#254[dead]
Re: Show HN: Getting GLM 5.2 running on my slow computer
#255[dead]
Re: Show HN: Getting GLM 5.2 running on my slow computer
#256Earlier quoted context omitted.
Not sure if mmapping is the right way. In my own tests I noticed that simple mmapping will produce many small reads and not keep the SSD queue saturated. So if RAM is large enough to cache most experts, that is a rounding error. But if the base weigths without experts fill more than half or RAM and you basically need to load in a few experts for each layer of each token, the latency gets important and mmap sadly bloc…
On the technical details of mmap I agree (at least while single threaded which is how I believe I'm running it), but making it async does sound like an interesting method for speeding it up. My only goal with my testing was get as large of a model as possible running on a computer that "can't run it." I'm going to have to actually read through the code and figure out how it really works to really make any good optimi…
My recommendation if you are willing to push it further is to keep KV and always needed weights pinned in memory, so they do not get evicted. This is likely already the case, as they are touched on each token. mmap is slow on evicted pages, as it does not load the whole tensor, but only the touched pages. And it does this through a page fault, thus blocking your code. So it loads a page hands control back to you and the code goes on to touch another evicted page, repeating the loop. Now on an HDD that is not a big problem (yes reads can be coalesced, but a HDD is fundamentally serial in reading) while an SSD can overlay reads better and it is good to keep a few reads in flight at all times to keep its queue fed.
One option is locking the pages. But for that size you need extra privileges.
I experimented with some options. For example: one problem with io_uring is that it still reads to page cache so your reads gets copied in memory after they landed. Now if you pass O_DIRECT that does not happen, but it has its own can of worms.
For full transparency: I had opus write the io_uring layer into llama.cpp for me. And it yielded something slightly short of a 2x tok/s speedup vs simple mmaping. Also I noticed that disabling the warmup and initial test dramatically increases startup time.
The medusa paper looks interesting. My work was a few months ago, multi-token decoding was not a thing then.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#257Earlier quoted context omitted.
> I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway. What could be more valuable than outputting the exact thing you asked for?
Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
Using the superpowers skill will help fill in the gaps for what’s needed to one-shot (it’s more setup so you can one-shot).
And if it gets it wrong, at least you have 80% scaffolding that you can ask to iterate on.
Hoping you’ll get 100% you asked for is going to fail. Hell… you can even achieve this with humans!
Re: Show HN: Getting GLM 5.2 running on my slow computer
#258[flagged]
Re: Show HN: Getting GLM 5.2 running on my slow computer
#259Sounds more like crawling instead of running, but at least it’s moving.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#260Earlier quoted context omitted.
How was qwen3.6 launched? The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
Yeah, I really should know. But I'm using whatever ollama gave me by default, which probably isn't optimal.
Ollama uses 4 bit quants and a very short context window by default. It can easily break on anything more complex than a simple chat.