This is technically impressive, but is it usable in practice?
Show HN: Getting GLM 5.2 running on my slow computer
131–140 of 269 posts
Re: Show HN: Getting GLM 5.2 running on my slow computer
#132[dead]
Re: Show HN: Getting GLM 5.2 running on my slow computer
#133Earlier quoted context omitted.
This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is…
> I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway. What could be more valuable than outputting the exact thing you asked for?
Because the thing you get, from a prompt like that - even with a sota llm like fable - is a Potemkin village.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#134[flagged]
Re: Show HN: Getting GLM 5.2 running on my slow computer
#135This sort of thing is a lot of fun. I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation. It's been interesting learning about the balance of factors for performant metal kernels on unified memory…
I'd be interested in seeing this when you put it up! I've managed to get upwards of 300 tokens a second somewhat consistently on the desktop I already had gotten for gaming/personal project stuff several years ago (64 GB RAM, Ryzen 9 7900 X 12-core; I already had a fairly large m.2 SSD and a Radeon 6900 XT to plug into it), but only by severely compromising on the model (a custom quantized Qwen3.5-27B-Q2_K which someone published on huggingface) so that I can use a context size large enough that interfacing through opencode doesn't manage to run out of context while summarizing for compaction and then trigger a second "nested" compaction that instead just seems to cause it to lose all context and ask for a full new set of instructions from scratch. If yours ends up being anywhere as close to the one I've been using in quality, and it works on a 36 GB Mac, it sounds like it would be worth it for me to try out!
(For clarity, I have almost no idea what pretty much any of the terminology for models even means or how it translates to what the actual experience is. My strategy has mostly been using the free tier of OpenCode Zen to ask it stuff about what models and llama.cpp configurations to use in the hopes that I could maybe bootstrap something halfway decent to use locally, and so far what I've managed to get running is mediocre but at least passable)
Re: Show HN: Getting GLM 5.2 running on my slow computer
#136[dead]
Re: Show HN: Getting GLM 5.2 running on my slow computer
#137I love it but where do you find that NVMe SSD for less than the price of an h100 fan let alone the memory
NVMe SSD prices had being gone down in price for a while, and the spikes are actually a lot more recent than you might think. From double checking my Amazon history, I bought my wife a 2 TB NVMe SSD for $160 back in November; it's now listed at three times that. I imagine that a lot of people just have them already from the past few years.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#138[flagged]
Re: Show HN: Getting GLM 5.2 running on my slow computer
#139Earlier quoted context omitted.
I have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!
I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused. I think another route mi…
At least for NVME, it is the write cycles that are limited. Read cycles are non-destructive and essentially unlimited.
Re: Show HN: Getting GLM 5.2 running on my slow computer
#140Earlier quoted context omitted.
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise. Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
Which CPU gen are you suggesting, is there any writeup on such setup where In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment. Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control. Kudos to OP thou…
I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB) and then adding the DIMM kits from the more reputable refurb server part vendors on ebay.
It won't be fast at all, for certain, but it'll have enough memory to prove a configuration and be able to really use gargantuan GGUF format LLMs in the latest compiled llama-server. Re: electricity, I pay the equivalent of $0.07 ro $0.09 USD per kWh so it's not an extreme burden to have a theoretical 500W server running. Something like $35 to $50 of electricity a month if it's 500W 24x7.