Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting GLM 5.2 running on my slow computer

github.com

Post reply on HN
Print view View original

121–130 of 269 posts

Re: Show HN: Getting GLM 5.2 running on my slow computer

#121

Post by Infernal »

SubiculumCode wrote: post #101

Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?

I have a similar question and I’m inferring the answer is no - look at the cache hit rate of 23% for the 128GB M5 Max. I had previously assumed that the 40B active meant that a set of layers was chosen as THE expert for a given prompt and generation was then limited to those layers until complete. But in that case you’d have expected the expert caching to have a super high hit rate once you had enough RAM to hold an entire expert’s worth of layers.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#122

Post by stingraycharles »

paytonjjones wrote: post #97
JumpCrisscross wrote: post #92

Earlier quoted context omitted.

The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028. (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)

Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.

Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028.

Expectations seem to be rising at a faster rate than models can improve.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#123

Post by pianopatrick »

I'm curious but don't know much about the internals of LLMs - could you use a similar architecture with other models that have "layers"? I mean, could you have one layer do its work, then remove that layer from RAM, load the next layer from disk, and have that layer activate on the result of the first layer?

Re: Show HN: Getting GLM 5.2 running on my slow computer

#124

Post by Abishek_Muthian »

walrus01 wrote: post #111
JumpCrisscross wrote: post #92

Earlier quoted context omitted.

The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028. (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)

For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise. Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.

Which CPU gen are you suggesting, is there any writeup on such setup where In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment.

Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control.

Kudos to OP though, They had clear goals and they achieved it.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#125

Post by fwipsy »

walrus01 wrote: post #79
bigiain wrote: post #72

Earlier quoted context omitted.

> on hardware that ordinary people can afford These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd? sigh

The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".

Does it have to be DDR5? Is the limit RAM speed, or SSD speed?

Re: Show HN: Getting GLM 5.2 running on my slow computer

#126

Post by killingtime74 »

sakesun wrote: post #77

I just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.

If I understand correctly the basic structure hasn't changed since Transformers were invented. It's just gotten bigger and the models have had better training and also some optimizations. I don't think this one has optimizations.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#128

Post by walrus01 »

Abishek_Muthian wrote: post #124
walrus01 wrote: post #111

Earlier quoted context omitted.

For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise. Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.

Which CPU gen are you suggesting, is there any writeup on such setup where In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment. Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control. Kudos to OP thou…

I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff.

Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#129

Post by walrus01 »

fwipsy wrote: post #125
walrus01 wrote: post #79

Earlier quoted context omitted.

The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".

Does it have to be DDR5? Is the limit RAM speed, or SSD speed?

I was just using that as an example of constant on going price rises, it was the most mundane and not particularly fast ddr5 6000 stuff. The 6400 is even more ridiculous.

Re: Show HN: Getting GLM 5.2 running on my slow computer

#130

Post by kristianp »

SubiculumCode wrote: post #101

Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?

I you look at https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.
Post reply on HN