Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-5.2 – How to Run Locally

unsloth.ai

Post reply on HN
Print view View original

301–310 of 328 posts

Re: GLM-5.2 – How to Run Locally

#301

Post by pheggs »

stingraycharles wrote: post #181
3stacks wrote: post #54

Earlier quoted context omitted.

Maybe there's a conversation to be had about how much is enough... Unless something beyond my imagination happened, I would be happy enough with Opus 4.5 levels of productivity

This really sounds like “640kb should be enough”. I’m sorry, but I just can’t imagine us running smaller models than we are using right now in 5-10 years from now.

Probably not. But I also cant imagine the hardware not improving. Also there is surely a sweet spot regarding price and quality. I have strong doubts that much larger models will give much better results, which is not like disk space where every additional bit makes it one bit better

Re: GLM-5.2 – How to Run Locally

#302

Post by downut »

downut wrote: post #277
discordance wrote: post #151

Earlier quoted context omitted.

Running that full load is at least 600 W, so in a day ~14 kWh. At $0.2 a kWH, that would be $2.80/day or $1k a year of op-ex in electricity. Unless you really want privacy or the fuzzy feeling of owning your own, it’s cheaper, more convenient and has much faster tok/s if you pay a hyper scaler. That said, I do like the direction we are heading and look forward to seeing what host your own hardware we get in 2 years.

I have rooftop solar and I have been building credit with my electric utility even though the daily high temperature is well over 100F outside and a comfortable 75F inside. That includes running three AMD 12 thread 128GB systems with obsolete GPUs 24x7x365. I'm not a gamer, so 6 years ago I went low-end low-power GPUs. Boy am I dumb. Currently running the qwen3.6:27b, 35b, and gemma4:31b models just fine. As soon as…

I am fascinated that I got down voted. I mean, isn't what I'm doing here nearly ideal? Or maybe not: why? My solar panels shade my roof under the incessant sun of the Sonoran Desert and turn a fraction of the insolation into electrical power that allows me to do almost SOTA local LLM stuff inside my house for free[1] that the parent commenter thought to be economically infeasible. Of course it's slow! So what! Right now I'm transcribing to text a podcast with whisper.cpp and it will take about as much time as the original podcast duration but I will be able to read it in 1/20th of the time.

Alternative interpretation of a downvote: we should all be enslaved to corporate electrical generation provided to "local" electric utility corporations so that we are economically incentivized to use cloud LLM providers. That's weird, no?

Teach me.

[1] It's a small nice house that cost ~$330K not too far off from my city center. This isn't rich privilege boasting.

Re: GLM-5.2 – How to Run Locally

#304

Post by cjbprime »

walrus01 wrote: post #180
cjbprime wrote: post #179

I've got access to a 192GB RAM Mac Studio, which is below the stated minimum RAM. Can swapping off fast disk be used to make it work out, especially since it's MoE?

Seems like a good way to shorten the lifespan of an NVME SSD significantly by using up its TB written lifespan, if you let it extensively swap. Also the performance will be absolutely abysmal like 0.1 tok/second.

Both sentences are likely wrong. It's a written lifespan (technically an "erased" one), not a read lifespan. The weights are only needed from disk read-only. And Mac NVMe interfaces are surprisingly fast.

Edit: Oh, I think you maybe thought I meant swapping working RAM off disk? I didn't. I meant swapping weights off disk into RAM on-demand.

Re: GLM-5.2 – How to Run Locally

#305

Post by walrus01 »

cjbprime wrote: post #304
walrus01 wrote: post #180

Earlier quoted context omitted.

Seems like a good way to shorten the lifespan of an NVME SSD significantly by using up its TB written lifespan, if you let it extensively swap. Also the performance will be absolutely abysmal like 0.1 tok/second.

Both sentences are likely wrong. It's a written lifespan (technically an "erased" one), not a read lifespan. The weights are only needed from disk read-only. And Mac NVMe interfaces are surprisingly fast. Edit: Oh, I think you maybe thought I meant swapping working RAM off disk? I didn't. I meant swapping weights off disk into RAM on-demand.

Yes, I meant that 192GB of RAM even with the worst quantization would result in such a large model going deep into disk swap when it entirely runs out of RAM. At least 100GB worth, if MacOS will even allow that without freezing or crashing or OOM killing the process.

Re: GLM-5.2 – How to Run Locally

#306

Post by walrus01 »

re-thc wrote: post #248
walrus01 wrote: post #178

Earlier quoted context omitted.

That's true, there's a lot of places where power is considerably more expensive than $0.20 USD/kWh. But also the 600W figure assumes that it's fully loaded 24x7x365. Running a system that will be 600W under max CPU usage on all cores and RAM and a few 3090-class GPUs, that same system might be only 90W or around there when idle at 0.00 unix load. If we say: (600 * 24 * 31)/1000 = 446kWh in a month at full load 24 hou…

> But also the 600W figure assumes that it's fully loaded 24x7x365. It isn't 100% efficient. Even the best PSUs aren't.

I was referring to a 600W load as measured at the wall, such as if you plugged a desktop system with a single power supply, and ordinary IEC C13/C14 to NEMA 5-15 male power cord into a kill-a-watt to measure its instantaneous wattage.

Of course the cumulative wattage of all the stuff in a x86-64 type desktop workstation is going to be a different figure than the draw from the AC wall power socket, since even the best power supplies are somewhere around 83-86% efficient in reality. Could easily be 700W, 800W, or some other figure, depending on what CPUs and GPUs a person puts in it.

Re: GLM-5.2 – How to Run Locally

#307

Post by cjbprime »

walrus01 wrote: post #305
cjbprime wrote: post #304

Earlier quoted context omitted.

Both sentences are likely wrong. It's a written lifespan (technically an "erased" one), not a read lifespan. The weights are only needed from disk read-only. And Mac NVMe interfaces are surprisingly fast. Edit: Oh, I think you maybe thought I meant swapping working RAM off disk? I didn't. I meant swapping weights off disk into RAM on-demand.

Yes, I meant that 192GB of RAM even with the worst quantization would result in such a large model going deep into disk swap when it entirely runs out of RAM. At least 100GB worth, if MacOS will even allow that without freezing or crashing or OOM killing the process.

You still have a core misunderstanding. Only one layer of weights is required in memory at a time. A forward pass can be over-simplified as a matrix multiplication of each layer, one at a time.

There is no swapping of working RAM. We're just talking about loading the weights read-only data into RAM on-demand for each layer. It is only as slow as your storage interface.

Re: GLM-5.2 – How to Run Locally

#308

Post by itemize123 »

traceroute66 wrote: post #202
matheusmoreira wrote: post #187

Earlier quoted context omitted.

We do want privacy, and we also want to own the hardware so the US can't just turn it off whenever it feels like it. I think the main reason not to run locally is to get the full models instead of quantized versions.

> We do want privacy, and we also want to own the hardware so the US can't just turn it off whenever it feels like it. I agree and I prefer on-prem where possible. The Apple Mac Studios have been great for that although I don't have enough of them to run GLM-5.2 without heavy quantization. I'm also waiting for the Apple next product refresh which I hope will enable me to do more with less. Meanwhile there are hosted…

tinfoil doesn't seem to distinguish between cached and input tokens. I think that would make it untenably expensive

Re: GLM-5.2 – How to Run Locally

#309

Post by Tepix »

theYipster wrote: post #284
Tepix wrote: post #232

Earlier quoted context omitted.

$50K seems low if you want to run, say, GLM 5.2 4bit fast enough for a team for devs. You need something like 6x RTX Pro 6000 at $11800 each plus a nice server (add $10000) = $80800 and then quite a bit of electricity.

You don't need all of the model in VRAM. 1 or 2 RTX Pro 6000s will do. $50K will get you there very nicely, and on a 1600 watt PSU if you go for the MAX-Q versions. (The same wattage PSU I'm typing this on, and have been using over the last 5 years.)

If you want decent performance (more than say 20 tokens/s) for your dev team, you absolutely do need all of the model in VRAM.

Re: GLM-5.2 – How to Run Locally

#310

Post by Tepix »

MaKey wrote: post #271
Tepix wrote: post #228

Earlier quoted context omitted.

My hope is on Intel Crescent Island with 480GB. I don't need 8x H200 performance (and cost), but I would like to run GLM 5.2 Q8.

I'd love to too, but I guess Crescent Island with 480 GB will cost something like $10-12k or even more.

Yes it will be out of reach for at home but it will be a good option for small companies.
Post reply on HN