Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

121–130 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#121

Post by BoorishBears »

rsync wrote: post #81
kgeist wrote: post #9

>$40k gets you almost-Opus GLM 5.2 is "almost Opus," and it needs at least 8xH200s for comfortable inference (so it's closer to $400k than $40k). They suggest using this modified model: >A REAP-pruned (≈22% of experts removed), Int8-mix NVFP4 quantized version of GLM-5.2, ≈594B parameters. I wonder how it behaves in practice outside of benchmarks. Qwen3.6, even at 6-bit quantization, often gets stuck in loops while r…

"GLM 5.2 is "almost Opus," and it needs at least 8xH200s for comfortable inference ..." What is the behavior if one were to run GLM 5.2 with only a single H200 ? Would it fail to run at all, or would it just run so slowly as to be unusable ? I would like to prove out the build, and concept, of a SOTA model locally, but then backfill the rest of the GPUs in 18-24 months when they cost significantly less ...

> in 18-24 months when they cost significantly less ...

going to need you to sit down for this one...

Re: Jamesob's guide to running SOTA LLMs locally

#122

Post by sanderjd »

BoorishBears wrote: post #121
rsync wrote: post #81

Earlier quoted context omitted.

"GLM 5.2 is "almost Opus," and it needs at least 8xH200s for comfortable inference ..." What is the behavior if one were to run GLM 5.2 with only a single H200 ? Would it fail to run at all, or would it just run so slowly as to be unusable ? I would like to prove out the build, and concept, of a SOTA model locally, but then backfill the rest of the GPUs in 18-24 months when they cost significantly less ...

> in 18-24 months when they cost significantly less ... going to need you to sit down for this one...

Say more. My expectation is that the current gen of gpus will start being replaced by the next gen, and then it may be possible to get used ones that are still within their useful life at lower prices. My expectation is also that memory vendors are likely to increase production, which will drive those prices down eventually. Maybe not over the next 18-24 months though.

Re: Jamesob's guide to running SOTA LLMs locally

#123

Post by aetherspawn »

maxignol wrote: post #77

Did not seem to find how much tokens per second he achieved with this setup ?

80 tok/s which is kind of a lot for GLM. My experience running 80 tok/s on other LLM is that it ~seems faster than cloud inference, but that obviously depends what you use, in my case ChatGPT.

Re: Jamesob's guide to running SOTA LLMs locally

#124

Post by stingraycharles »

Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

I would very much recommend first using a cloud vendor and setting up an LLM running on there to get a taste of what it’s like before buying the full hardware.

Re: Jamesob's guide to running SOTA LLMs locally

#125

Post by CamperBob2 »

SwellJoe wrote: post #80

I recently wrote up how I run local LLMs, because several folks had asked ( https://swelljoe.com/post/how-i-run-local-llms/ ) and I think even my setup, which I spent maybe $4200 on, half on a Strix Halo and half on upgrades for my desktop, would be too expensive to justify today. I bought before prices went through the roof, and only did so because I like to tinker with hardware...not because I expected it to ever p…

Buying four $13000 GPUs and several thousand dollars worth of supporting hardware seems crazy

Especially when you realize you really want 8 of them. But...

You're not running a model that's equal to what you get when you buy GLM tokens from Z.ai.

... to be perfectly clear: you have no earthly idea what you're getting when you buy GLM tokens from Z.ai. Your options are to run locally, rent cloud hardware, or hope for the best.

Re: Jamesob's guide to running SOTA LLMs locally

#126

Post by SwellJoe »

CamperBob2 wrote: post #125
SwellJoe wrote: post #80

I recently wrote up how I run local LLMs, because several folks had asked ( https://swelljoe.com/post/how-i-run-local-llms/ ) and I think even my setup, which I spent maybe $4200 on, half on a Strix Halo and half on upgrades for my desktop, would be too expensive to justify today. I bought before prices went through the roof, and only did so because I like to tinker with hardware...not because I expected it to ever p…

Buying four $13000 GPUs and several thousand dollars worth of supporting hardware seems crazy Especially when you realize you really want 8 of them. But... You're not running a model that's equal to what you get when you buy GLM tokens from Z.ai. ... to be perfectly clear: you have no earthly idea what you're getting when you buy GLM tokens from Z.ai. Your options are to run locally, rent cloud hardware, or hope for…

OK, that's true, too, but they have a vested interest in GLM being as good as possible. They're nipping at the heels of the big guys, they don't want to ruin that by hobbling their best model with a lossy quantization.

Re: Jamesob's guide to running SOTA LLMs locally

#127

Post by irishcoffee »

echelon wrote: post #49
jacobgold wrote: post #47

> "~$40k At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus." That is equivalent to 16.8 years of Claude Opus 4.8 or Codex GPT 5.5 at $200/mo. I'm a huge fan of running local models, but they're still wildly expensive, lower quality, and possibly dangerous (if backdoored). I sincerely wish this wasn't the case.

Stop trying to run them locally, folks. You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Rent cloud GPUs! You get to participate in the ownership, data control, price control, and hacking culture without having to Frankenstein some hobbyist box that costs a ton, is distilled down to functional uselessness, and is a PITA to maintain.

> You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset?

Like a car? Because I don’t want to depend on uber or a taxi service.

What does fiber have to do with anything? I don’t need the internet to run my local models.

Re: Jamesob's guide to running SOTA LLMs locally

#128

Post by AdieuToLogic »

echelon wrote: post #49
jacobgold wrote: post #47

> "~$40k At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus." That is equivalent to 16.8 years of Claude Opus 4.8 or Codex GPT 5.5 at $200/mo. I'm a huge fan of running local models, but they're still wildly expensive, lower quality, and possibly dangerous (if backdoored). I sincerely wish this wasn't the case.

Stop trying to run them locally, folks. You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Rent cloud GPUs! You get to participate in the ownership, data control, price control, and hacking culture without having to Frankenstein some hobbyist box that costs a ton, is distilled down to functional uselessness, and is a PITA to maintain.

> Stop trying to run them locally, folks.

"Locally" is a relative qualifier if one defines locality as not being reliant on a SaaS vendor. IOW, locality does not necessarily imply execution on machines specifically owned/operated by an organization.

> Rent cloud GPUs!

This would qualify as "locally" in the above definition. There is also a case to be made that h/w ownership (GPUs included) and operation can result in a net cost reduction for some use-cases.

However, where exposing intellectual property results in regulatory violations and/or undue legal exposure, running models "locally" is not only a good option, it is the only option.

Re: Jamesob's guide to running SOTA LLMs locally

#129

Post by KronisLV »

CamperBob2 wrote: post #20
Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

All very true. Right now, running GLM 5.2 at its full BF16 quantization level needs 1.5 TB of VRAM. You can't run this locally at a usable speed for less than $250K or so, and frankly I'd be surprised if it could be done for less than $500K. The best NV4FP quant for 5.2 appears to be lukealonso's at https://huggingface.co/lukealonso/GLM-5.2-NVFP4 , and it is capable of good throughput (75-100 tps) without losing much…

> $100K USD

With z.AI GLM Coding Subscription for 1344 USD per year, that buys you 74 years.

Maybe if you want to host the model for a group of people or really need no artificial token limits, or maybe cannot use cloud models, then it makes more sense.

Post reply on HN