Jamesob's guide to running SOTA LLMs locally

Post by scheme271 » Sat, Jul 04, 2026, 2:04 AM UTC

sanderjd wrote: post #122

BoorishBears wrote: post #121

Earlier quoted context omitted.

> in 18-24 months when they cost significantly less ... going to need you to sit down for this one...

Say more. My expectation is that the current gen of gpus will start being replaced by the next gen, and then it may be possible to get used ones that are still within their useful life at lower prices. My expectation is also that memory vendors are likely to increase production, which will drive those prices down eventually. Maybe not over the next 18-24 months though.

Post by pbgcp2026 » Sat, Jul 04, 2026, 2:49 AM UTC

irishcoffee wrote: post #127

echelon wrote: post #49

Earlier quoted context omitted.

Stop trying to run them locally, folks. You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Rent cloud GPUs! You get to participate in the ownership, data control, price control, and hacking culture without having to Frankenstein some hobbyist box that costs a ton, is distilled down to functional uselessness, and is a PITA to maintain.

> You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Like a car? Because I don’t want to depend on uber or a taxi service. What does fiber have to do with anything? I don’t need the internet to run my local models.

Post by jzer0cool » Sat, Jul 04, 2026, 3:27 AM UTC

maxxxml wrote: post #63

What harness is the best for local LLMs? I've been researching optimizing local LLM agent harness performance with context/ tools. Quite the endeavor and would love to learn what users prefer for this type of workflow.

Post by gizajob » Sat, Jul 04, 2026, 3:31 AM UTC

Aurornis wrote: post #22

CamperBob2 wrote: post #20

Earlier quoted context omitted.

All very true. Right now, running GLM 5.2 at its full BF16 quantization level needs 1.5 TB of VRAM. You can't run this locally at a usable speed for less than $250K or so, and frankly I'd be surprised if it could be done for less than $500K. The best NV4FP quant for 5.2 appears to be lukealonso's at https://huggingface.co/lukealonso/GLM-5.2-NVFP4 , and it is capable of good throughput (75-100 tps) without losing much…

> It'll almost certainly be worth it, given the abusive behavior we've seen and will continue to see from the major closed-model providers. The proper financial comparison for GLM-5.2 would be one of the providers on OpenRouter or renting a server as needed. Compare apples to apples. You will almost certainly never break even compared to paying per token. Local LLMs at this scale are only worth it if you have extreme…

Post by gizajob » Sat, Jul 04, 2026, 3:36 AM UTC

echelon wrote: post #49

jacobgold wrote: post #47

> "~$40k At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus." That is equivalent to 16.8 years of Claude Opus 4.8 or Codex GPT 5.5 at $200/mo. I'm a huge fan of running local models, but they're still wildly expensive, lower quality, and possibly dangerous (if backdoored). I sincerely wish this wasn't the case.

Stop trying to run them locally, folks. You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Rent cloud GPUs! You get to participate in the ownership, data control, price control, and hacking culture without having to Frankenstein some hobbyist box that costs a ton, is distilled down to functional uselessness, and is a PITA to maintain.