Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

171–180 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#171

Post by broadsidepicnic »

echelon wrote: post #49
jacobgold wrote: post #47

> "~$40k At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus." That is equivalent to 16.8 years of Claude Opus 4.8 or Codex GPT 5.5 at $200/mo. I'm a huge fan of running local models, but they're still wildly expensive, lower quality, and possibly dangerous (if backdoored). I sincerely wish this wasn't the case.

Stop trying to run them locally, folks. You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Rent cloud GPUs! You get to participate in the ownership, data control, price control, and hacking culture without having to Frankenstein some hobbyist box that costs a ton, is distilled down to functional uselessness, and is a PITA to maintain.

> You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset?

Single mode fiber can serve for tens of years without problems and push the fastest speeds available today. I do not understand this comparison.

Re: Jamesob's guide to running SOTA LLMs locally

#172

Post by bradfa »

vient wrote: post #62
Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

Wonder if AMD MI350P release will affect setups like this. From what I've heard, the price will be pretty similar to RTX PRO 6000 while having 50% more VRAM which is additionally an HBM3E instead of GDDR7.

I’m also watching Intel Celestial with 160GB of LPDDR. Noticed lower memory throughput than AMD or NVIDIA, but potentially significantly lower cost per card. Two of them would likely run deepseek-v4-flash sized models pretty decently.

Re: Jamesob's guide to running SOTA LLMs locally

#173

Post by crymeth0t »

It's nutty to me that anyone would go to such great lengths to use LLMs -- especially chasing the bleeding edge like this. If Claude and co. disappeared tomorrow, I wouldn't flinch.

I don't understand why people are exchanging their brain wrinkles for access to a slop machine. I wonder if a good analogue would be a skilled carpenter being offered access to a machine which excretes furniture (one or two levels of quality beneath Ikea). Does it do the job? Most of the time. Does the carpenter enjoy the process? No.

Re: Jamesob's guide to running SOTA LLMs locally

#174

Post by bradfa »

BoorishBears wrote: post #134
sanderjd wrote: post #122

Earlier quoted context omitted.

Say more. My expectation is that the current gen of gpus will start being replaced by the next gen, and then it may be possible to get used ones that are still within their useful life at lower prices. My expectation is also that memory vendors are likely to increase production, which will drive those prices down eventually. Maybe not over the next 18-24 months though.

The only thing that diminishes the value of a GPU right now is unsupported features with outsized value during inference and/or training (like FP4 support) and it takes time for those features to actually take off And labs are fully leaning into pricing for intelligence, so their margins are improving very quickly (which allows them to pay even more for existing compute) I'd be shocked if current prices aren't the bo…

Many newer Chinese lab models are releasing with int4 native weights. Latest NVIDIA generation GPUs have a hard time with this and can actually be slower than previous generations. This may make Blackwell depreciate faster than other recent generations.

Re: Jamesob's guide to running SOTA LLMs locally

#175

Post by TacticalCoder »

aka-rider wrote: post #160
Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

I run Qwen3.6 on RTX4090, and it does amazing job for the most parts. For coding task, one needs to break the session among multiple calls I made https://github.com/aka-rider/orqestra but it's possible to do the same in almost any modern harness directly. The main idea is: - separate session that burns context on reading code and calling tools (context7, etc) -> markdown report "here are relevant patrts of code, docs…

> Qwen3.6 can run for hours looking for a complex bugs in read-only mode, and usually it gets it. Proposed fix would probably be hacky, but so as Sonnet's

I'll go on a tangent but to me that's what we're all seeing. It's the "record number of CVEs found by AIs" thing: these tools are extremely good at searching inside code. And that is a godsend.

We' got people (claiming they're from Anthropic) posting comment saying: "Yes GLM 5.2 found that security bug in library xxx, but we just tried with Fable and it found it too".

More code-searching, more bugs finding. Dick-measuring contests on bug finding abilities.

But the headlines we don't see at all are: "1000 CVEs found by AI, 1000 CVEs fixed by code written by AI". These are nowhere to be found.

We don't see "GLM 5.2 suggested an elegant fix to CVE-2027-xxxxxx" to then have a paid Anthropic shill posting "Fable suggested an eleganter fix than GLM 5.2".

These headlines are, as of 2026, nowhere.

You wrote the result would be "hacky". Here's why I saw from a top, paid for, SOTA model from the top company of the moment: instead of doing two integers comparison (literally one line of code) to verify that a value is between a range, the thing somehow noticed a "pattern" in the hexadecimal representation of the two values and went insane. It started converting the value to its hexadecimal string representation and then started doing substring string matching on that.

"Hacky" is too nice of a word.

This is pure garbage.

Those who go hiking "while their agents ship features" don't realize the level of underperforming, buggy, insecure crap that their LLMs are generating.

I found it very interesting the schism between those who use LLMs to find issues but who verify/modify or even don't use at all the fix they suggest and those who vibe-code while on a yoga retreat.

It's 2026: LLMs do find bugs. But can they fix them?

And do we even care: isn't finding a bug 99% of the job?

Re: Jamesob's guide to running SOTA LLMs locally

#176

Post by aka-rider »

NBJack wrote: post #167
aka-rider wrote: post #160

Earlier quoted context omitted.

I run Qwen3.6 on RTX4090, and it does amazing job for the most parts. For coding task, one needs to break the session among multiple calls I made https://github.com/aka-rider/orqestra but it's possible to do the same in almost any modern harness directly. The main idea is: - separate session that burns context on reading code and calling tools (context7, etc) -> markdown report "here are relevant patrts of code, docs…

I'm afraid prompts and clever arrangements of data don't really negate the parent post warnings. It's great if it works for you and your projects. Unfortunately, I can almost guarantee your approach will break down once you get a project large enough or switch to a less popular language. My favorite example is Godot; most local models just can't get it through their thick AI skull that code alone won't be enough to g…

There is no replacement for large models, indeed. And this is not the point I'm trying to make. There are numerous applications for self-hosted models.

As a simplest example, when you ask "explain what this code does" advantage of large models is negligible.

I tried Fable, "look at this repo, find all bugs" — yeah, neither Qwen nor Opus can do this.

> I can almost guarantee your approach will break down once you get a project large enough or switch to a less popular language.

I can guarantee you it is not, I used my Qwen on 10-15 years of PHP — I just know how and where it will break; what to ask for, what not. Orqestra was/is self-hosted, being developed by, well, orchestra of Qwen agents.

Moreover, Opus and GPT-5.5 break similarly, yeah they will withstand much more pressure, but they will hallucinate and loop nevertheless. My Qwen experience translates seamlessly. I learned so much about agentic engineering, harnesses, tooling, building custom MCPs...

Re: Jamesob's guide to running SOTA LLMs locally

#177

Post by aka-rider »

TacticalCoder wrote: post #175
aka-rider wrote: post #160

Earlier quoted context omitted.

I run Qwen3.6 on RTX4090, and it does amazing job for the most parts. For coding task, one needs to break the session among multiple calls I made https://github.com/aka-rider/orqestra but it's possible to do the same in almost any modern harness directly. The main idea is: - separate session that burns context on reading code and calling tools (context7, etc) -> markdown report "here are relevant patrts of code, docs…

> Qwen3.6 can run for hours looking for a complex bugs in read-only mode, and usually it gets it. Proposed fix would probably be hacky, but so as Sonnet's I'll go on a tangent but to me that's what we're all seeing. It's the "record number of CVEs found by AIs" thing: these tools are extremely good at searching inside code. And that is a godsend. We' got people (claiming they're from Anthropic) posting comment saying…

The best metaphor I heard about LLMs so far - it's a search engine. The bigger the model the bigger the search space. Small models tend to have a "tunnel vision" or fall into "rabbit holes" - they have less visible options to choose from.

> underperforming, buggy, insecure crap that their LLMs are generating

The biggest challenges with AI-generated code are: models actively destroy security features, Opus explained to me once that authorization mechanism is "bad development experience" all while making a backdoor (he made a skeleton key if token=="test" then all permissions granted). Also models actively destroy QA gates. I don't even complain when they delete tests - at least it's visible, they can flip condition to make a test pass, and with vast code changes these are hard to spot.

I myself, and some people I know "vibe-code" professionally though, but then we often assess not the code but it's behaviour. For instance, whether hand-made tests are all pass, p95 is under 50ms, and so on, I may not care about the implementation details.

On the other hand, my friend told me about garage owner he visited, 60 yrs old auto-mechanic, CRM, parts inventory management, payments processing terminal, passwords in txt, people's personal data God knows where, could be unprotected MySQL looking into the Internet bare for all we know.

2026 onwards will be wild.

Re: Jamesob's guide to running SOTA LLMs locally

#178

Post by aayush0325 »

aka-rider wrote: post #160
Aurornis wrote: post #16

I play with local LLMs a lot. I've spent more on hardware than I should. I'm friends with a local group of people who have spent a lot more than I have. The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this build is going to cost more like…

I run Qwen3.6 on RTX4090, and it does amazing job for the most parts. For coding task, one needs to break the session among multiple calls I made https://github.com/aka-rider/orqestra but it's possible to do the same in almost any modern harness directly. The main idea is: - separate session that burns context on reading code and calling tools (context7, etc) -> markdown report "here are relevant patrts of code, docs…

my experience has been similar, qwen is very good at ALMOST getting the job done for large tasks and does fine on smaller/medium tasks.

Re: Jamesob's guide to running SOTA LLMs locally

#179

Post by CamperBob2 »

gizajob wrote: post #138
Aurornis wrote: post #26

Earlier quoted context omitted.

> hedge against the various tail risks of third-party providers raising prices They could 10X the prices and you’d still be better off. It’s also unlikely that prices go up enough to warrant a $100K local investment to prevent paying a couple bucks per million tokens. > or denying you service I guess you’re not familiar with OpenRouter? There are many providers there. There are providers outside of OpenRouter. There…

People seem to miss that with local models you can have them burning their wee digital brains out 24/7, which is a different class of AI usage than that from online models even at a few dollars per million tokens.

There's a definite psychological branch point. With a remote provider, no matter how readily you can afford it, your mindset is always going to be, "I should think twice about what I'm doing. I hate to waste tokens." With your own hardware, your mindset is more like, "I should try to get more done. I hate to see this thing just sitting there idle."

Re: Jamesob's guide to running SOTA LLMs locally

#180

Post by broadsidepicnic »

djx22 wrote: post #170

in my experience running models that have been heavily quantized(q4) or altered to some extent has never made me say “wow, this is so amazing”. On the contrary, the model ended up in the thrash bin after a few prompts. I have an RTX 6000 PRO with 96GB, and what I can run comfortably is Qwen 3.6 27B or MoE, Gemma 4 31B. This is as far as it goes when you run the model at full precision and maximum context length. They…

what you maybe forget here is the use case for people and businesses who can not send the data to 3rd party due to privacy/contractual reasons. This is what I'm looking at, we're bound by strict policies for data sharing outside of our premises.
Post reply on HN