Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

211–220 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#211

Post by alexhans »

lancekey wrote: post #71
alexhans wrote: post #3

Useful tip. From a strategic standpoint of privacy, cost and control, I immediately went for local models, because that allowed to baseline tradeoffs and it also made it easier to understand where vendor lock-in could happen, or not get too narrow in perspective (e.g. llama.cpp/open router depending on local/cloud [1] ). With the explosion of popularity of CLI tools (claude/continue/codex/kiro/etc) it still makes sen…

Can you say a bit more about evals and your approach?

High level, the approach is:

- I'm pain point driven:

  - I can't compare what I can't measure. 

  - I can't trust to run this "AI" tool to run on its own
- That's automation, which is about intentionality (can I describe what I want?) and risk profile understanding (What's the blast radius/worst that could happen)

Then I treat it as if it was an Integration Test/Test Driven Development exercise of sorts.

- I don't start designing an entire cloud infrastructure.

- I make sure the "agent" is living in the location where the users actually live so that it can be the equivalent of an extra paid set of hands.

- I ask questions or replicate user stories and use deterministic tests wherever I can. Don't just go for LLMaaJ. What's the simplest thing you can think of?

- The important thing is rapid iteration and control. Just like in a unit testing scenario it's not about just writing a 100 tests but the ones that qualitatively allow you to move as fast as possible.

- At this stage where the space is moving so fast and we're learning so much, don't assume or try to over-optimize places that don't hurt and instead think about minimalism, ease of change, parameterization and ease of comparison with other components that form "the black box" and with itself.

- Once you have the benchmarks that you want, you can decide things like pick the cheapest model/agent configuration that does the job within the acceptable timeframe.

Happy to go deeper on these. I have some practical/runnable samples/text I can share on the topic after the weekend. I'll drop a link here when it's ready

Re: Claude Code: connect to a local model when your quota runs out

#212

Post by swyx »

heyyeah wrote: post #159
swyx wrote: post #5

i mean the other obvious answer is to plug in to the other claude code proxies that other model companies have made for you: https://docs.z.ai/devpack/tool/claude https://www.cerebras.ai/blog/introducing-cerebras-code or i guess one of the hosted gpu providers if you're basically a homelabber and wanted an excuse to run quantized models on your own device go for it but dont lie and mutter under your own tin foil hat…

It's definitely a backup solution but even since I was drafting the blog, Qwen3 Coder Next was released. It's a functional stop gap if you want to keep things local. I try to be up front in the blog for people to "Reduce your expectations about speed and performance!" (Also, I love your podcast!)

ha tyyy. sorry for not being thaat much of a /r/localllama guy

Re: Claude Code: connect to a local model when your quota runs out

#213

Post by deaux »

tracker1 wrote: post #199
deaux wrote: post #183

Earlier quoted context omitted.

That's the setup you want for serious work yes, so probably $60kish all-in(?). Which is a big chunk of money for an individual, but potentially quite reasonable for a company. Being able to get effectively _frontier-level local performance_ for that money was completely unthinkable so far. Correct me if I'm wrong, but I think Deepseek R1 hardware requirements were far costlier on release, and it had a much bigger gap…

I'm not so sure on that... even if one $60k machine can handle the load of 5 developers at a time, you're still looking at 5 years of service to recoup $200/mo/dev and that doesn't even consider other improvements to hardware or the models service providers offer over that same period of time. I'd probably rather save the capex, and use the rented service until something much more compelling comes along.

At this point in time, 100% agreed. But what matters is the trend line. Two years ago nothing came close, if you wanted frontier-level "private" hosting you'd need an enterprise contract with OpenAI for many $millions. Then R1 came, it was incredibly expensive and still quite off. Now it's $60k and basically frontier.

Re: Claude Code: connect to a local model when your quota runs out

#214

Post by deaux »

VladVladikoff wrote: post #201
deaux wrote: post #122

Earlier quoted context omitted.

I disagree, based on having used it extensively over the last week. I find it to be at least as strong as Sonnet 4.5 and 5.2-Codex on the majority of tasks, often better. Note that even among the big 3, each of them has a domain where they're better than the other two. It's not better than Codex (x-)high at debugging non-UI code - but neither is Opus or Gemini. It's not better than Gemini at UI design - but neither i…

What hardware are you running it on?

I'm not running it locally, just using cloud inference. The people I know who do use RTX 6000s, picking the quant based on how many of them they've got. Chained M3 ultra setups are fine to play around with but too slow for actual use as a dev.

Re: Claude Code: connect to a local model when your quota runs out

#215

Post by EagnaIonat »

shen wrote: post #189
EagnaIonat wrote: post #150

Earlier quoted context omitted.

https://ollama.com Although I'm starting to like LMStudio more, as it has more features that Ollama is missing. https://lmstudio.ai You can then get Claude to create the MCP server to talk to either. Then a CLAUDE.md that tells it to read the models you have downloaded, determine their use and when to offload. Claude will make all that for you as well.

Which local models are you using for the 32gb MacBooks?

Mainly gpt-oss-20b as the thinking mode is really good. I occasionally use granite4 as it is a very fast model. But any 4GB model should easily be used.

Re: Claude Code: connect to a local model when your quota runs out

#216

Post by tga »

monch1962 wrote: post #128

What are peoples' current suggestions for using Claude Code with a locally hosted LLM running on regular consumer hardware (for the sake of discussion, assume you're spending $US500-ish on a mini PC, which would get you a reasonably decent CPU, 32Gb RAM and a cheapish GPU)? I get that it's not going to work as well as hosted/subscription services like Claude/Gemini/Codex/..., but sometimes those aren't an option

You can run something like Qwen 2.5 Coder on a regular machine (https://huggingface.co/Qwen/Qwen2.5-Coder-7B) but it's really not in the same universe with Claude Code, it will be slow and generate bad code.

It might make sense to run a small LLM locally for general conversation or very specific tasks, but they're not a serious option for agentic coding.

Re: Claude Code: connect to a local model when your quota runs out

#217

Post by tracker1 »

deaux wrote: post #213
tracker1 wrote: post #199

Earlier quoted context omitted.

I'm not so sure on that... even if one $60k machine can handle the load of 5 developers at a time, you're still looking at 5 years of service to recoup $200/mo/dev and that doesn't even consider other improvements to hardware or the models service providers offer over that same period of time. I'd probably rather save the capex, and use the rented service until something much more compelling comes along.

At this point in time, 100% agreed. But what matters is the trend line. Two years ago nothing came close, if you wanted frontier-level "private" hosting you'd need an enterprise contract with OpenAI for many $millions. Then R1 came, it was incredibly expensive and still quite off. Now it's $60k and basically frontier.

Of course... it's definitely interesting. I'm also thinking that there are times where you insource vs outsource to a SaaS that's going to do the job for you and you have one less thing to really worry about. Comparing cost to begin with was just a point I was curious about, so I ran the numbers. I can totally see a point where you have that power in a local developer workstation (power requirements notwithstanding), good luck getting that much power to an outlet in your home office. Let alone other issues.

Right now, I think we've probably got 3-5 years of manufacturing woes to work through and another 3-5 years beyond that to get power infrastructure where it needs to be to support it... and even then, I don't think all the resources we can reasonably throw at a combination of mostly nuclear and solar will get there as quickly as it's needed.

That also doesn't consider the bubble itself, or the level of poor to mediocre results altogether even at the frontier level. I mean for certain tasks, it's very close to human efforts in a really diminished timeframe, for others it isn't... and even then, people/review/qa/qc will become the bottleneck for most things in practice.

I've managed to get weeks of work done in a day with AI, but then still have to follow-up for a couple days of iteration on following features... still valuable, but it's mixed. I'm more bullish today than even a few months ago all the same.

Re: Claude Code: connect to a local model when your quota runs out

#218

Post by zingar »

zingar wrote: post #6

I guess I should be able to use this config to point Claude at the GitHub copilot licensed models (including anthropic models). That’s pretty great. About 2/3 of the way through every day I’m forced to switch from Claude (pro license) to amp free and the different ergonomics are quite jarring. Open source folks get copilot tokens for free so that’s another pro license I don’t have to worry about.

Update: I managed to do exactly this using the GitHub project that OP links to.

Re: Claude Code: connect to a local model when your quota runs out

#219

Post by K0balt »

PlatoIsADisease wrote: post #161
K0balt wrote: post #95

Earlier quoted context omitted.

Mmmm, not really. I have both a4x 3090 box and a Mac m1 with 64 gb. I find that the Mac performs about the same as a 2x 3090. That’s nothing stellar, but you can run 70b models at decent quants with moderate context windows. Definitely useful for a lot of stuff.

>quants >moderate context windows Really had to modify the problem to make it seem equal? Not that quants are that bad, but the context windows thing is the difference between useful and not useful.

Equal to the 2x3090? Yeah it’s about equal in every way, context windows included.

As for useful at that scale?

I use mine for coding a fair bit, and I don’t find it a detractor overall. It enforces proper API discipline, modularity, and hierarchal abstraction. Perhaps the field of application makes that more important though. (Writing firmware and hardware drivers).

It also brings the advantage of focusing exclusively on the problems that are presented in the limited context, and not wandering off on side quests that it makes up.

I find it works well up to about 1KLOC at a time.

I wouldn’t imply they were equal to commercial models, but I would definitely say that local models are very useful tools.

They are also stable, which is not something I can say for SOTA models. You cal learn how to get the best results from a model and the ground doesn’t move underneath you just when you’re on a roll.

Post reply on HN