Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

11–20 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#11

Post by cyanydeez »

alexhans wrote: post #3

Useful tip. From a strategic standpoint of privacy, cost and control, I immediately went for local models, because that allowed to baseline tradeoffs and it also made it easier to understand where vendor lock-in could happen, or not get too narrow in perspective (e.g. llama.cpp/open router depending on local/cloud [1] ). With the explosion of popularity of CLI tools (claude/continue/codex/kiro/etc) it still makes sen…

I think control should be top of the list here. You're talking about building work flows, products and long term practices around something that's inherently non-deterministic.

And the probability that any given model you use today is the same as what you use tomorrow is doubly doubtful:

1. The model itself will change as they try to improve the cost-per-test improves. This will necessarily make your expectations non-deterministic.

2. The "harness" around that model will change as business-cost is tightened and the amount of context around the model is changed to improve the business case which generates the most money.

Then there's the "cataclysmic" lockout cost where you accidently use the wrong tool that gets you locked out of the entire ecosystem and you are black listed, like a gambler in vegas who figures out how to count cards and it works until the house's accountant identifies you as a non-negligible customer cost.

It's akin to anti-union arguments where everyone "buying" into the cloud AI circus thinks they're going to strike gold and completely ignores the fact that very few will and if they really wanted a better world and more control, they'd unionize and limit their illusions of grandeur. It should be an easy argument to make, but we're seeing about 1/3 of the population are extremely susceptible to greed based illusions.,

Re: Claude Code: connect to a local model when your quota runs out

#12

Post by usef- »

esafak wrote: post #10

Or they could just let people use their own harnesses again...

That wouldn't solve this problem.

And they do? That's what the API is.

The subscription always seemed clearly advertised for client usage, not general API usage, to me. I don't know why people are surprised after hacking the auth out of the client. (note in clients they can control prompting patterns for caching etc, it can be cheaper)

Re: Claude Code: connect to a local model when your quota runs out

#13

Post by wkirby »

My experience thus far is that the local models are a) pretty slow and b) prone to making broken tool calls. Because of (a) the iteration loop slows down enough to where I wander off to do other tasks, meaning that (b) is way more problematic because I don't see it for who knows how long.

This is, however, a major improvement from ~6 months ago when even a single token `hi` from an agentic CLI could take >3 minutes to generate a response. I suspect the parallel processing of LMStudio 0.4.x and some better tuning of the initial context payload is responsible.

6 months from now, who knows?

Re: Claude Code: connect to a local model when your quota runs out

#14

Post by esafak »

usef- wrote: post #12
esafak wrote: post #10

Or they could just let people use their own harnesses again...

That wouldn't solve this problem. And they do? That's what the API is. The subscription always seemed clearly advertised for client usage, not general API usage, to me. I don't know why people are surprised after hacking the auth out of the client. (note in clients they can control prompting patterns for caching etc, it can be cheaper)

End users -- people who use harnesses -- have subscriptions so that makes no sense. General API usage is for production.

Re: Claude Code: connect to a local model when your quota runs out

#15

Post by btbuildem »

I'm confused, wasn't this already available via env vars? ANTHROPIC_BASE_URL and so on, and yes you may have to write a thin proxy to wrap the calls to fit whatever backend you're using.

I've been running CC with Qwen3-Coder-30B (FP8) and I find it just as fast, but not nearly as clever.

Re: Claude Code: connect to a local model when your quota runs out

#18

Post by drifkin »

mogoman wrote: post #4
alexhans wrote: post #3

Useful tip. From a strategic standpoint of privacy, cost and control, I immediately went for local models, because that allowed to baseline tradeoffs and it also made it easier to understand where vendor lock-in could happen, or not get too narrow in perspective (e.g. llama.cpp/open router depending on local/cloud [1] ). With the explosion of popularity of CLI tools (claude/continue/codex/kiro/etc) it still makes sen…

can you recommend a setup with ollama and a cli tool? Do you know if I need a licence for Claude if I only use my own local LLM?

we recently added a `launch` command to Ollama, so you can set up tools like Claude Code easily: https://ollama.com/blog/launch

tldr; `ollama launch claude`

glm-4.7-flash is a nice local model for this sort of thing if you have a machine that can run it

Re: Claude Code: connect to a local model when your quota runs out

#19

Post by usef- »

esafak wrote: post #14
usef- wrote: post #12

Earlier quoted context omitted.

That wouldn't solve this problem. And they do? That's what the API is. The subscription always seemed clearly advertised for client usage, not general API usage, to me. I don't know why people are surprised after hacking the auth out of the client. (note in clients they can control prompting patterns for caching etc, it can be cheaper)

End users -- people who use harnesses -- have subscriptions so that makes no sense. General API usage is for production.

"Production" what?

The API is for using the model directly with your own tools. It can be in dev, or experiments, or anything.

Subscriptions are for using the apps Claude + code. That's what it always said when you sign up.

Re: Claude Code: connect to a local model when your quota runs out

#20

Post by vorticalbox »

drifkin wrote: post #18
mogoman wrote: post #4

Earlier quoted context omitted.

can you recommend a setup with ollama and a cli tool? Do you know if I need a licence for Claude if I only use my own local LLM?

we recently added a `launch` command to Ollama, so you can set up tools like Claude Code easily: https://ollama.com/blog/launch tldr; `ollama launch claude` glm-4.7-flash is a nice local model for this sort of thing if you have a machine that can run it

I have been using glm-4.7 a bunch today and it’s actually pretty good.

I set up a bot on 4claw and although it’s kinda slow, it took twenty minutes to load 3 subs and 5 posts from each then comment on interesting ones.

It actually managed to correctly use the api via curl though at one point it got a little stuck as it didn’t escape its json.

I’m going to run it for a few days but very impressed so for for such a small model.

Post reply on HN