Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

31–40 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#31

Post by teaearlgraycold »

zozbot234 wrote: post #29
paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago. That's not "nothing" and is plenty good enough for everyday work.

Having used K2.5 I’d judge it to be a little better than that. Maybe as good as proprietary models from last June?

Re: Claude Code: connect to a local model when your quota runs out

#32

Post by teaearlgraycold »

reilly3000 wrote: post #30
zozbot234 wrote: post #29

Earlier quoted context omitted.

The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago. That's not "nothing" and is plenty good enough for everyday work.

Which takes a $20k thunderbolt cluster of 2 512GB RAM Mac Studio Ultras to run at full quality…

Which while expensive is dirt cheap compared to a comparable NVidia or AMD system.

Re: Claude Code: connect to a local model when your quota runs out

#33

Post by eli »

usef- wrote: post #19
esafak wrote: post #14

Earlier quoted context omitted.

End users -- people who use harnesses -- have subscriptions so that makes no sense. General API usage is for production.

"Production" what? The API is for using the model directly with your own tools. It can be in dev, or experiments, or anything. Subscriptions are for using the apps Claude + code. That's what it always said when you sign up.

Production = people who can afford to pay API rates for a coding harness

Re: Claude Code: connect to a local model when your quota runs out

#34

Post by paxys »

zozbot234 wrote: post #29
paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago. That's not "nothing" and is plenty good enough for everyday work.

LOCAL models. No one is running Kimi 2.5 on their Macbook or RTX 4090.

Re: Claude Code: connect to a local model when your quota runs out

#35

Post by usef- »

eli wrote: post #33
usef- wrote: post #19

Earlier quoted context omitted.

"Production" what? The API is for using the model directly with your own tools. It can be in dev, or experiments, or anything. Subscriptions are for using the apps Claude + code. That's what it always said when you sign up.

Production = people who can afford to pay API rates for a coding harness

Saying their prices are too high is an understandable complaint; I'm only arguing against the complaint that people were stopped from hacking the subscriptions.

LLMs are a hyper-competitive market at the moment, and we have a wealth of options, so if Anthropic is overpricing their API they'll likely be hurting themselves.

Re: Claude Code: connect to a local model when your quota runs out

#36

Post by blharr »

teaearlgraycold wrote: post #32
reilly3000 wrote: post #30

Earlier quoted context omitted.

Which takes a $20k thunderbolt cluster of 2 512GB RAM Mac Studio Ultras to run at full quality…

Which while expensive is dirt cheap compared to a comparable NVidia or AMD system.

What speed are you getting at that level of hardware though?

Re: Claude Code: connect to a local model when your quota runs out

#37

Post by nik282000 »

paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

> intelligence

Whether it's a giant corporate model or something you run locally, there is no intelligence there. It's still just a lying engine. It will tell you the string of tokens most likely to come after your prompt based on training data that was stolen and used against the wishes of its original creators.

Re: Claude Code: connect to a local model when your quota runs out

#38

Post by SchemaLoad »

teaearlgraycold wrote: post #32
reilly3000 wrote: post #30

Earlier quoted context omitted.

Which takes a $20k thunderbolt cluster of 2 512GB RAM Mac Studio Ultras to run at full quality…

Which while expensive is dirt cheap compared to a comparable NVidia or AMD system.

It's still very expensive compared to using the hosted models which are currently massively subsidised. Have to wonder what the fair market price for these hosted models will be after the free money dries up.

Re: Claude Code: connect to a local model when your quota runs out

#39

Post by [deleted] »

alexhans wrote: post #7
mogoman wrote: post #4

Earlier quoted context omitted.

can you recommend a setup with ollama and a cli tool? Do you know if I need a licence for Claude if I only use my own local LLM?

What are your needs/constraints (hardware constraints definitely a big one)? The one I mentioned called continue.dev [1] is easy to try out and see if it meets your needs. Hitting local models with it should be very easy (it calls APIs at a specific port) [1] - https://github.com/continuedev/continue

[deleted]
Post reply on HN