Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

91–100 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#91

Post by seanmcdirmid »

paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

> (ones you run on beefy 128GB+ RAM machines)

PC or Mac? A PC, ya, no way, not without beefy GPUs with lots of VRAM. A mac? Depends on the CPU, an M3 Ultra with 128GB of unified RAM is going to get closer, at least. You can have decent experiences with a Max CPU + 64GB of unified RAM (well, that's my setup at least).

Re: Claude Code: connect to a local model when your quota runs out

#92

Post by bigyabai »

MuffinFlavored wrote: post #90
0xbadcafebee wrote: post #79

Earlier quoted context omitted.

Most benchmarks show very little improvement of "full quality" over a quantized lower-bit model. You can shrink the model to a fraction of its "full" size and get 92-95% same performance, with less VRAM use.

> You can shrink the model to a fraction of its "full" size and get 92-95% same performance, with less VRAM use. Are there a lot of options how "how far" do you quantize? How much VRAM does it take to get the 92-95% you are speaking of?

> Are there a lot of options how "how far" do you quantize?

So many: https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie...

> How much VRAM does it take to get the 92-95% you are speaking of?

For inference, it's heavily dependent on the size of the weights (plus context). Quantizing an f32 or f16 model to q4/mxfp4 won't necessarily use 92-95% less VRAM, but it's pretty close for smaller contexts.

Re: Claude Code: connect to a local model when your quota runs out

#93

Post by QuantumNomad_ »

seanmcdirmid wrote: post #91
paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

> (ones you run on beefy 128GB+ RAM machines) PC or Mac? A PC, ya, no way, not without beefy GPUs with lots of VRAM. A mac? Depends on the CPU, an M3 Ultra with 128GB of unified RAM is going to get closer, at least. You can have decent experiences with a Max CPU + 64GB of unified RAM (well, that's my setup at least).

Which models do you use, and how do you run them?

Re: Claude Code: connect to a local model when your quota runs out

#94

Post by parthsareen »

htsh wrote: post #46
hkpatel3 wrote: post #8

Openrouter can also be used with claude code. https://openrouter.ai/docs/guides/claude-code-integration

thanks! came in here to ask this. we can do much better with a cheap model on openrouter (glm 4.7, kimi, etc.) than anything that I can run on my lowly 3090 :)

Also recently added ollama launch claude if you want to connect to cloud models from there :)

Re: Claude Code: connect to a local model when your quota runs out

#95

Post by K0balt »

PlatoIsADisease wrote: post #63
zozbot234 wrote: post #59

Earlier quoted context omitted.

70B dense models are way behind SOTA. Even the aforementioned Kimi 2.5 has fewer active parameters than that, and then quantized at int4. We're at a point where some near-frontier models may run out of the box on Mac Mini-grade hardware, with perhaps no real need to even upgrade to the Mac Studio.

>may I'm completely over these hypotheticals and 'testing grade'. I know Nvidia VRAM works, not some marketing about 'integrated ram'. Heck look at /r/locallama/ There is a reason its entirely Nvidia.

Mmmm, not really. I have both a4x 3090 box and a Mac m1 with 64 gb. I find that the Mac performs about the same as a 2x 3090. That’s nothing stellar, but you can run 70b models at decent quants with moderate context windows. Definitely useful for a lot of stuff.

Re: Claude Code: connect to a local model when your quota runs out

#96

Post by plagiarist »

bdangubic wrote: post #80
teaearlgraycold wrote: post #68

Earlier quoted context omitted.

For sure Claude Code isn’t profitable

Neither was Uber and … and …

Businesses will desire me for my insomnia once Anthropics starts charging congestion pricing.

Re: Claude Code: connect to a local model when your quota runs out

#97

Post by raw_anon_1111 »

0xbadcafebee wrote: post #84
raw_anon_1111 wrote: post #9

Or just don’t use Claude Code and use Codex CLI. I have yet to hit a quota with Codex working all day. I hit the Claude limits within an hour or less. This is with my regular $20/month ChatGpT subscription and my $200 a year (company reimbursed) Claude subscription.

You're getting downvoted because people here don't know that the specific agent you pick can pollute your context and waste your tokens. Claude's system prompt is enormous, to say nothing of things like context windows and hidden subagents.

I am using Codex-cli with my regular $20 a month ChatGPT subscription. Never once had to worry about tokens, request etc. I logged in with my regular ChatGPT account and didn’t have to use an API key

Re: Claude Code: connect to a local model when your quota runs out

#99

Post by MuffinFlavored »

bigyabai wrote: post #92
MuffinFlavored wrote: post #90

Earlier quoted context omitted.

> You can shrink the model to a fraction of its "full" size and get 92-95% same performance, with less VRAM use. Are there a lot of options how "how far" do you quantize? How much VRAM does it take to get the 92-95% you are speaking of?

> Are there a lot of options how "how far" do you quantize? So many: https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie... > How much VRAM does it take to get the 92-95% you are speaking of? For inference, it's heavily dependent on the size of the weights (plus context). Quantizing an f32 or f16 model to q4/mxfp4 won't necessarily use 92-95% less VRAM, but it's pretty close for smaller contexts.

Thank you. Could you give a tl;dr on "the full model needs ____ this much VRAM and if you do _____ the most common quantization method it will run in ____ this much VRAM" rough estimate please?

Re: Claude Code: connect to a local model when your quota runs out

#100

Post by girvo »

0xbadcafebee wrote: post #82
paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

The best local models are literally right behind Claude/Gemini/Codex. Check the benchmarks. That said, Claude Code is designed to work with Anthropic's models. Agents have a buttload of custom work going on in the background to massage specific models to do things well.

The benchmarks simply do not match my experience though. I don’t put that much stock in them anymore.
Post reply on HN