Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

121–130 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#121

Post by omneity »

MuffinFlavored wrote: post #99
bigyabai wrote: post #92

Earlier quoted context omitted.

> Are there a lot of options how "how far" do you quantize? So many: https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie... > How much VRAM does it take to get the 92-95% you are speaking of? For inference, it's heavily dependent on the size of the weights (plus context). Quantizing an f32 or f16 model to q4/mxfp4 won't necessarily use 92-95% less VRAM, but it's pretty close for smaller contexts.

Thank you. Could you give a tl;dr on "the full model needs ____ this much VRAM and if you do _____ the most common quantization method it will run in ____ this much VRAM" rough estimate please?

It’s a trivial calculation to make (+/- 10%).

Number of params == “variables” in memory

VRAM footprint ~= number of params * size of a param

A 4B model at 8 bits will result in 4GB vram give or take, same as params. At 4 bits ~= 2GB and so on. Kimi is about 512GB at 4 bits.

Re: Claude Code: connect to a local model when your quota runs out

#122

Post by deaux »

Aurornis wrote: post #117
deaux wrote: post #109

Earlier quoted context omitted.

Until last week, you would've been right. Kimi K2.5 is absolutely competitive for coding. Unless you include it in "frontier", but that has usually been used to refer to "Big 3".

> Kimi K2.5 is absolutely competitive for coding. Kimi K2.5 is good, but it's still behind the main models like Claude's offerings and GPT-5.2. Yes, I know what the benchmarks say, but the benchmarks for open weight models have been overpromising for a long time and Kimi K2.5 is no exception. Kimi K2.5 is also not something you can easily run locally without investing $5-10K or more. There are hosted options you can…

I disagree, based on having used it extensively over the last week. I find it to be at least as strong as Sonnet 4.5 and 5.2-Codex on the majority of tasks, often better. Note that even among the big 3, each of them has a domain where they're better than the other two. It's not better than Codex (x-)high at debugging non-UI code - but neither is Opus or Gemini. It's not better than Gemini at UI design - but neither is Opus or Codex. It's not better than Opus at tool usage and delegation - but neither is Gemini or Codex.

Re: Claude Code: connect to a local model when your quota runs out

#123

Post by seanmcdirmid »

QuantumNomad_ wrote: post #93
seanmcdirmid wrote: post #91

Earlier quoted context omitted.

> (ones you run on beefy 128GB+ RAM machines) PC or Mac? A PC, ya, no way, not without beefy GPUs with lots of VRAM. A mac? Depends on the CPU, an M3 Ultra with 128GB of unified RAM is going to get closer, at least. You can have decent experiences with a Max CPU + 64GB of unified RAM (well, that's my setup at least).

Which models do you use, and how do you run them?

I have a M3 max 64GB.

For VS Code code completion in Continue using a Qwen3-coder 7b model. For CLI work Qwen coder 32b for sidebar. 8 bit quant for both.

I need to take a look at Qwen3-coder-next, it is supposed to have made things much faster with a larger model.

Re: Claude Code: connect to a local model when your quota runs out

#124

Post by bandrami »

mvkel wrote: post #107

Why anyone wouldn't want to be using the SOTA model at all times baffles me. Going dumb/cheap just ends up costing more, in the short and long term.

Some of us don't like sending all of our business processes to a remote vendor

Re: Claude Code: connect to a local model when your quota runs out

#125

Post by bushbaba »

Aurornis wrote: post #104
andai wrote: post #86

Earlier quoted context omitted.

Yeah this is why I ended up getting Claude subscription in the first place. I was using GLM on ZAI coding plan (jerry rigged Claude Code for $3/month), but finding myself asking Sonnet to rewrite 90% of the code GLM was giving me. At some point I was like "what the hell am I doing" and just switched. To clarify, the code I was getting before mostly worked, it was just a lot less pleasant to look at and work with. Mig…

> but finding myself asking Sonnet to rewrite 90% of the code GLM was giving me. At some point I was like "what the hell am I doing" and just switched. This is a very common sequence of events. The frontier hosted models are so much better than everything else that it's not worth messing around with anything lesser if doing this professionally. The $20/month plans go a long way if context is managed carefully. For a…

For many companies. They’d be better to pay $200/month and layoff 1% of the workforce to pay for it.

Re: Claude Code: connect to a local model when your quota runs out

#127

Post by majormajor »

paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

The amount of "prompting" stuff (meta-prompting?) the "thinking" models do behind the scenes even beyond what the harnesses do is massive; you could of course rebuild it locally, but it's gonna make it just that much slower.

I expect it'll come along but I'm not gonna spend the $$$$ necessary to try to DIY it just yet.

Re: Claude Code: connect to a local model when your quota runs out

#128

Post by monch1962 »

What are peoples' current suggestions for using Claude Code with a locally hosted LLM running on regular consumer hardware (for the sake of discussion, assume you're spending $US500-ish on a mini PC, which would get you a reasonably decent CPU, 32Gb RAM and a cheapish GPU)?

I get that it's not going to work as well as hosted/subscription services like Claude/Gemini/Codex/..., but sometimes those aren't an option

Re: Claude Code: connect to a local model when your quota runs out

#129

Post by [deleted] »

paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

[deleted]

Re: Claude Code: connect to a local model when your quota runs out

#130

Post by nico »

Would love to be able to have customizable local model that only knows the stack I’m using

For example choosing a model that knows rails 8 and react development running on a mac and using docker

Ideally that would make the model small enough to be competitive running locally

Post reply on HN