Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

171–180 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#171

Post by Balinares »

girvo wrote: post #100
0xbadcafebee wrote: post #82

Earlier quoted context omitted.

The best local models are literally right behind Claude/Gemini/Codex. Check the benchmarks. That said, Claude Code is designed to work with Anthropic's models. Agents have a buttload of custom work going on in the background to massage specific models to do things well.

The benchmarks simply do not match my experience though. I don’t put that much stock in them anymore.

I've repeatedly seen Opus 4.5 manufacture malpractice and then disable the checks complaining about it in order to be able to declare the job done, so I would agree with you about benchmarks versus experience.

Re: Claude Code: connect to a local model when your quota runs out

#173

Post by bdangubic »

plagiarist wrote: post #96
bdangubic wrote: post #80

Earlier quoted context omitted.

Neither was Uber and … and …

Businesses will desire me for my insomnia once Anthropics starts charging congestion pricing.

that is coming for sure to replace the "500" errors

Re: Claude Code: connect to a local model when your quota runs out

#174

Post by holoduke »

auggierose wrote: post #154
holoduke wrote: post #149

Earlier quoted context omitted.

I now have 3 x 100 plans. Only then I an able to full time use it. Otherwise I hit the limits. I am q heavy user. Often work on 5 apps at the same time.

Shouldn't the 200 plan give you 4x?? Why 3 x 100 then?

Good point. Need to look into that one. Pricing is also changing constantly with Claude

Re: Claude Code: connect to a local model when your quota runs out

#175

Post by dec0dedab0de »

cyanydeez wrote: post #11
alexhans wrote: post #3

Useful tip. From a strategic standpoint of privacy, cost and control, I immediately went for local models, because that allowed to baseline tradeoffs and it also made it easier to understand where vendor lock-in could happen, or not get too narrow in perspective (e.g. llama.cpp/open router depending on local/cloud [1] ). With the explosion of popularity of CLI tools (claude/continue/codex/kiro/etc) it still makes sen…

I think control should be top of the list here. You're talking about building work flows, products and long term practices around something that's inherently non-deterministic. And the probability that any given model you use today is the same as what you use tomorrow is doubly doubtful: 1. The model itself will change as they try to improve the cost-per-test improves. This will necessarily make your expectations non…

It's akin to anti-union arguments where everyone "buying" into the cloud AI circus thinks they're going to strike gold and completely ignores the fact that very few will and if they really wanted a better world and more control, they'd unionize and limit their illusions of grandeur.

Most Anti-Union arguments I have heard have been about them charging too much in dues, union leadership cozying up to management, and them acting like organized crime doing things like smashing windows of non-union jobs. I have never heard anyone be against unions because they thought they would make it rich on their own.

Re: Claude Code: connect to a local model when your quota runs out

#176

Post by lostmsu »

andai wrote: post #106
MuffinFlavored wrote: post #88

Earlier quoted context omitted.

Did you eventually move to a $20/mo Claude plan, $100/mo Claude plan, $200/mo, or API based? if API based, how much are you averaging a month?

The $20 one, but it's hobby use for me, would probably need the $200 one if I was full time. Ran into the 5 hour limit in like 30 minutes the other day. I've also been testing OpenClaw. It burned 8M tokens during my half hour of testing, which would have been like $50 with Opus on the API. (Which is why everyone was using it with the sub, until Anthropic apparently banned that.) I was using GLM on Cerebras instead, s…

Try Codex. It's better (subjectively, but objectively they are in the same ballpark), and its $20 plan is way more generous. I can use gpt-5.2 on high (prefer overall smarter models to -codex coding ones) almost nonstop, sometimes a few in parallel before I hit any limits (if ever).

Re: Claude Code: connect to a local model when your quota runs out

#177

Post by lostmsu »

paxys wrote: post #114
CamperBob2 wrote: post #62

Earlier quoted context omitted.

Not the GP but the new Qwen-Coder-Next release feels like a step change, at 60 tokens per second on a single 96GB Blackwell. And that's at full 8-bit quantization and 256K context, which I wasn't sure was going to work at all. It is probably enough to handle a lot of what people use the big-3 closed models for. Somewhat slower and somewhat dumber, granted, but still extraordinarily capable. It punches way above its w…

"Single 96GB Blackwell" is still $15K+ worth of hardware. You'd have to use it at full capacity for 5-10 years to break even when compared to "Max" plans from OpenAI/Anthropic/Google. And you'd still get nowhere near the quality of something like Opus. Yes there are plenty of valid arguments in favor of self hosting, but at the moment value simply isn't one of them.

If you are not planning to batch, you can run it much cheaper with Ryzen AI Max SoC devices.

Hell, if you are willing to go even slower, any GPU + ~80GB of RAM will do it.

Re: Claude Code: connect to a local model when your quota runs out

#178

Post by cat_plus_plus »

paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

Depends on whether you want a programmer or a therapist. Given clear description of class structure and key algorithms, Qwen3-Code is way more likely to do exactly what is being asked than any Gemini model. If you want to turn a vague idea into a design, yeah cloud bot is better. Let's not forget that cloud bots have web search, if you hook up a local model to GPT Researcher or Onyx frontend, you will see reasonable performance, although open ended research is where cloud model scale does pay off. Provided it actually bothers to search rather than hallucinating to save backend costs. Also local uncensored model is way better at doing proper security analysis of your app / network.

Re: Claude Code: connect to a local model when your quota runs out

#179

Post by tkz1312 »

bigiain wrote: post #115
deaux wrote: post #109

Earlier quoted context omitted.

Until last week, you would've been right. Kimi K2.5 is absolutely competitive for coding. Unless you include it in "frontier", but that has usually been used to refer to "Big 3".

Looks like you need at least a quarter terabyte or so of ram to run that though? (At todays ram prices upgrading to that for me would pay for a _lot_ of tokens...)

unfortunately running anything locally for serious personal use makes no financial sense at all right now.

4x rtx 6000 pro is probably the minimum you need to have something reasonable for coding work.

Re: Claude Code: connect to a local model when your quota runs out

#180

Post by cat_plus_plus »

danw1979 wrote: post #165
polynomial wrote: post #133

Earlier quoted context omitted.

Depending on what your usage requirements are, Mac Minis running UMA over RDMA is becoming a feasible option. At roughly 1/10 of the cost you're getting much much more than 1/10 the performance. (YMMV) https://buildai.substack.com/i/181542049/the-mac-mini-moment

I did not expect this to be a limiting factor in the mac mini RDMA setup ! - > Thermal throttling: Thunderbolt 5 cables get hot under sustained 15GB/s load. After 10 minutes, bandwidth drops to 12GB/s. After 20 minutes, 10GB/s. Your 5.36 tokens/sec becomes 4.1 tokens/sec. Active cooling on cables helps but you’re fighting physics. Thermal throttling of network cables is a new thing to me…

I admire patience of anyone who runs dense models on unified memory. Personally, I would rather feed an entire programming book or code directory to a sparse model and get an answer in 30 seconds and then use cloud in rare cases it's not enough.
Post reply on HN