Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

181–190 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#181

Post by eek2121 »

BuildTheRobots wrote: post #147
EagnaIonat wrote: post #141

Earlier quoted context omitted.

The secret is to not run out of quota. Instead have Claude know when to offload work to local models and what model is best suited for the job. It will shape the prompt for the model. Then have Claude review the results. Massive reduction in costs. btw, at least on Macbooks you can run good models with just M1 32GB of memory.

I don't suppose you could point to any resources on where I could get started. I have a M2 with 64gb of unified memory and it'd be nice to make it work rather than burning Github credits.

LM Studio is fantastic for playing with local models.

Re: Claude Code: connect to a local model when your quota runs out

#182

Post by eek2121 »

paxys wrote: post #114
CamperBob2 wrote: post #62

Earlier quoted context omitted.

Not the GP but the new Qwen-Coder-Next release feels like a step change, at 60 tokens per second on a single 96GB Blackwell. And that's at full 8-bit quantization and 256K context, which I wasn't sure was going to work at all. It is probably enough to handle a lot of what people use the big-3 closed models for. Somewhat slower and somewhat dumber, granted, but still extraordinarily capable. It punches way above its w…

"Single 96GB Blackwell" is still $15K+ worth of hardware. You'd have to use it at full capacity for 5-10 years to break even when compared to "Max" plans from OpenAI/Anthropic/Google. And you'd still get nowhere near the quality of something like Opus. Yes there are plenty of valid arguments in favor of self hosting, but at the moment value simply isn't one of them.

I run it on my machine, which has a a 4090 and 64gb RAM.

Re: Claude Code: connect to a local model when your quota runs out

#183

Post by deaux »

tkz1312 wrote: post #179
bigiain wrote: post #115

Earlier quoted context omitted.

Looks like you need at least a quarter terabyte or so of ram to run that though? (At todays ram prices upgrading to that for me would pay for a _lot_ of tokens...)

unfortunately running anything locally for serious personal use makes no financial sense at all right now. 4x rtx 6000 pro is probably the minimum you need to have something reasonable for coding work.

That's the setup you want for serious work yes, so probably $60kish all-in(?). Which is a big chunk of money for an individual, but potentially quite reasonable for a company. Being able to get effectively _frontier-level local performance_ for that money was completely unthinkable so far. Correct me if I'm wrong, but I think Deepseek R1 hardware requirements were far costlier on release, and it had a much bigger gap to market lead than Kimi K2.5. If this trend continues the big 3 are absolutely finished when it comes to enterprise and they'll only have consumer left. Altman and Amodei will be praying to the gods that China doesn't keep this rate of performance/$ improvement up while also releasing all as open weights.

Re: Claude Code: connect to a local model when your quota runs out

#184

Post by altern8 »

paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

I was wondering the same thing, e.g. if it takes tens or hundreds of millions of dollars to train and keep a model up-to-date, how can an open source one compete with that?

Re: Claude Code: connect to a local model when your quota runs out

#185

Post by deaux »

ianlevesque wrote: post #144
deaux wrote: post #122

Earlier quoted context omitted.

I disagree, based on having used it extensively over the last week. I find it to be at least as strong as Sonnet 4.5 and 5.2-Codex on the majority of tasks, often better. Note that even among the big 3, each of them has a domain where they're better than the other two. It's not better than Codex (x-)high at debugging non-UI code - but neither is Opus or Gemini. It's not better than Gemini at UI design - but neither i…

Yeah Kimi-K2.5 is the first open weights model that actually feels competitive with the closed models, and I've tried a lot of them now.

Same, I'm still not sure where it shines though. In each of the three big domains I named, the respective top performing closed model still seems to have the edge. That keeps me from reaching for it more often. Fantastic all-rounder for sure.

Re: Claude Code: connect to a local model when your quota runs out

#186

Post by AlwaysRock »

Not exactly the same but I wish copilot/github allowed you to have two plans. A company sponsored plan and your own plan. If I run out of requests on my company plan I should be able to use my own plan. Likewise, If I have 1 github account that is used for work and non work code, I should be able to route copilot to use a company or personal plan.

Re: Claude Code: connect to a local model when your quota runs out

#187

Post by Larrikin »

AlwaysRock wrote: post #186

Not exactly the same but I wish copilot/github allowed you to have two plans. A company sponsored plan and your own plan. If I run out of requests on my company plan I should be able to use my own plan. Likewise, If I have 1 github account that is used for work and non work code, I should be able to route copilot to use a company or personal plan.

Why would you want to mix your personal plan with your company plan and subject yourself to the company auditing your personal GitHub, computer, etc. If the company wants you using LLMs then they should pay for it and increase your limits.

Re: Claude Code: connect to a local model when your quota runs out

#188

Post by fourside »

AlwaysRock wrote: post #186

Not exactly the same but I wish copilot/github allowed you to have two plans. A company sponsored plan and your own plan. If I run out of requests on my company plan I should be able to use my own plan. Likewise, If I have 1 github account that is used for work and non work code, I should be able to route copilot to use a company or personal plan.

It’s wild to me that you’d want to spend your personal money to use productivity tools for work. If your work machine broke would your first instinct be to buy your own replacement or to have work pay for it?

Re: Claude Code: connect to a local model when your quota runs out

#189

Post by shen »

EagnaIonat wrote: post #150
BuildTheRobots wrote: post #147

Earlier quoted context omitted.

I don't suppose you could point to any resources on where I could get started. I have a M2 with 64gb of unified memory and it'd be nice to make it work rather than burning Github credits.

https://ollama.com Although I'm starting to like LMStudio more, as it has more features that Ollama is missing. https://lmstudio.ai You can then get Claude to create the MCP server to talk to either. Then a CLAUDE.md that tells it to read the models you have downloaded, determine their use and when to offload. Claude will make all that for you as well.

Which local models are you using for the 32gb MacBooks?

Re: Claude Code: connect to a local model when your quota runs out

#190

Post by hnfong »

PlatoIsADisease wrote: post #162
hnfong wrote: post #103

Earlier quoted context omitted.

> Heck look at /r/locallama/ There is a reason its entirely Nvidia. That's simply not true. NVidia may be relatively popular, but people use all sorts of hardware there. Just a random couple of recent self-reported hardware from comments: - https://www.reddit.com/r/LocalLLaMA/comments/1qw15gl/comment... - https://www.reddit.com/r/LocalLLaMA/comments/1qw0ogw/analysi... - https://www.reddit.com/r/LocalLLaMA/comments/1q…

I specifically mentioned "hypotheticals and 'testing grade'." Then you sent over links describing such. In real world use, Nvidia is probably over 90%.

r/locallamma/ is not entirely Nvidia.

You have a point that at scale everybody except maybe Google is using Nvidia. But r/locallama is not your evidence of that, unless you apply your priors, filter out all the hardware that don't fit your so called "hypotheticals and 'testing grade'" criteria, and engage in circular logic.

PS: In fact locallamma does not even cover your "real world use". Most mentions of Nvidia are people who have older GPUs eg. 3090s lying around, or are looking at the Chinese VRAM mods to allow them run larger models. Nobody is discussing how to run a cluster of H200s there.

Post reply on HN