Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

191–200 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#191

Post by gpm »

altern8 wrote: post #184
paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

I was wondering the same thing, e.g. if it takes tens or hundreds of millions of dollars to train and keep a model up-to-date, how can an open source one compete with that?

Less than a billion of dollars to become the arbiter of truth probably sounds like a great deal to the well off dictatorial powers of the world. So long as models can be trained to have a bias (and it's hard to see that going away) I'd be pretty surprised if they stop being released for free.

Which definitely has some questionable implications... but just like with advertising it's not like paying makes the incentives for the people capable of training models to put their thumbs on the scales go away.

Re: Claude Code: connect to a local model when your quota runs out

#192

Post by the_harpia_io »

sReinwald wrote: post #153
the_harpia_io wrote: post #136

[flagged]

Not saying the frontier models aren't smarter than the ones I can run on my two 4090s (they absolutely are) but I feel like you're exaggerating the security implications a bit. We've seen some absolutely glaring security issues with vibe-coded apps / websites that did use Claude (most recently Moltbook). No matter whether you're vibe coding with frontier models or local ones, you simply cannot rely on the model knowi…

[flagged]

Re: Claude Code: connect to a local model when your quota runs out

#193

Post by the_harpia_io »

tossandthrow wrote: post #138
the_harpia_io wrote: post #136

[flagged]

Yes, models are aligned differently. But that is a quality of the model. Obviously it must be assumed that the model one falls back on is good enough - including security alignment.

[flagged]

Re: Claude Code: connect to a local model when your quota runs out

#194

Post by the_harpia_io »

Zardoz84 wrote: post #137
the_harpia_io wrote: post #136

[flagged]

I would always prefer something local. By definition it's more secure, as you are not sending your code on the wire to a third party server, and hope that they comply with the "We will not train our models with your data".

[flagged]

Re: Claude Code: connect to a local model when your quota runs out

#195

Post by reedlaw »

I've already tried to do what the article claims to be doing: handing-off the context of the current session to another model. I tried various combinations of hooks, prompts and workarounds, but nothing worked like the first screenshot in the article implies ("You've hit your limit [...] Use an open source local LLM"). The best I could come up with is to watch for the warning of high usage and then ask Claude to create a HANDOFF.md with the current context. Then I could load that into another model. Anyone have any better solutions?

Re: Claude Code: connect to a local model when your quota runs out

#196

Post by polynomial »

danw1979 wrote: post #165
polynomial wrote: post #133

Earlier quoted context omitted.

Depending on what your usage requirements are, Mac Minis running UMA over RDMA is becoming a feasible option. At roughly 1/10 of the cost you're getting much much more than 1/10 the performance. (YMMV) https://buildai.substack.com/i/181542049/the-mac-mini-moment

I did not expect this to be a limiting factor in the mac mini RDMA setup ! - > Thermal throttling: Thunderbolt 5 cables get hot under sustained 15GB/s load. After 10 minutes, bandwidth drops to 12GB/s. After 20 minutes, 10GB/s. Your 5.36 tokens/sec becomes 4.1 tokens/sec. Active cooling on cables helps but you’re fighting physics. Thermal throttling of network cables is a new thing to me…

Luckily we're having a record cold winter and your setup can double as a personal space heater.

Re: Claude Code: connect to a local model when your quota runs out

#197

Post by tracker1 »

paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

For my relatively limited exposure, I'm not sure if I'd be able to tolerate it. I've found Claude/Opus to e pretty nice to work with... by contrast, I find Github Copilot to be the most annoying thing I've ever tried to work with.

Because of how the plugin works in VS code, on my third day of testing with Claude Code, I didn't click the Claude button and was accidentally working with CoPilot for about three hours of torture when I realized I wasn't in Claude Code. Will NEVER make that mistake again... I can only imagine anything I can run at any decent speed lcoally will be closer to the latter. I pretty quickly reach a "I can do this faster/better myself" point... even a few times with Claude/Opus, so my patience isn't always the greatest.

That said, I love how easy it is to build up a scaffold of a boilerplate app for the sole reason to test a single library/function in isolation from a larger application. In 5-10 minutes, I've got enough test harness around what I'm trying to work on/solve that it lets me focus on the problem at hand, while not worrying about doing this on the integrated larger project.

I've still got some thinking and experimenting to do with improving some of my workflows... but I will say that AI Assist has definitely been a multiplier in terms of my own productivity. At this point, there's literally no excuse not to have actual code running experiments when learning something new, connecting to something you haven't used before... etc. in terms of working on a solution to a problem. Assuming you have at least a rudimentary understanding of what you're actually trying to accomplish in the piece you are working on. I still don't have enough trust to use AI to build a larger system, or for that matter to truly just vibe code anything.

Re: Claude Code: connect to a local model when your quota runs out

#198

Post by CamperBob2 »

eek2121 wrote: post #182
paxys wrote: post #114

Earlier quoted context omitted.

"Single 96GB Blackwell" is still $15K+ worth of hardware. You'd have to use it at full capacity for 5-10 years to break even when compared to "Max" plans from OpenAI/Anthropic/Google. And you'd still get nowhere near the quality of something like Opus. Yes there are plenty of valid arguments in favor of self hosting, but at the moment value simply isn't one of them.

I run it on my machine, which has a a 4090 and 64gb RAM.

How fast is it?

Re: Claude Code: connect to a local model when your quota runs out

#199

Post by tracker1 »

deaux wrote: post #183
tkz1312 wrote: post #179

Earlier quoted context omitted.

unfortunately running anything locally for serious personal use makes no financial sense at all right now. 4x rtx 6000 pro is probably the minimum you need to have something reasonable for coding work.

That's the setup you want for serious work yes, so probably $60kish all-in(?). Which is a big chunk of money for an individual, but potentially quite reasonable for a company. Being able to get effectively _frontier-level local performance_ for that money was completely unthinkable so far. Correct me if I'm wrong, but I think Deepseek R1 hardware requirements were far costlier on release, and it had a much bigger gap…

I'm not so sure on that... even if one $60k machine can handle the load of 5 developers at a time, you're still looking at 5 years of service to recoup $200/mo/dev and that doesn't even consider other improvements to hardware or the models service providers offer over that same period of time.

I'd probably rather save the capex, and use the rented service until something much more compelling comes along.

Post reply on HN