Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

161–170 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#161

Post by PlatoIsADisease »

K0balt wrote: post #95
PlatoIsADisease wrote: post #63

Earlier quoted context omitted.

>may I'm completely over these hypotheticals and 'testing grade'. I know Nvidia VRAM works, not some marketing about 'integrated ram'. Heck look at /r/locallama/ There is a reason its entirely Nvidia.

Mmmm, not really. I have both a4x 3090 box and a Mac m1 with 64 gb. I find that the Mac performs about the same as a 2x 3090. That’s nothing stellar, but you can run 70b models at decent quants with moderate context windows. Definitely useful for a lot of stuff.

>quants

>moderate context windows

Really had to modify the problem to make it seem equal? Not that quants are that bad, but the context windows thing is the difference between useful and not useful.

Re: Claude Code: connect to a local model when your quota runs out

#162

Post by PlatoIsADisease »

hnfong wrote: post #103
PlatoIsADisease wrote: post #63

Earlier quoted context omitted.

>may I'm completely over these hypotheticals and 'testing grade'. I know Nvidia VRAM works, not some marketing about 'integrated ram'. Heck look at /r/locallama/ There is a reason its entirely Nvidia.

> Heck look at /r/locallama/ There is a reason its entirely Nvidia. That's simply not true. NVidia may be relatively popular, but people use all sorts of hardware there. Just a random couple of recent self-reported hardware from comments: - https://www.reddit.com/r/LocalLLaMA/comments/1qw15gl/comment... - https://www.reddit.com/r/LocalLLaMA/comments/1qw0ogw/analysi... - https://www.reddit.com/r/LocalLLaMA/comments/1q…

I specifically mentioned "hypotheticals and 'testing grade'."

Then you sent over links describing such.

In real world use, Nvidia is probably over 90%.

Re: Claude Code: connect to a local model when your quota runs out

#163

Post by apercu »

bushbaba wrote: post #125
Aurornis wrote: post #104

Earlier quoted context omitted.

> but finding myself asking Sonnet to rewrite 90% of the code GLM was giving me. At some point I was like "what the hell am I doing" and just switched. This is a very common sequence of events. The frontier hosted models are so much better than everything else that it's not worth messing around with anything lesser if doing this professionally. The $20/month plans go a long way if context is managed carefully. For a…

For many companies. They’d be better to pay $200/month and layoff 1% of the workforce to pay for it.

The issue is they often choose the wrong 1%.

Re: Claude Code: connect to a local model when your quota runs out

#164

Post by samch »

sathish316 wrote: post #83

Some native Claude code options when your quota runs out: 1. Switch to extra usage, which can be increased on the Claude usage page: https://claude.ai/settings/usage 2. Logout and Switch to API tokens (using the ANTHROPIC_API_KEY environment variable) instead of a Claude Pro subscription. Credits can be increased on the Anthropic API console page: https://platform.claude.com/settings/keys 3. Add a second 20$/month ac…

I hadn’t thought about using their first-party API offering, but I will look into it.

Personally, I’ve used AWS Bedrock as the fallback when my plan runs out, and that seems to work well in my experience. I believe you can now connect to Azure as well.

Re: Claude Code: connect to a local model when your quota runs out

#165

Post by danw1979 »

polynomial wrote: post #133
reilly3000 wrote: post #30

Earlier quoted context omitted.

Which takes a $20k thunderbolt cluster of 2 512GB RAM Mac Studio Ultras to run at full quality…

Depending on what your usage requirements are, Mac Minis running UMA over RDMA is becoming a feasible option. At roughly 1/10 of the cost you're getting much much more than 1/10 the performance. (YMMV) https://buildai.substack.com/i/181542049/the-mac-mini-moment

I did not expect this to be a limiting factor in the mac mini RDMA setup ! -

> Thermal throttling: Thunderbolt 5 cables get hot under sustained 15GB/s load. After 10 minutes, bandwidth drops to 12GB/s. After 20 minutes, 10GB/s. Your 5.36 tokens/sec becomes 4.1 tokens/sec. Active cooling on cables helps but you’re fighting physics.

Thermal throttling of network cables is a new thing to me…

Re: Claude Code: connect to a local model when your quota runs out

#166

Post by miroljub »

deaux wrote: post #109
Aurornis wrote: post #104

Earlier quoted context omitted.

> but finding myself asking Sonnet to rewrite 90% of the code GLM was giving me. At some point I was like "what the hell am I doing" and just switched. This is a very common sequence of events. The frontier hosted models are so much better than everything else that it's not worth messing around with anything lesser if doing this professionally. The $20/month plans go a long way if context is managed carefully. For a…

Until last week, you would've been right. Kimi K2.5 is absolutely competitive for coding. Unless you include it in "frontier", but that has usually been used to refer to "Big 3".

I've been using MiniMax-M2.1 lately. Although benchmarks show it comparable with Kimi 2.5 and Sonnet 4.5, I find it more pleasant to use.

I still have to occasionally switch to Opus in Opencode planning mode, but not having to rely on Sonnet anymore makes my Claude subscription last much longer.

Re: Claude Code: connect to a local model when your quota runs out

#167

Post by HarHarVeryFunny »

DennisP wrote: post #72
paxys wrote: post #34

Earlier quoted context omitted.

LOCAL models. No one is running Kimi 2.5 on their Macbook or RTX 4090.

On Macbooks, no. But there are a few lunatics like this guy: https://www.youtube.com/watch?v=bFgTxr5yst0

Wow!

I've never heard of this guy before, but I see he's got 5M YouTube subscribers, which I guess is the clout you need to have Apple loan (I assume) you $50K worth of Mac Studios!

I'll be interesting to see how model sizes, capability, and local compute prices evolve.

A bit off topic, but I was in best buy the other day and was shocked to see 65" TVs selling for $300 ... I can remember the first large flat screen TVs (plasma?) selling for 100x that ($30K) when they first came out.

Re: Claude Code: connect to a local model when your quota runs out

#169

Post by kilroy123 »

EagnaIonat wrote: post #141
paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

The secret is to not run out of quota. Instead have Claude know when to offload work to local models and what model is best suited for the job. It will shape the prompt for the model. Then have Claude review the results. Massive reduction in costs. btw, at least on Macbooks you can run good models with just M1 32GB of memory.

I strongly think you're on to something here. I wish Apple would invest heavily in something like this.

The big powerful models think about tasks, then offload some stuff to a drastically cheaper cloud model or the model running on your hardware.

Re: Claude Code: connect to a local model when your quota runs out

#170

Post by danw1979 »

DennisP wrote: post #72
paxys wrote: post #34

Earlier quoted context omitted.

LOCAL models. No one is running Kimi 2.5 on their Macbook or RTX 4090.

On Macbooks, no. But there are a few lunatics like this guy: https://www.youtube.com/watch?v=bFgTxr5yst0

He must be mad, accepting $50k of free (probably loaned?) hardware from Apple !

Great demo video though. Nice to see some benchmarks of Exo with this cluster across various models.

Post reply on HN