Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code: connect to a local model when your quota runs out

boxc.net

Post reply on HN
Print view View original

61–70 of 223 posts

Re: Claude Code: connect to a local model when your quota runs out

#61

Post by [deleted] »

bityard wrote: post #52
paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

Correct, a rack full of datacenter equipment is not going to compete with anything that fits on your desk or lap. Well spotted. But as a counterpoint: there are whole communities of people in this space who get significant value from models they run locally. I am one of them.

[deleted]

Re: Claude Code: connect to a local model when your quota runs out

#62

Post by CamperBob2 »

Gravey wrote: post #58
bityard wrote: post #52

Earlier quoted context omitted.

Correct, a rack full of datacenter equipment is not going to compete with anything that fits on your desk or lap. Well spotted. But as a counterpoint: there are whole communities of people in this space who get significant value from models they run locally. I am one of them.

Would you mind sharing your hardware setup and use case(s)?

Not the GP but the new Qwen-Coder-Next release feels like a step change, at 60 tokens per second on a single 96GB Blackwell. And that's at full 8-bit quantization and 256K context, which I wasn't sure was going to work at all.

It is probably enough to handle a lot of what people use the big-3 closed models for. Somewhat slower and somewhat dumber, granted, but still extraordinarily capable. It punches way above its weight class for an 80B model.

Re: Claude Code: connect to a local model when your quota runs out

#63

Post by PlatoIsADisease »

zozbot234 wrote: post #59
PlatoIsADisease wrote: post #54

Earlier quoted context omitted.

[flagged]

70B dense models are way behind SOTA. Even the aforementioned Kimi 2.5 has fewer active parameters than that, and then quantized at int4. We're at a point where some near-frontier models may run out of the box on Mac Mini-grade hardware, with perhaps no real need to even upgrade to the Mac Studio.

>may

I'm completely over these hypotheticals and 'testing grade'.

I know Nvidia VRAM works, not some marketing about 'integrated ram'. Heck look at /r/locallama/ There is a reason its entirely Nvidia.

Re: Claude Code: connect to a local model when your quota runs out

#64

Post by zozbot234 »

CamperBob2 wrote: post #62
Gravey wrote: post #58

Earlier quoted context omitted.

Would you mind sharing your hardware setup and use case(s)?

Not the GP but the new Qwen-Coder-Next release feels like a step change, at 60 tokens per second on a single 96GB Blackwell. And that's at full 8-bit quantization and 256K context, which I wasn't sure was going to work at all. It is probably enough to handle a lot of what people use the big-3 closed models for. Somewhat slower and somewhat dumber, granted, but still extraordinarily capable. It punches way above its w…

IIRC, that new Qwen model has 3B active parameters so it's going to run well enough even on far less than 96GB VRAM. (Though more VRAM may of course help wrt. enabling the full available context length.) Very impressive work from the Qwen folks.

Re: Claude Code: connect to a local model when your quota runs out

#65

Post by sealeck »

PlatoIsADisease wrote: post #54
reilly3000 wrote: post #30

Earlier quoted context omitted.

Which takes a $20k thunderbolt cluster of 2 512GB RAM Mac Studio Ultras to run at full quality…

[flagged]

Are you an NVIDIA fanboy?

This is a _remarkably_ aggressive comment!

Re: Claude Code: connect to a local model when your quota runs out

#66

Post by redwood_ »

CamperBob2 wrote: post #62
Gravey wrote: post #58

Earlier quoted context omitted.

Would you mind sharing your hardware setup and use case(s)?

Not the GP but the new Qwen-Coder-Next release feels like a step change, at 60 tokens per second on a single 96GB Blackwell. And that's at full 8-bit quantization and 256K context, which I wasn't sure was going to work at all. It is probably enough to handle a lot of what people use the big-3 closed models for. Somewhat slower and somewhat dumber, granted, but still extraordinarily capable. It punches way above its w…

Agree, these new models are a game changer. I switched from Claude to Qwen3-Coder-Next for day-to-day on dev projects and don't see a big difference. Just use Claude when I need comprehensive planning or review. Running Qwen3-Coder-Next-Q8 with 256K context.

Re: Claude Code: connect to a local model when your quota runs out

#67

Post by bcyn »

eek2121 wrote: post #16

I gotta say, the local models are catching up quick. Claude is definitely still ahead, but things are moving right along.

Which models perform anywhere close to Opus 4.5? In my experience none of the local models are even in the same ballpark.

Re: Claude Code: connect to a local model when your quota runs out

#68

Post by teaearlgraycold »

cactusplant7374 wrote: post #43
SchemaLoad wrote: post #38

Earlier quoted context omitted.

It's still very expensive compared to using the hosted models which are currently massively subsidised. Have to wonder what the fair market price for these hosted models will be after the free money dries up.

Inference is profitable. Maybe we hit a limit and we don't need as many expensive training runs in the future.

For sure Claude Code isn’t profitable

Re: Claude Code: connect to a local model when your quota runs out

#70

Post by DANmode »

paxys wrote: post #22

> Reduce your expectations about speed and performance! Wildly understating this part. Even the best local models (ones you run on beefy 128GB+ RAM machines) get nowhere close to the sheer intelligence of Claude/Gemini/Codex. At worst these models will move you backwards and just increase the amount of work Claude has to do when your limits reset.

and you really should be measuring based on the worst-case scenario for tools like this.
Post reply on HN