Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

news.ycombinator.com

Post reply on HN
Print view View original

381–390 of 620 posts

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#381

Post by Rzor »

catapart wrote: post #258

tough ask, but since we're here: has anyone done this with 16GB of VRAM? I've been getting projects finished with LM Studio, but it definitely could stand to be more efficient. lots of time wasted with trying to get models to understand a problem with so few tokens.

RX 9060 XT 16GB here on google/gemma-4-26b-a4b-qat using LM Studio. Context 65k, 23 layers on the GPU, 7 on the CPU, model in memory, mmapped. I'm getting 23-33 tks. Started experimenting 3 days ago (with gemma-4-e4b), don't know what half those settings mean, but 26B, even quantified, feels significantly better at a few small projects I asked it to create ("create a image converter using ffmpeg in bash", "create a canvas animation with real physics, no libraries"[1]).

It's faster than I can read, but it feels slow as hell. I think 40-50 tks is probably much more comfortable and I hope I can reach that when trying this on llamacpp soon enough.

[0] - https://pastes.io/9gaARxE8

[1] - https://jsfiddle.net/pou4nbh9/1/

Model: https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-gg...

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#382

Post by verdverm »

chakspak wrote: post #173
lambda wrote: post #155

Earlier quoted context omitted.

This is very similar to my setup. Pi in a container (I do let it have network access, just no access to creds or anything, only the one directory that I'm working on at the time and my ~/.pi directory), talking to llama.cpp in another container. I'm on a Strix Halo 128 GiB unified memory laptop. I've never used the frontier models in earnest, I don't believe in using proprietary tools for my programming, so I can't r…

Hopefully this isn't off-topic, but your setup sounds just like mine, Strix Halo and (I'm assuming) llama.cpp on ROCm, and I'm finding that the Qwen hybrid models don't handle prompt caching and instead re-process the context in full on every turn. I'm wondering if you were able to solve this and how?

There is a bug in llama-cpp for qwen/gemma models, use vLLM instead

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#383

Post by lloyd-christmas »

amarshall wrote: post #270
heipei wrote: post #116

Earlier quoted context omitted.

Same here, I use Qwen 3.6 27b (Q6 quant) with llama.cpp on an RTX 5090 using the pi agent exclusively now. The fact that it's local means that I never have to think about token pricing, quotas, time of day, or data sensitivity. I have limited the GPU from 600W to 450W which means the system stays whisper quiet during inference. I have become so "lazy" (in a good way), so far that I've started using the model for lots…

What context length and kv cache quant (if any) are you using? And MTP?

Not the person you asked, but I have a 9700 which has the same VRAM, and running Q6 on it with unquantized kv gives me 50k context. Putting -ctv q8_0 ups that to 70k. I normally run Q4 with unquantized kv @ 130k at 50 t/s (mtp 3), with the disclaimer that I'm running PCIe gen4x8, so I'm slightly slowed. I've found that quantizing k leads to broken json on tool calls, which is fairly unrecoverable, but YMMV.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#385

Post by Rzor »

zaptheimpaler wrote: post #89

I tried gemma-4-26B-A4B just to see if it could help me read/sort my emails on a relatively under-powered setup (16GB VRAM + 32GB RAM) and it's not going well.. the model burns 24K tokens just on searching for the right tool and then dumps the email contents into context - i tried to get it to use code-mode to save context but the code-mode implementation can't save files so it was useless and im going to try to swit…

So there was a problem with gemma 4 when it comes to tool calling that Google apparently fixed like 2 or 3 days ago. I remember reading something about this.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#386

Post by CamperBob2 »

arjie wrote: post #347
mtone wrote: post #263

Earlier quoted context omitted.

Here's a DeepSeek-V4-Flash benchmark on 2X RTX Pro 6000: - Prefill: ~10K tok/s - Decode: 190 | 375 | 980 tok/s (for 1 | 4 | 16 concurrent requests) - GPU power draw during benchmark: Average: 585W | Max: 849W | Limit: 1200W with undervolt. Idle PC is 125W. I've asked it to calculate the following considering a realistic blend of cached prompts and decode for agentic dev scenario. Electricity-only (@ USD $0.08/kWh) Us…

Vouched your comment. Very cool. What are you running on to get 190 tok/s? I get 400 tok/s at c=4 but c=1 is slower than you.

Not OP, but I am seeing up to 260 tokens/second output at c=1 with the recipe at https://github.com/local-inference-lab/rtx6kpro/blob/master/... using 4x 6k cards. Average is more like 200.

There may be a way to get the 2-bit quantized version running even faster on a pair of them.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#387

Post by anubhavgupta »

anubhav200 wrote: post #57

Yes, llama.cpp, qwen27b, 35b, claude code. Llama-cpp-manager for managing llama.cpp configs ( https://github.com/anubhavgupta/llama-cpp-manager )

Machine: CPU: intel 275hx GPU: Nvidia 5090 Mobile (24GB) RAM: 64GB

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#388

Post by anubhavgupta »

anubhavgupta wrote: post #387
anubhav200 wrote: post #57

Yes, llama.cpp, qwen27b, 35b, claude code. Llama-cpp-manager for managing llama.cpp configs ( https://github.com/anubhavgupta/llama-cpp-manager )

Machine: CPU: intel 275hx GPU: Nvidia 5090 Mobile (24GB) RAM: 64GB

One more thing, I also use it along with Whisper-NPU, a speech to text utility that runs on NPU of Intel 275hx and doesn't consumes any GPU resources.

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#389

Post by anubhavgupta »

anubhavgupta wrote: post #388
anubhavgupta wrote: post #387

Earlier quoted context omitted.

Machine: CPU: intel 275hx GPU: Nvidia 5090 Mobile (24GB) RAM: 64GB

One more thing, I also use it along with Whisper-NPU, a speech to text utility that runs on NPU of Intel 275hx and doesn't consumes any GPU resources.

Whisper-NPU (https://github.com/anubhavgupta/whisper-npu)

Re: Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?

#390

Post by mtone »

arjie wrote: post #347
mtone wrote: post #263

Earlier quoted context omitted.

Here's a DeepSeek-V4-Flash benchmark on 2X RTX Pro 6000: - Prefill: ~10K tok/s - Decode: 190 | 375 | 980 tok/s (for 1 | 4 | 16 concurrent requests) - GPU power draw during benchmark: Average: 585W | Max: 849W | Limit: 1200W with undervolt. Idle PC is 125W. I've asked it to calculate the following considering a realistic blend of cached prompts and decode for agentic dev scenario. Electricity-only (@ USD $0.08/kWh) Us…

Vouched your comment. Very cool. What are you running on to get 190 tok/s? I get 400 tok/s at c=4 but c=1 is slower than you.

I am using the `voipmonitor/vllm:lucifer` docker from the RTX6K discord community discussed at the same link the other commenter posted. It is based around this PR https://github.com/vllm-project/vllm/pull/43477
Post reply on HN