101–110 of 196 posts
What are folks motivation for using local coding models? Is it privacy and there's no cloud host you trust? I love local models for some use cases. However for coding there is a big gap between the quality of models you can run at home and those you can't (at least on hardware I can afford) like GLM 4.6, Sonnet 4.5, Codex 5, Qwen Coder 408. What makes local coding models compelling?
On a Macbook pro 64GB I use Qwen3-Coder-30B-A3B Q4 quant with llama.cpp. For VSCode I use continue.dev as it allows to set my own (short) system prompt. I get around 50token/sec generation and prompt processing 550t/s. When giving well defined small tasks, it is as good as any frontier model. I like the speed and low latency and the availability while on the plane/train or off-grid. Also decent FIM with the llama.cpp…
Would you use a different quant with a 128 GB machine? Could you link the specific download you used on huggingface? I find a lot of the options there to be confusing.
On 128GB I would definitely run a larger model, probably with ~10B active parameters. All depends how many tokens per second is comfortable for you.
To get an idea of the speed difference, there is a benchmark page for llama.cpp on Apple silicon here: https://github.com/ggml-org/llama.cpp/discussions/4167
About quant selection: https://gist.github.com/Artefact2/b5f810600771265fc1e3944228...
And my workaround for 'shortening' prompt processing time: I load the files I want to work on (usually 1-3) into context with the instruction: read the code and wait. And while the LLM is doing the prompt processing I write my instructions of what I want to have done. Usually the LLM is long finished with PP before I am finished with writing instructions. Due to KV caching the LLM then gives almost instantly the answer.
Earlier quoted context omitted.
I was playing with the new IBM Granite models. They are quick/small and they do seem accurate. You can even try them online in the browser because they are small enough to be loaded via the filesystem: https://huggingface.co/spaces/ibm-granite/Granite-4.0-Nano-W... Not only are they a lot more recent than gemma, they seem really good at tool calling, so probably good for coding tools. I haven’t personally tried it my…
Interesting. Is there a way to load this into Ollama? Doing things in browser is a cool flex, but my interest is specifically in privacy respecting LLMs -- my goal is to run the most powerful one I can on my personal machine, with the end goal being those little queries I used to send to "the cloud" can be done offline, privately.
Yes, the granite 4 models are on ollama:
https://ollama.com/library/granite4
> but my interest is specifically in privacy respecting LLMs -- my goal is to run the most powerful one I can on my personal machine
The HF Spaces demo for granite 4 nano does run on your local machine, using Transformers.js and ONNX. After downloading the model weights you can disconnect from the internet and things should still work. It's all happening in your browser, locally.
Of course ollama is preferable for your own dev environment. But ONNX and transformers.js is amazingly useful for edge deployment and easily sharing things with non-technical users. When I want to bundle up a little demo for something I typically just do that instead of the old way I did things (bundle it all up on a server and eat the inference cost).
What are folks motivation for using local coding models? Is it privacy and there's no cloud host you trust? I love local models for some use cases. However for coding there is a big gap between the quality of models you can run at home and those you can't (at least on hardware I can afford) like GLM 4.6, Sonnet 4.5, Codex 5, Qwen Coder 408. What makes local coding models compelling?
I don't ever want to be dependent on a cloud service to be productive, and I don't want to have to pay money to experiment with code. Paying money for probabilistically generated tokens is effectively gambling. I don't like to gamble.
On a Macbook pro 64GB I use Qwen3-Coder-30B-A3B Q4 quant with llama.cpp. For VSCode I use continue.dev as it allows to set my own (short) system prompt. I get around 50token/sec generation and prompt processing 550t/s. When giving well defined small tasks, it is as good as any frontier model. I like the speed and low latency and the availability while on the plane/train or off-grid. Also decent FIM with the llama.cpp…
There is an open issue about adding support for Qwn3 which I have been monitoring, would love to use Qwen3 if possible. Issue - https://github.com/ggml-org/llama.vscode/issues/55
Tangential question. What do people use for search? What search engines provide the best quality to cost ratios? Also are there good solutions for searching through a local collection of documents?
ddg (python lib) is free and I'd say good enough for most tasks. (I think the endpoint is unofficial, but from what I've heard it's fine for typical usage.) There's also google, which gives you 100 requests a day or something. Here's the search.py I use import os import json from req import get # https://programmablesearchengine.google.com/controlpanel/create GOOGLE_SEARCH_API_KEY = os.getenv('GOOGLE_SEARCH_API_KEY')…
Hardware: MacBook Pro M4 Max, 128GB Platform: LMStudio (primarily) & Ollama Models: - qwen/qwen3-coder-30b A3B Instruct 8-bit MLX - mlx-community/gpt-oss-120b-MXFP4-Q8 For code generation especially for larger projects, these models aren't as good as the cutting edge foundation models. For summarizing local git repos/libraries, generating documentation and simple offline command-line tool-use they do a good job. I fi…
I have recently added claude skills to it. So, all the claude skills can be executed locally on your mac too.
Earlier quoted context omitted.
I don't ever want to be dependent on a cloud service to be productive, and I don't want to have to pay money to experiment with code. Paying money for probabilistically generated tokens is effectively gambling. I don't like to gamble.
Where did you get your free GPU from?
1. $ npm install -g @openai/codex
2. $ brew install ollama; ollama serve
3. $ ollama pull gpt-oss:20b
4. $ codex --oss -m gpt-oss:20b
This runs locally without Internet. Idk if there’s telemetry for codex, but you should be able to turn that off if so.
You need an M1 Mac or better with at least 24GB of GPU memory. The model is pretty big, about 16GB of disk space in ~/.ollama
Be careful - the 120b model is 1.5× better than this 20b variant, but takes 5× higher requirements.
For anyone who wants to see some real workstations that do this, you may want to check out Alex Ziskind's channel on YouTube: https://www.youtube.com/@AZisk At this point, pretty much all he does is review workstations for running LLM's and other machine-learning adjacent tasks. I'm not his target demographic, but because I'm a dev, his videos are constantly recommended to me on YouTube. He's a good presenter and his…