I wonder how long it will be before 400B+ local models achieving 2K TPS become a reality for under $10K
Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
141–150 of 196 posts
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#142I sometimes still code with a local LLM but can't imagine doing it on a laptop. I have a server that has GPUs and runs llama.cpp behind llama-swap (letting me switch between models quickly). The best local coding setup I've been able to do so far is using Aider with gpt-oss-120b. I guess you could get a Ryzen AI Max+ with 128GB RAM to try and do that locally but non-nVidia hardware is incredibly slow for coding usage…
gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…
> I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc)
If you share the scripts to gather the GCP documentation this, that'd be great. Because I have had an idea to do something like this, and the part I don't want to deal with is getting the data
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#143I use local for code completions only. Which means models supporting FIM tokens. My current setup is the llama-vscode plugin + llama-server running Qwen/Qwen2.5-Coder-7B-Instruct. It leads to very fast completions, and don't have to worry about internet outages which take me out of the zone. I do wish qwen-3 released a 7B model supporting FIM tokens. 7B seems to be the sweet spot for fast and usable completions
qwen3-coder-30B-A3B supports FIM and should be faster than the 7B if you got the vram.
I use bartowkski’s Q8 quant over dual 3090s and it gets up to 100tok/sec. The Q4 quant on a single 3090 is very fast and decently smart.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#144Hardware: MacBook Pro M4 Max, 128GB Platform: LMStudio (primarily) & Ollama Models: - qwen/qwen3-coder-30b A3B Instruct 8-bit MLX - mlx-community/gpt-oss-120b-MXFP4-Q8 For code generation especially for larger projects, these models aren't as good as the cutting edge foundation models. For summarizing local git repos/libraries, generating documentation and simple offline command-line tool-use they do a good job. I fi…
I have a Mac Studio with the M4 Max and 128GB RAM
The Qwen3-coder model you use is pretty good. You can enable the LM Studio API and install the qwen CLI and point to the API endpoint. This basically gives you functionality similar to Claude code.
I agree that the code quality is not on part with gpt5-codex and Claude. I also haven't tried z.ai's models locally yet. I think on a Mac with that size GLM 4.5 Air should be able to run.
For README generation I like gemma3-27b-it-qat and gpt-oss-120b.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#145Laptop: Apple M2 Max, 32GB memory (2023)
Setup:
Terminal:
- Ghostty + Starship for modern terminal experience
- Homebrew to install system packages
IDE:
- Zed (can connect to local models via LM-Studio server)
- also experimenting with warp.dev
LLMs:
- LM-studio as open-source model playground
- GPT-OSS 20B
- QWEN3-Coder-30B-AEB-quantized-4bit
- Gemma3-12B
Other utilities:
- Rectangle.app (window tile manager)
- Wispr.flow - create voice notes
- Obsidian - track markdown notes
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#146Earlier quoted context omitted.
The catch is ollama cloud is likely to increase prices and/or decrease usage limit levels soon. Free tier has more restrictions than their $20/mo tier. They claim to not store anything ( https://ollama.com/cloud ) but you'll have to clarify what you mean by "private" (your model likely runs on shared hardware with other users).
I agree. "Free" usage could mean tradeoff. But for side-project and experiments, to accesss open source model like gpt-oss, as my machine can not run, I think I will accept it.
My experience with the free tier and qwen3-coder cloud is the hourly limit gets you about 250k tokens input and then your usage is paused till the hour is up. Enough to try something very small.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#147Earlier quoted context omitted.
I don't ever want to be dependent on a cloud service to be productive, and I don't want to have to pay money to experiment with code. Paying money for probabilistically generated tokens is effectively gambling. I don't like to gamble.
Where did you get your free GPU from?
GPUs can do other things.
Cloud service LLM providers cannot.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#148If you're going to get a MacBook, get the Pro, it has a built-in fan, you don't want the heat just sitting there on the MacBook Air. Same with the Mac mini, get the studio instead, it has a fan, the Mini does not. I don't know about you but I wouldn't want my brand new laptop / desktop to be heating up the entire time I'm coding with 0 cool off. If you go the Mac route, I recommend getting TG Pro, the default fan set…
> I do think Macs are phenomenal at running local LLMs if you get the right one. How does the prompt processing speed look like today? I think it was either M3 or M4 together with 128GB, trying to run even slightly longer prompts took forever for the initial prompt processing so whatever speed gain you get at inference, basically didn't matter. Maybe it works better today?
I have only ever used the M4 (on my wife's Macbook Air) and M4 Pro (on my Macbook Pro) and it was reasonable speeds, I was able to tie LM Studio with PyCharm and ask it questions about code, but my context Window kept running out, I don't think the 24GB model is the right choice, the key thing you have to also look out for is for example I might hvae 24GB of RAM, but only 16 of it can be used as VRAM, so I'm more competitive than my 3080 in terms of VRAM, though my 3080 could probably run circles around my M4 Pro if it wanted to.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#149For new folks, you can get a local code agent running on your Mac like this: 1. $ npm install -g @openai/codex 2. $ brew install ollama; ollama serve 3. $ ollama pull gpt-oss:20b 4. $ codex --oss -m gpt-oss:20b This runs locally without Internet. Idk if there’s telemetry for codex, but you should be able to turn that off if so. You need an M1 Mac or better with at least 24GB of GPU memory. The model is pretty big, ab…
LM Studio is even easier, and things like JetBrains IDEs will sync to LM Studio, same with Zed.
Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
#150qwen3 coder 30B with ollama server via continue.
Big box Win11 Ryzen9-7900,128GB-DDR5,8TB,5090.
Use for all of the above. Works pretty well for simple coding tasks.
Haven't given it much in the way of smoke tests yet.
It's quite snappy, but of course the limitation is context size if I run it solely on GPU.
Haven't plugged OSS into the pipeline yet.
Wanted to use code specific model.