Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 196 posts

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#111

Post by oofbey »

jwpapi wrote: post #14

On a side note I really thing latency is still important. Is there some benefit in choosing location for where you get your responses from? Like with Openrouter f.e. Also I could think that a local model just for autocomplete could help reducing latency for completion suggestions.

Latency matters for the autocomplete models. But IMHO those suck and generally just get in the way.

For the big agentic tasks or reasoned questions, the many seconds or even minutes of LLM time dwarf RTT even to another continent.

Side note: I recently had GPT5 in Cursor spend fully 45 minutes on one prompt chewing on why a bug was flaky, and it figured it out! Your laptop is not gonna do that anytime soon.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#112

Post by egberts1 »

Ollama, 16-CPU Xenon E6320 (old), 1.9Ghz, 120GB DDRAM4, 240TB RAID5 SSDs, on Dell Precision T710 ("The Beast"). NO GPU. 20b (n oooooot f aah st at all). Pure CPU bound. Tweaked for 256KB chunking into RAG.

Ingested election laws of 50 states, territories and Federal.

Goal. Mapping out each feature of the election and deal with (in)consistent terminologies sprouted by different university-trained public administration. This is the crux of hallunications: getting a diagram of ballot handling and their terminologies.

Then maybe tackle the multitude ways of election irregularities, or at least point out integrity gaps at various locales.

https://figshare.com/articles/presentation/Election_Frauds_v...

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#113

Post by whitehexagon »

Qwen3:32b on MBP M1 Pro 32GB running Asahi linux. Mainly command line for some help with armv8 assembly, and some SoC stuff (this week explaining I2C protocol). I couldnt find any good intro on the web-of-ads. It's not much help with Zig, but then nothing seems to keep up with Zig at the moment.

I get a steady stream of tokens, slightly slower than my reading pace, which I find is more than fast enough. In fact I´d only replace with exact same, or maybe M2 + Asahi with enough RAM to run the bigger Qwen3 model.

I saw qwen3-coder mentioned here. I didnt know about that one. Anyone got any thoughts on how that compares to qwen3? Will it also fit in 32GB?

I'm not interested in agents, or tool integration, and especially wont use anything cloud. I like to own my env. and code top-to-bottom. Having also switched to Kate and Fossil it feels like my perfect dev environment.

Currently using an older Ollama, but will switch to llama.cpp now that ollama has pivoted away from offline only. I got llama.cpp installed, but not sure how to reuse my models from ollama, I thought ollama was just a wrapper, but they seems to be different model formats?

[edit] be sure to use it powered, linux is a bit battery heavy, but Qwen3 will pull 60W+ and flatten a battery real fast.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#114

Post by nickthegreek »

gcr wrote: post #109

For new folks, you can get a local code agent running on your Mac like this: 1. $ npm install -g @openai/codex 2. $ brew install ollama; ollama serve 3. $ ollama pull gpt-oss:20b 4. $ codex --oss -m gpt-oss:20b This runs locally without Internet. Idk if there’s telemetry for codex, but you should be able to turn that off if so. You need an M1 Mac or better with at least 24GB of GPU memory. The model is pretty big, ab…

have you been able to build or reiterate anything of value using just 20b to vibe code?

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#115

Post by mooiedingen »

Vim+ollama-vim Start new file with at the top in the comments the instructions needed to follow to become the solution to the problem and let it work like a sort of auto complete... example: # The Following is a python # Script that uses the # libraries requests and # BeautifullSoup to scrape # url_to_scrape = input( # "what url do i need to # fetch?") import ... """autocompletes from here the rest""" anywhere in a script one can # comment ' Instructions this way i find the most effective instead of asking Write me a script for this or that.. take a coding model and finetune it with commonly used snippets of code... This is completely customizable and will stay coherent to your own writing style.. i made embeddings per language, even md. python javascript vimscript lua php html json(however output is json) xml Css ...

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#116

Post by xemdetia »

scosman wrote: post #56
voakbasda wrote: post #51

Earlier quoted context omitted.

Zero trust in remote systems run by others with unknowable or questionable motives.

Makes sense that you'd run locally then. But really no host you trust to not keep data? Big tech with no-log guarantees and contractual liability? Companies with no-log guarantees and clear inference business model to protect like Together/Fireworks? Motives seem aligned. I'd run locally if I could without compromise. But the gap from GLM 4.5 Air to GLM 4.6 is huge for productivity.

This really isn't an all or nothing sort of situation. Many of the AI players have a proven record of simply not following existing norms. Until there is a consumer oriented player who is not presuming that training on my private data and ideas is permitted it only makes sense to do some stuff things locally. Beyond that many of the companies providing AI have either weird limits or limitations that interrupt me. I just know as an individual or a fledgling company I am simply not big enough to fight some of these players and win, and the compliance around companies running AI transparently is too new for me to rely on so the rules of engagement are all over the place. Also don't forget in a few years when the dust settles that company with that policy you like is highly likely to be consumed by a company who may not share the same ethics but your data is still held by them.

Why take a chance?

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#118

Post by platevoltage »

I've been using qwen2.5-coder for code assistant and code completion which has worked pretty well. I recently started trying mistral:7b-instruct. I use Continue with VS Code. It works ok. I'm limited to 16GB on an M2 MacBook Pro. I definitely wish I had more RAM to play with.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#119

Post by nicce »

jetsnoc wrote: post #81
CubsFan1060 wrote: post #60

Earlier quoted context omitted.

What is the Agent Organizer you use?

It’s a Claude agent prompt. I don’t recall who originally shared it, so I can’t yet attribute the source, but I’ll track that down shortly and add proper attribution here. Here’s the Claude agent markdown: https://github.com/lst97/claude-code-sub-agents/blob/main/ag... Edit: Updated from the old Pastebin link to the GitHub version. Attribution found: lst97 on GitHub

How it looks like Claude agent is written by Claude...

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#120

Post by nicce »

nprateem wrote: post #104
jckahn wrote: post #78

Earlier quoted context omitted.

I don't ever want to be dependent on a cloud service to be productive, and I don't want to have to pay money to experiment with code. Paying money for probabilistically generated tokens is effectively gambling. I don't like to gamble.

Where did you get your free GPU from?

The problem is the same as owning the house vs. renting.
Post reply on HN