Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 196 posts

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#181

Post by fm2606 »

gkfasdfasdf wrote: post #175
fm2606 wrote: post #57

Earlier quoted context omitted.

gpt-oss-120b is amazing. I created a RAG agent to hold most of GCP documentation (separate download, parsing, chunking, etc). ChatGPT finished a 50 question quiz in 6 min with a score of 46 / 50. gpt-oss-120b took over an hour but got 47 / 50. All the other local LLMs I tried were small and performed way worse, like less than 50% correct. I ran this on an i7 with 64gb of RAM and an old nvidia card with 8g of vram. ED…

What were you using for RAG? Did you build your own or some off the shelf solution (e.g. openwebui)

I used pg vector chunking on paragraphs. For the answers I saved in a flat text file and then parsed to what I needed.

For parsing and vectorizing of the GCP docs I used a Python script. For reading each quiz question, getting a text embedding and submitting to an LLM, I used Spring AI.

It was all roll your own.

But like I stated in my original post I deleted it without backup or vcs. It was the wrong directory that I deleted. Rookie mistake for which I know better.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#183

Post by scosman »

realityfactchex wrote: post #68
scosman wrote: post #46

What are folks motivation for using local coding models? Is it privacy and there's no cloud host you trust? I love local models for some use cases. However for coding there is a big gap between the quality of models you can run at home and those you can't (at least on hardware I can afford) like GLM 4.6, Sonnet 4.5, Codex 5, Qwen Coder 408. What makes local coding models compelling?

> compelling >> motivation It's the only way to be sure it's not being trained on. Most people never come up with any truly novel ideas to code. That's fine. There's no point in those people not submitting their projects to LLM providers. This lack of creativity is so prevalent, that many people believe that it is not possible to come up with new ideas (variants: it's all been tried before; or: it would inevitably be…

There are guarantees from several providers that they don’t train on, or even retain, a copy of your data. You are right they could be lying, but some are big enough that would be catastrophic to them from a liability point of view.

Re:creative competition - that’s interesting. I open source much of my creative work so I guess that’s never been a concern of mine.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#184

Post by zamadatix »

hacker_homie wrote: post #96
zamadatix wrote: post #33

Earlier quoted context omitted.

Do you run into hibernation/sleep issues under current mainline Linux kernels by chance? I have this laptop and that's the only thing which isn't working out of the box for me on the Linux side, but it works fine in Windows. I know it's officially supported under the Ubuntu LTS, but I was hoping that wouldn't be needed as I do want a newer+customized kernel.

Under current kernels (6.17) it seems there is an issue with the webcam driver, https://bugzilla.kernel.org/show_bug.cgi?id=220702 . looks like there are still some issues with sleep/webcam at this time, they might be fixed by the 6.18 release. I got sleep working by disabling webcam in the bios for now.

Well shucks, my sleep was still broken after disabling that :/. Will have to keep poking at it - thanks!

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#186

Post by windexh8er »

embedding-shape wrote: post #174
windexh8er wrote: post #173

Earlier quoted context omitted.

It's a terminal only (TUI) tool today. They're releasing a graphical (GUI) version in the future.

> It's a terminal only (TUI) tool today. But to use that TUI you need a desktop, or at least a laptop I guess, but that distinction doesn't make sense. Are they referring to the GUI being the "Desktop Version"? Never heard it put that way before if so.

> But to use that TUI you need a desktop...

No, you don't need a "desktop" to use a TUI. It's terminal based and has nothing to do with the desktop environment you're in.

Alao, if you have a "desktop" that assumes you're using a GUI. Pretty straightforward.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#187

Post by ponyous »

windexh8er wrote: post #155
gcr wrote: post #109

For new folks, you can get a local code agent running on your Mac like this: 1. $ npm install -g @openai/codex 2. $ brew install ollama; ollama serve 3. $ ollama pull gpt-oss:20b 4. $ codex --oss -m gpt-oss:20b This runs locally without Internet. Idk if there’s telemetry for codex, but you should be able to turn that off if so. You need an M1 Mac or better with at least 24GB of GPU memory. The model is pretty big, ab…

I've been really impressed by OpenCode [0]. The limitations of all the frontier TUI is removed and it is feature complete and performant compared to Codex or Claude Code. [0] https://opencode.ai/

What kind of API subscription are you using? I found opencode to be incredibly expensive - prompts costing $5, while with aider I did it for <$0.1.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#188

Post by knownjorbist »

egberts1 wrote: post #112

Ollama, 16-CPU Xenon E6320 (old), 1.9Ghz, 120GB DDRAM4, 240TB RAID5 SSDs, on Dell Precision T710 ("The Beast"). NO GPU. 20b (n oooooot f aah st at all). Pure CPU bound. Tweaked for 256KB chunking into RAG. Ingested election laws of 50 states, territories and Federal. Goal. Mapping out each feature of the election and deal with (in)consistent terminologies sprouted by different university-trained public administration…

So much intelligence devoted to what is obviously a huge con - the Big Lie?

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#189

Post by gunalx »

embedding-shape wrote: post #167
gunalx wrote: post #133

Earlier quoted context omitted.

I have in many cases had better results with the 20b model, over the 120b model. Mostly because it is faster and I can iterate prompts quicker to choerce it to follow instructions.

> had better results with the 20b model, over the 120b model The difference of quality and accuracy of the responses between the two is vastly different though, if tok/s isn't your biggest priority, especially when using reasoning_effort "high". 20B works great for small-ish text summarization and title generation, but for even moderately difficult programming tasks, 20B fails repeatedly while 120B gets it right on t…

But the 120b model has just as bad if not worse formatting issues, compared to the 20b one. For simple refactorings, or chatting about possible solutions i actually feel teh 20b halucinates less than the 120b, even if it is less competent. Migth also be because of 120b not liking being in q8, or not being properly deployed.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#190

Post by fultonn »

firefax wrote: post #169
fultonn wrote: post #103

Earlier quoted context omitted.

> Is there a way to load this into Ollama? Yes, the granite 4 models are on ollama: https://ollama.com/library/granite4 > but my interest is specifically in privacy respecting LLMs -- my goal is to run the most powerful one I can on my personal machine The HF Spaces demo for granite 4 nano does run on your local machine, using Transformers.js and ONNX. After downloading the model weights you can disconnect from the i…

Thanks for this pointer and explanation, I appreciate it. Also my "dev enviornment" is vi -- I come from infosec (so basically a glorified sysadmin) so I'm mostly making little bash and python scripts, so I'm learning a lot of new things about software engineering as I explore this space :-) Edit: Hey which of the models on that page were you referring to? I'm grabbing one now that's apparently double digit GB? Or we…

> Edit: Hey which of the models on that page were you referring to?

I was referring to the smaller ones -- `granite4:micro`, `granite4-latest`, `granite4:350m`.

> I'm grabbing one now that's apparently double digit GB?

You are probably downloading one of these two ids: `granite4:small-h` or `granite4:32b-a9b-h`.

The "small" model _is_ small in relative terms, but is also the largest of the currently released granite models! At 32B parameters (19GB download) it's runnable locally but not in the same "run on your laptop with acceptable performance" category of the nano/micro models.

> Also my "dev enviornment" is vi -- I come from infosec (so basically a glorified sysadmin) so I'm mostly making little bash and python scripts, so I'm learning a lot of new things about software engineering as I explore this space :-)

Shameless plug: if you're writing Python scripts to automate things using small locally hosted models, consider trying out https://github.com/generative-computing/mellea

Mellea tries to nudge toward good software engineering practices -- breaking down big tasks into smaller parts, checking outputs after nondeterministic steps, thinking in terms of data structures and invariants rather than flow charts, etc. We built it with "actual fully automated robust workflows" in mind. You can use it with big models or small models, but it really shines when used with small models.

Post reply on HN