Ask HN: How are you doing RAG locally? • Hacker News Board

Re: Ask HN: How are you doing RAG locally?

#41

Post by geuis » Thu, Jan 15, 2026, 7:59 AM UTC

I don't. I actually write code.

To answer the question more directly, I've spent the last couple of years with a few different quant models mostly running on llama.cpp and ollama, depending. The results are way slower than the paid token api versions, but they are completely free of external influence and cost.

However the models I've tests generally turn out to be pretty dumb at the quant level I'm running to be relatively fast. And their code generation capabilities are just a mess not to be dealt with.

Re: Ask HN: How are you doing RAG locally?

#42

Post by petesergeant » Thu, Jan 15, 2026, 8:10 AM UTC

baalimago wrote: post #36

I thought that context building via tooling was shown to be more effective than rag in practically every way? Question being: WHY would I be doing RAG locally?

For code, maybe? For documents, no, text embeddings are magical alien technology.

Re: Ask HN: How are you doing RAG locally?

#43

Post by petesergeant » Thu, Jan 15, 2026, 8:11 AM UTC

jeanloolz wrote: post #19

Sqlite-vec

I’ve got it deployed in production for a dataset that changes infrequently and it works really well

Re: Ask HN: How are you doing RAG locally?

#44

Post by softwaredoug » Thu, Jan 15, 2026, 8:18 AM UTC

I built a Pandas extension SearchArray, I just use that (plus in memory embeddings) for any toy thing

https://github.com/softwaredoug/searcharray

Re: Ask HN: How are you doing RAG locally?

#45

Post by Bombthecat » Thu, Jan 15, 2026, 8:21 AM UTC

AnythingLLM for documents, amazing tool!

Re: Ask HN: How are you doing RAG locally?

#46

Post by sidrag22 » Thu, Jan 15, 2026, 8:31 AM UTC

beret4breakfast wrote: post #37

For the purposes of learning, I’ve built a chatbot using ollama, streamlit, chromadb and docling. Mostly playing around with embedding and chunking on a document library.

i took a similar path, i spun up a discord bot, used ollama, pgvector, docling for random documents, and made some specialized chunking strategies for some clunkier json data. its been a little while since i messed with it, but i really did enjoy it when i was.

it all moves so fast, i wouldnt be surprised if everything i made is now crazy outdated and it was probably like 2 months ago.

Re: Ask HN: How are you doing RAG locally?

#47

Post by m00dy » Thu, Jan 15, 2026, 8:31 AM UTC

tebeka wrote: post #31

https://duckdb.org/2024/05/03/vector-similarity-search-vss

does duckdb scale well over large datasets for vector search ?

Re: Ask HN: How are you doing RAG locally?

#48

Post by woggy » Thu, Jan 15, 2026, 8:48 AM UTC

spqw wrote: post #40

I am surprised to see very few setups leveraging LSP support. (Language Server Protocol) It has been added to Claude Code last month. Most setups rely on naive grep.

I've written a few terminal tools on top of Roslyn to assist Claude in code analysis for C# code. Obviously the tools are also written with the help of Claude. Worked quite well.

Re: Ask HN: How are you doing RAG locally?

#49

Post by dmos62 » Thu, Jan 15, 2026, 9:02 AM UTC

esperent wrote: post #39

I'm lucky enough to have 95% of my docs in small markdown markdown files so I'm just... not (+) . I'm using SQLite FTS5 (full text search) to build a normal search index and using that. Well, I already had the index so I just wired it up to my mastra agents. Each file has a short description field, so if a keyword search surfaces the doc they check the description and if it matches, load the whole doc. This took abou…

Retrieval-augmented generation. What you described is a perfect example of a RAG. An embedding-based search might be more common, but that's a detail.

Re: Ask HN: How are you doing RAG locally?

#50

Post by lsb » Thu, Jan 15, 2026, 9:15 AM UTC

I'm using Sonnet with 1M Context Window at work, just stuffing everything in a window (it works fine for now), and I'm hoping to investigate Recursive Language Models with DSPy when I'm using local models with Ollama