Ask HN: How are you doing RAG locally? • Hacker News Board

Re: Ask HN: How are you doing RAG locally?

#111

Post by canadiantim » Thu, Jan 15, 2026, 3:10 PM UTC

jankovicsandras wrote: post #34

CuriouslyC wrote: post #12

Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.

You can do hybrid search in Postgres. Shameless plug: https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain ) The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.

Wow very impressive library great work!

Re: Ask HN: How are you doing RAG locally?

#112

Post by miohtama » Thu, Jan 15, 2026, 3:11 PM UTC

autogn0me wrote: post #32

https://github.com/ggozad/haiku.rag/ - the embedded lancedb is convenient and has benchmarks; uses docling. qwen3-embedding:4b, 2560 w/ gpt-oss:20b.

+1 for Haiku! It's very simple to get up and running.

Re: Ask HN: How are you doing RAG locally?

#113

Post by cipherself » Thu, Jan 15, 2026, 3:31 PM UTC

rao-v wrote: post #30

CuriouslyC wrote: post #12

Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.

Anybody know of a good service / docker that will do BM25 + vector lookup without spinning up half a dozen microservices?

Here's a Dockerfile that will spin up postgres with pgvector and paradedb https://gist.github.com/cipherself/5260fea1e2631e9630081fb7d...

You can use pgvector for the vector lookup and paradedb for bm25.

Re: Ask HN: How are you doing RAG locally?

#114

Post by mooball » Thu, Jan 15, 2026, 3:32 PM UTC

i thought rag/embeddings were dead with the large context windows. thats what i get for listening to chatgpt.

Re: Ask HN: How are you doing RAG locally?

#115

Post by amscotti » Thu, Jan 15, 2026, 3:33 PM UTC

More of a proof of concept to test out ideas, but here's my approach for local RAG, https://github.com/amscotti/local-LLM-with-RAG

Using Ollama for the embeddings with “nomic-embed-text”, with LanceDB for the vector database. Recently updated it to use “agentic” RAG, but probably not fully needed for a small project.

Re: Ask HN: How are you doing RAG locally?

#116

Post by __mharrison__ » Thu, Jan 15, 2026, 3:35 PM UTC

Grep (rg)

Re: Ask HN: How are you doing RAG locally?

#117

Post by whacked_new » Thu, Jan 15, 2026, 3:41 PM UTC

Aachen wrote: post #84

beklein wrote: post #64

Most of my complex documents are, luckily, Markdown files. I can recommend https://github.com/tobi/qmd/ . It’s a simple CLI tool for searching in these kinds of files. My previous workflow was based on fzf, but this tool gives better results and enables even more fuzzy queries. I don’t use it for code, though.

Given that preface, I was really expecting that link to be a grepping tool rewritten in golang or something, or perhaps customised for markdown to weigh matches in "# heading title"s heavier for example

Here's a rust one: https://github.com/BeaconBay/ck

I haven't used it extensively, but semantic grep alone was kind of worth it.

Re: Ask HN: How are you doing RAG locally?

#118

Post by WilcoKruijer » Thu, Jan 15, 2026, 3:42 PM UTC

aqula wrote: post #51

spqw wrote: post #40

I am surprised to see very few setups leveraging LSP support. (Language Server Protocol) It has been added to Claude Code last month. Most setups rely on naive grep.

LSP is not great for non-editor use cases. Everything is cursor position oriented.

There are actions that don't require cursor position, like document/workspace symbols, that could be useful.

Re: Ask HN: How are you doing RAG locally?

#119

Post by yandrypozo » Thu, Jan 15, 2026, 3:53 PM UTC

SamLeBarbare wrote: post #61

sqlite + FTS + sqlite-vec + local LLM for reranking results (reasoning model)

this's pretty cool, which LLM are you using currently?

Re: Ask HN: How are you doing RAG locally?

#120

Post by turnsout » Thu, Jan 15, 2026, 3:54 PM UTC

The Claude Code model highlights the power of simple search (grep) and selective reads (only reading in excerpts). The only time I vectorize is when I explicitly want to similarity-based searching, but that's actually pretty rare.