Ask HN: How are you doing RAG locally?
31–40 of 166 posts
Re: Ask HN: How are you doing RAG locally?
#32https://github.com/ggozad/haiku.rag/ - the embedded lancedb is convenient and has benchmarks; uses docling. qwen3-embedding:4b, 2560 w/ gpt-oss:20b.
Re: Ask HN: How are you doing RAG locally?
#33If your data aren't too large, you can use faiss-cpu and pickle https://pypi.org/project/faiss-cpu/
Shoud it be:
If the total size of your data isn't loo large...?
Data being a plural gets me.
You might have small datums but a lot of kilobytes!
Re: Ask HN: How are you doing RAG locally?
#34Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
You can do hybrid search in Postgres.
Shameless plug: https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain )
The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.
Re: Ask HN: How are you doing RAG locally?
#35[flagged]
A new account, named after the thinking you're linking just looks like spam.
Also I've got no idea what this product does, this is just a generic page of topical ai buzzwords
Don't tell me what it is, /show me why/ you built it. Then go back and keep that reasoning in, show me why I should care
Re: Ask HN: How are you doing RAG locally?
#36I thought that context building via tooling was shown to be more effective than rag in practically every way?
Question being: WHY would I be doing RAG locally?
Re: Ask HN: How are you doing RAG locally?
#37For the purposes of learning, I’ve built a chatbot using ollama, streamlit, chromadb and docling. Mostly playing around with embedding and chunking on a document library.
Re: Ask HN: How are you doing RAG locally?
#38I just use a web server and a search engine.
TL;DR: - chunk files, index chunks - vector/hybrid search over the index - node app to handle requests (was the quickest to implement, LLMs understand OpenAPI well)
I wrote about it here: https://laurentcazanove.com/blog/obsidian-rag-api
Re: Ask HN: How are you doing RAG locally?
#39I'm lucky enough to have 95% of my docs in small markdown markdown files so I'm just... not (+). I'm using SQLite FTS5 (full text search) to build a normal search index and using that. Well, I already had the index so I just wired it up to my mastra agents.
Each file has a short description field, so if a keyword search surfaces the doc they check the description and if it matches, load the whole doc.
This took about one hour to set up and works very well.
(+) At least, I don't think this counts as RAG. I'm honestly a bit hazy on the definition. But there's no vectordb anyway.
Re: Ask HN: How are you doing RAG locally?
#40I am surprised to see very few setups leveraging LSP support. (Language Server Protocol)
It has been added to Claude Code last month.
Most setups rely on naive grep.