If your data aren't too large, you can use faiss-cpu and pickle https://pypi.org/project/faiss-cpu/
For the uneducated, how large is too large? Curious.
FAISS runs in RAM. If your dataset can't fit into ram, FAISS is not the right tool.
Ask HN: How are you doing RAG locally?
21–30 of 166 posts
Re: Ask HN: How are you doing RAG locally?
#22[deleted]
Re: Ask HN: How are you doing RAG locally?
#23I built a lib for myself https://pypi.org/project/piragi/
That looks great! Is there a way to store / cache the embeddings?
Re: Ask HN: How are you doing RAG locally?
#24Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
I agree. Someone here posted a drop-in for grep that added the ability to do hybrid text/vector search but the constant need to re-index files was annoying and a drag. Moreover, vector search can add a ton of noise if the model isn't meant for code search and if you're not using a re-ranker.
For all intents and purposes, running gpt-oss 20B in a while loop with access to ripgrep works pretty dang well. gpt-oss is a tool calling god compared to everything else i've tried, and fast.
Re: Ask HN: How are you doing RAG locally?
#25Embedded usearch vector database. https://github.com/unum-cloud/USearch
Re: Ask HN: How are you doing RAG locally?
#26Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
I've gotten great results applying it to file paths + signatures. Even better if you also fuse those results with BM25.
Re: Ask HN: How are you doing RAG locally?
#27Any suggestion what to use as embeddings model runtime and semantic search in C++?
Re: Ask HN: How are you doing RAG locally?
#28The Nextcloud MCP Server [0] supports Qdrant as a vectordb to store embeddings and provide semantic search across your personal documents. This enables any LLM & MCP client (e.g. claude code) into a RAG system that you can use to chat with your files.
For local deployments, Qdrant supports storing embeddings in memory as well as in a local directory (similar to sqlite) - for larger deployments Qdrant supports running as a standalone service/sidecar and can be made available over the network.
Re: Ask HN: How are you doing RAG locally?
#29I have done some experiments with nomic embedding through Ollama and ChromaDB.
Works well, but I didn't tested on larger scale
Re: Ask HN: How are you doing RAG locally?
#30Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
Anybody know of a good service / docker that will do BM25 + vector lookup without spinning up half a dozen microservices?