[dead]
Ask HN: How are you doing RAG locally?
11–20 of 166 posts
Re: Ask HN: How are you doing RAG locally?
#12Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
Re: Ask HN: How are you doing RAG locally?
#13lee101/gobed https://github.com/lee101/gobed static embedding models so they are embedded in milliseconds and on gpu search with a cagra style on gpu index with a few things for speed like int8 quantization on the embeddings and fused embedding and search in the same kernel as the embedding really is just a trained map of embeddings per token/averaging
Re: Ask HN: How are you doing RAG locally?
#14Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
static embedding models im finding quite fast
lee101/gobed https://github.com/lee101/gobed is 1ms on gpu :) would need to be trained for code though the bigger code llm embeddings can be high quality too so its just yea about where is ideal on the pareto fronteir really , often yea though your right it tends to be bm25 or rg even for code but yea more complex solutions are kind of possible too if its really important the search is high quality
Re: Ask HN: How are you doing RAG locally?
#15[flagged]
Re: Ask HN: How are you doing RAG locally?
#16[flagged]
Re: Ask HN: How are you doing RAG locally?
#17sqlite's bm25
Re: Ask HN: How are you doing RAG locally?
#18I built a lib for myself https://pypi.org/project/piragi/
Re: Ask HN: How are you doing RAG locally?
#19Sqlite-vec
Re: Ask HN: How are you doing RAG locally?
#20Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
With AI needing more access to documentation, WDYT about using RAG for documentation retrieval?