sqlite + FTS + sqlite-vec + local LLM for reranking results (reasoning model)
Ask HN: How are you doing RAG locally?
61–70 of 166 posts
Re: Ask HN: How are you doing RAG locally?
#62You don’t need a vector database or graph, it really depends on your existing infrastructure , file types and needs.
The newer “agent” search approach can just query a file system or api. It’s slightly slower but easier to setup and maintain as no extra infrastructure.
Re: Ask HN: How are you doing RAG locally?
#63Anyone use these approaches with academic pdfs?
Anyone using them for electronics datasheets?
Re: Ask HN: How are you doing RAG locally?
#64Most of my complex documents are, luckily, Markdown files.
I can recommend https://github.com/tobi/qmd/ . It’s a simple CLI tool for searching in these kinds of files. My previous workflow was based on fzf, but this tool gives better results and enables even more fuzzy queries. I don’t use it for code, though.
Re: Ask HN: How are you doing RAG locally?
#65Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
Anybody know of a good service / docker that will do BM25 + vector lookup without spinning up half a dozen microservices?
Elasticsearch / Opensearch is the industry standard for this
Re: Ask HN: How are you doing RAG locally?
#66Earlier quoted context omitted.
What order of magnitude would you define as „large“ in this case?
like over 1tb.
Some people are using DuckDB for large datasets, https://duckdb.org/docs/stable/guides/performance/working_wi... , but you'd probably do some testing under the specific conditions of your rig to figure out if it is a good match or not.
Re: Ask HN: How are you doing RAG locally?
#67Giving the LLM tools with an OData query interface has worked well for me. In C# it's pretty trivial to set up an MCP server with OData querying for an arbitrary data model. At work we have an Excel sheet with 40k rows which the LLM was able to quickly and reliably analyse using this method.
Re: Ask HN: How are you doing RAG locally?
#68Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
Anybody know of a good service / docker that will do BM25 + vector lookup without spinning up half a dozen microservices?
Meilisearch
Re: Ask HN: How are you doing RAG locally?
#69Earlier quoted context omitted.
Anybody know of a good service / docker that will do BM25 + vector lookup without spinning up half a dozen microservices?
Elasticsearch / Opensearch is the industry standard for this
Used to be, but they're very complicated to operate compared to more modern alternatives and have just gotten more and more bloated over the years. Also require a bunch of different applications for different parts of the stack in order to do the same basic stuff as e.g. Meilisearch, Manticore or Typesense.
Re: Ask HN: How are you doing RAG locally?
#70Claude code / codex which internally uses ripgrep, and I'm unsure if it's using parallel mode. And, project specific static analyzers.
Studies generally show when you do agentic retrieval w/ text search, that's pretty good. Adding vector retrieval and graph rag, so the typical parallel multi-retrieval followed by reranking, gives a bit of speedup and quality lift. That lines up with my local flow experience, where it is only enough that I want that for $$$$ consumer/prosumer tools, and not easy enough for DIY that I want to invest in that locally. For those who struggle with tools like spotlight running when it shouldn't, that kind of thing turns me off on the cost/benefit side.
For code, I experiment with unsound tools (semgrep, ...) vs sound flow analyzers, carefully setup for the project. Basically, ai coders love to use grep/sed for global replace refactors and other global needs, but keeps tripped up on sound flow analysis. Similar to lint and type checking, that needs to be setup for a project and taught as a skill. I'm not happy with any of my experiments here yet however :(