Ask HN: How are you doing RAG locally?

Sure, I've managed both clusters and single node deployments in production until 2025 when I changed jobs. Elastic definitely does have its strengths, but they're increasingly enterprise-oriented and appear not to care a lot about open source deployments. At one point Elastic itself had a severe regression in an irreverible patch update (!?) which took weeks to fix, forcing us to recover from backup and recreate the index. The documentation is or has been ambigious and self-contradicting on a lot of points. The Debian Elastic Enterprise Search package upgrade script was incomplete, so there's a significant manual process for updating the index even for patch updates. The interfaces between the different components of the ELK stack are incoherent and there's literally a thousand ways to configure them. Default setups have changed a lot over the years, leading to incoherent documentation. You really need to be an expert at Elastic in order to run it well – or pay handsomely for the service. It's simply too complicated and costly for what it is, compared to more recent alternatives.

I've just released a casual personal project for Ephemeral GraphRAG. It's still experimental and open source: https://github.com/gibram-io/gibram

I have production agents which run vector search via FAISS locally ( in their env not 3rd party environments ), and for which I am creating embeddings for specific domains.

1 - agent memory ( its an ai coach so its the unique training methods that allow for instant adoption of new skills and distilling best fit skills for context )

2 - user memory ( the ai coaches memory of a user )

3 - session memory ( for long conversations, instead of compaction or truncation )

Then separately I have coding agents which I give semantic search, same system FAISS

- on command they create new memories from lessons ( consumes tokens * ) - they vector search FAISS when needing more context ( 2x greater agent alignment / outcomes this way )

And finally I forked openais codex terminal agent code to add - inbuilt vector search and injection

So I say "Find any uncovered TDD opportunity matching intent to actuality for auth on these 3 repos, write TDD coverage, and bring failures to my attention"

They set my message to {$query}

vector search on {$query}

embed results in their context window

programmatically - so no token consumption ( what a freaking dream )

thats open source if helpful

Its here

https://github.com/Next-AI-Labs-Inc/codex/tree/nextailabs

Im trying to determine where something like this fits in

https://huggingface.co/MongoDB/mdbr-leaf-ir

My gaps right now are ...

I am not training the agents yet, like fine tuning the underlying models.

Would love the simplest approach to test this, because at least with the codex clone I could easily swap out local models, but somehow doubting that they will be able to match performance of the outsourced models.

especially bc claude code just launched ahead of codex in the last week or so in quality, and they are closed source. Im seeing clear swarm agentic coding internally which is a dream for context window efficiency. ( in claude code as of today )