Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How are you doing RAG locally?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 166 posts

Re: Ask HN: How are you doing RAG locally?

#141

Post by folli »

alansaber wrote: post #108
eb0la wrote: post #97

We started with PGVector just because we already knew Postgres and it was easy to hand over to the operations people. After some time we noticed a semi-structured field in the prompt had a 100% match with the content needed to process the prompt. Turns out operators started puting tags both in the input and the documents that needed to match on every use case (not much, about 50 docs). Now we look for the field first…

Most vectordb is a hammer looking for a nail

I think it can be more efficient for two-step RAG so you can reuse the natural language query directly, but for agentic RAG it might indeed be overkill.

Re: Ask HN: How are you doing RAG locally?

#142

Post by rcarmo »

navar wrote: post #87

For the retrieval stage, we have developed a highly efficient, CPU-only-friendly text embedding model: https://huggingface.co/MongoDB/mdbr-leaf-ir It ranks #1 on a bunch of leaderboards for models of its size. It can be used interchangeably with the model it has been distilled from ( https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1... ). You can see an example comparing semantic (i.e., embeddings-based) s…

Hmmm. I recently created https://github.com/rcarmo/asterisk-embedding-model, need to look at this since I had very limited training resources.

Re: Ask HN: How are you doing RAG locally?

#143

Post by marwamc »

tubs wrote: post #137
marwamc wrote: post #129

Earlier quoted context omitted.

Will fix the links. Meanwhile here is the releases page. I develop on gitlab and mirror to github. Need to make that clear as well. https://gitlab.com/rhobimd-oss/shebe/-/releases

Ah, I tried the gitlab and the tarballs 404 for me there, sorry I should have been more specific in the original post! fwiw this does look interesting.

I see what's happening. I never validated those build artifacts... Thanks for the catch. Will rebuild notify you here.

Re: Ask HN: How are you doing RAG locally?

#144

Post by riku_iki »

cess11 wrote: post #66
m00dy wrote: post #60

Earlier quoted context omitted.

like over 1tb.

Some people are using DuckDB for large datasets, https://duckdb.org/docs/stable/guides/performance/working_wi... , but you'd probably do some testing under the specific conditions of your rig to figure out if it is a good match or not.

its clear many DuckDB sql queries can handle terabytes of data, but the question here was about vector search..

Re: Ask HN: How are you doing RAG locally?

#145

Post by alchemist1e9 »

HarHarVeryFunny wrote: post #91
aqula wrote: post #51

Earlier quoted context omitted.

LSP is not great for non-editor use cases. Everything is cursor position oriented.

Yes, something like TreeSitter would seem to be of more value - able to lookup symbols by name, and find the spans of source code where they are defined and used.

https://github.com/ast-grep/ast-grep

Re: Ask HN: How are you doing RAG locally?

#146

Post by navar »

jasonjmcghee wrote: post #134
navar wrote: post #87

For the retrieval stage, we have developed a highly efficient, CPU-only-friendly text embedding model: https://huggingface.co/MongoDB/mdbr-leaf-ir It ranks #1 on a bunch of leaderboards for models of its size. It can be used interchangeably with the model it has been distilled from ( https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1... ). You can see an example comparing semantic (i.e., embeddings-based) s…

How does performance (embedding speed and recall) compare to minish / model2vec static word embeddings?

I interacted with the authors of these models quite a bit!

These are very interesting models.

The tradeoff here is that you get even faster inference, but lose on retrieval accuracy [0].

Specifically, inference will be faster because essentially you are only doing tokenization + a lookup table + an average. So despite the fact that their largest model is 32M params, you can expect inference speeds to be higher than ours, which 23M params but it is transformer-based.

I am not sure about typical inference speeds on a CPU for their models, but with ours you can expect to do ~22 docs per second, and ~120 queries per second on a standard 2vCPU server.

As far as retrieval accuracy goes, on BEIR we score 53.55, all-MiniLM-L12-v2 (a widely adopted compact text embedding model) scores 42.69, while potion-8M scores 30.43.

I can't find their larger models but you can generally get an idea of the power level of different embedding models here: https://huggingface.co/spaces/mteb/leaderboard

If you want to run them on a CPU it may make sense to filter for smaller models (e.g., [0] "accuracy" in layman terms, not in accuracy vs recall terms. The correct word here would be "effectiveness".

Re: Ask HN: How are you doing RAG locally?

#147

Post by threecheese »

For my personal PKM slash “learn this crap”, I have a fully local hybrid search on my MacBook using MLX and SQLite.

I store file content blobs in SQLite, and use FTS5 (bm25) to maintain a fulltext index plus sqlite-vec for storing embeddings. Search uses both of these, and then reciprocal rank fusion gets the best results and pipes those to a local transformers model to judge. It’s all Python with mlx-lm and mlx-embeddings libraries, the models are grabbed from huggingface. It’s not the fastest, but it’s local and easy to understand (and for Claude to write, mostly).

Re: Ask HN: How are you doing RAG locally?

#148

Post by threecheese »

postalcoder wrote: post #24
CuriouslyC wrote: post #12

Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.

I agree. Someone here posted a drop-in for grep that added the ability to do hybrid text/vector search but the constant need to re-index files was annoying and a drag. Moreover, vector search can add a ton of noise if the model isn't meant for code search and if you're not using a re-ranker. For all intents and purposes, running gpt-oss 20B in a while loop with access to ripgrep works pretty dang well. gpt-oss is a t…

Say more!

Re: Ask HN: How are you doing RAG locally?

#149

Post by threecheese »

someguyiguess wrote: post #122
amscotti wrote: post #115

More of a proof of concept to test out ideas, but here's my approach for local RAG, https://github.com/amscotti/local-LLM-with-RAG Using Ollama for the embeddings with “nomic-embed-text”, with LanceDB for the vector database. Recently updated it to use “agentic” RAG, but probably not fully needed for a small project.

Woah. I am doing something very similar also using lancedb https://github.com/nicholaspsmith/lance-context Mine is much more basic than yours and I just started it a couple of weeks ago.

There are so many of us doing the same, just had a similar conversation at $work. It’s pretty exciting. I feel like I’m having to shove another 20 years of development experience into my brain with all these new concepts and abstractions, but the dots have been connecting!

Re: Ask HN: How are you doing RAG locally?

#150

Post by 3abiton »

navar wrote: post #87

For the retrieval stage, we have developed a highly efficient, CPU-only-friendly text embedding model: https://huggingface.co/MongoDB/mdbr-leaf-ir It ranks #1 on a bunch of leaderboards for models of its size. It can be used interchangeably with the model it has been distilled from ( https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1... ). You can see an example comparing semantic (i.e., embeddings-based) s…

And honestly in a lot of the cases bm25 has been the best approach used in many of the projects we deployed.
Post reply on HN