If like me you run models locally, it's pretty easy to run your own RAG locally also using a Vector Database like Qdrant for persistence, and a middle-layer like Mem0 for realtime retrial and updates. I documented the set-up steps here: https://leadprompt.sh/a/739-Building-an-Infinite-Memory-Loca...
RAG Is Simpler Than You Think
111–120 of 126 posts
Re: RAG Is Simpler Than You Think
#112I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably…
> people vastly underestimate full text search It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
> It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
It is, if people don't even stop to think if they need synonyms, periphrases, or mistaken neighbours.
As the blog post points out, more often than not you don't, particularly if your primary usecase is to search for technical keywords or codenames.
Re: RAG Is Simpler Than You Think
#113Earlier quoted context omitted.
What about splitting bigger content into chunks before embedding?
How are you gonna handle the relations that span across individual chunks... if a later chunk refers something from 2 chunks before using `it`, rather than proper name, how will you handle that? Because at query time, that later chunk would not match.
Humans usually have ways around that in longer documents eg page numbers, paragraphs, links.
If someone gave me a report, in my hands, that said “see ‘it’” I’d also be confused.
Re: RAG Is Simpler Than You Think
#114More LLM-generated text about LLMs. Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.
the biggest giveway is actually not the writing style, but the content
"using GPT-4o-mini for query rewriting" -> model from 2024, when RAG was trendy, and all the langchain, llama-index, etc, docs mentioned this specific model
Re: RAG Is Simpler Than You Think
#115More LLM-generated text about LLMs. Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.
Everything that is generate from a LLM is shit, I don't know why people continue using it. I'm waiting for this bubble to explode once for all so we can return doing things in the sane way.
Re: RAG Is Simpler Than You Think
#116Very little of this is RAG but rather just FTS with clever reformulation and re-ranking.
RAG is about providing an grounded response, given the actual data in the corpus.
Great article and content, nonetheless!!
Re: RAG Is Simpler Than You Think
#117Earlier quoted context omitted.
What about splitting bigger content into chunks before embedding?
How are you gonna handle the relations that span across individual chunks... if a later chunk refers something from 2 chunks before using `it`, rather than proper name, how will you handle that? Because at query time, that later chunk would not match.
Absolutely a novice in this topic, but I would imagine that by simply having sufficiently big chunks it's simply not a problem? You surely have enough information in like a couple of paragraphs to denote in vector space roughly what it is about. So that both chunks would get found by a vector search, and then whatever is the logic it may put the whole original text of those chunks into context, but in any case enough so that an LLM can "reason" about the references in-between the two.
Re: RAG Is Simpler Than You Think
#118I have built systems using all of these approaches (all in tandem). For the most part, the juice is not worth the squeeze (in building a highly optimised corpus-specific information retrieval strategy) outside of a very few fringe cases. The amount of technical discussion far outstrips the use case for RAG.
Re: RAG Is Simpler Than You Think
#119More LLM-generated text about LLMs. Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.
Everything that is generate from a LLM is shit, I don't know why people continue using it. I'm waiting for this bubble to explode once for all so we can return doing things in the sane way.
[dead]
Re: RAG Is Simpler Than You Think
#120I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably…
Re: the rube goldberg machine of diminishing returns
https://www.anthropic.com/engineering/contextual-retrieval
This is from two years ago, but I think it's still SotA?