RAG Is Simpler Than You Think • Hacker News Board

Re: RAG Is Simpler Than You Think

#81

Post by freakynit » Wed, Aug 26, 2026, 12:25 PM UTC

robrorcroptrer wrote: post #51

usernametaken29 wrote: post #29

Earlier quoted context omitted.

> embed everything the first time This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”

What about splitting bigger content into chunks before embedding?

How are you gonna handle the relations that span across individual chunks... if a later chunk refers something from 2 chunks before using `it`, rather than proper name, how will you handle that? Because at query time, that later chunk would not match.

Re: RAG Is Simpler Than You Think

#82

Post by khalic » Wed, Aug 26, 2026, 12:27 PM UTC

usernametaken29 wrote: post #29

refactor_master wrote: post #14

Here’s an even simpler take: just embed everything the first time, then track what was changed. Use a cheap model to summarize and clean up the documents/chats with summary and keywords. Unless you have entire libraries of books to embed it’s going to be a few hundred dollars of API calls. Then, throw it all in BigQuery. Handles all the vector stuff natively. Sprinkle an agentic bot UI thing on top to make it appear…

> embed everything the first time This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”

you won't get anything out of a whole book embedding anyway, even a structured page is too much

Re: RAG Is Simpler Than You Think

#83

Post by trivet » Wed, Aug 26, 2026, 12:31 PM UTC

Start with BM25 and only add embeddings when keyword search actually fails you. Saves a lot of pain.

Re: RAG Is Simpler Than You Think

#84

Post by ivansavz » Wed, Aug 26, 2026, 12:33 PM UTC

Does anyone have experience using SMLs for RAG (either as query rewriter or as generator for the final answer)?

I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.

I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."

Any info or advice would be appreciated.

Re: RAG Is Simpler Than You Think

#85

Post by josh_p » Wed, Aug 26, 2026, 12:33 PM UTC

usernametaken29 wrote: post #27

I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably…

I worked on getting an address database into elasticsearch years ago when it was still using modified tf-idf. Customers wanted FTS where a lot of the queries would be something like "100 First Ave, NY" or "200 2nd St, MN".

It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.

It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.

Re: RAG Is Simpler Than You Think

#86

Post by brazukadev » Wed, Aug 26, 2026, 12:33 PM UTC

ninkendo wrote: post #53

Lorean1 wrote: post #36

Earlier quoted context omitted.

Maybe if a person can't even google RAG they are not the intended audience of that article.

When I hear stuff like this I always imagine going to a restaurant and asking the waitress for a menu and them replying “lol just google it”. It’s not that I can’t or don’t know how, it’s rather that the expectation should be that a website should… link you to the information it believes to be relevant background. It’s why it’s called a “web”, linking is a core concept.

> When I hear stuff like this I always imagine going to a restaurant and asking the waitress for a menu and them replying “lol just google it”.

in this case there was a menu in the next empty table and you saw it but in place of getting it you want the waitress to get it for you. Which is a normal behavior but you could save your time by just getting the menu yourself.

Re: RAG Is Simpler Than You Think

#87

Post by respectattentio » Wed, Aug 26, 2026, 12:38 PM UTC

I believe embedding-based RAG, everybody is using, will end. As chips advance, you would use a big llm instead of word embedding for retrieval. It's much more accurate and extensive covering every topic.

Still need ~2 years to be replaced.

Re: RAG Is Simpler Than You Think

#88

Post by seanspradlin0 » Wed, Aug 26, 2026, 12:38 PM UTC

But over-engineering things is fun.

RAG is one of those things where I can hyper optimize to an absolutely needless degree.

Re: RAG Is Simpler Than You Think

#89

Post by mmargenot » Wed, Aug 26, 2026, 12:46 PM UTC

usernametaken29 wrote: post #27

I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably…

And you get bm25 for free with so many modern setups! I do still love to experiment with tuning semantic search for your specific corpus via various kinds of embeddings, but bm25 is hard to beat.

Re: RAG Is Simpler Than You Think

#90

Post by quijoteuniv » Wed, Aug 26, 2026, 12:47 PM UTC

usernametaken29 wrote: post #27

I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably…

On my last go at making my own rag i still got better results by collecting the data and uploading to a project in open(butclosed)ai. My own rag, used by an agent was giving poorer results, and even the agent prefered (derailed)to not use it and look for the info itself rather than using the rag