Solving the out-of-context chunk problem for RAG
1–10 of 93 posts
Re: Solving the out-of-context chunk problem for RAG
#2RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.
Re: Solving the out-of-context chunk problem for RAG
#3RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.
What about fresh data like an extremely relevant news headline that was published 10 minutes ago? Private data that I don’t want stored offsite but am okay trusting an enterprise no log api? Providing realtime context to LLMs isn’t “hacky”, model intelligence and RAG can complement each other and make advancements in tandem
Re: Solving the out-of-context chunk problem for RAG
#4The easiest solution to this is to stuff the heading into the chunk. The heading is hierarchical navigation within the sections of the document.
I found Azure Document Intelligence specifically with the Layout Model to be fantastic for this because it can identify headers. All the better if you write a parser for the output JSON to track depth and stuff multiple headers from the path into the chunk.
Re: Solving the out-of-context chunk problem for RAG
#5RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.
What about fresh data like an extremely relevant news headline that was published 10 minutes ago? Private data that I don’t want stored offsite but am okay trusting an enterprise no log api? Providing realtime context to LLMs isn’t “hacky”, model intelligence and RAG can complement each other and make advancements in tandem
I don't think the parents idea was to bake all information into the model, just that current RAG feels cumbersome to use (but then again, so do most things AI right now) and information access should be intrinsic part of the model.
Re: Solving the out-of-context chunk problem for RAG
#6Earlier quoted context omitted.
What about fresh data like an extremely relevant news headline that was published 10 minutes ago? Private data that I don’t want stored offsite but am okay trusting an enterprise no log api? Providing realtime context to LLMs isn’t “hacky”, model intelligence and RAG can complement each other and make advancements in tandem
I don't think the parents idea was to bake all information into the model, just that current RAG feels cumbersome to use (but then again, so do most things AI right now) and information access should be intrinsic part of the model.
[deleted]
Re: Solving the out-of-context chunk problem for RAG
#7I'd like to see more evaluation data. There are 100s of RAG strategies, most of them only work on specific types of queries.
Re: Solving the out-of-context chunk problem for RAG
#8The easiest solution to this is to stuff the heading into the chunk. The heading is hierarchical navigation within the sections of the document. I found Azure Document Intelligence specifically with the Layout Model to be fantastic for this because it can identify headers. All the better if you write a parser for the output JSON to track depth and stuff multiple headers from the path into the chunk.
Contextual chunk headers
The idea here is to add in higher-level context to the chunk by prepending a chunk header. This chunk header could be as simple as just the document title, or it could use a combination of document title, a concise document summary, and the full hierarchy of section and sub-section titles.
That is from the article. Is this different from your suggested approach?
Re: Solving the out-of-context chunk problem for RAG
#9RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.
That's so vague I can't tell what you're suggesting. What specifically do you think needs solving at the model level? What should work differently?
Re: Solving the out-of-context chunk problem for RAG
#10Earlier quoted context omitted.
What about fresh data like an extremely relevant news headline that was published 10 minutes ago? Private data that I don’t want stored offsite but am okay trusting an enterprise no log api? Providing realtime context to LLMs isn’t “hacky”, model intelligence and RAG can complement each other and make advancements in tandem
I don't think the parents idea was to bake all information into the model, just that current RAG feels cumbersome to use (but then again, so do most things AI right now) and information access should be intrinsic part of the model.
Is there a specific shortcoming of the model that could be improved, or are we simply seeking better APIs?