Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Contextual Retrieval

anthropic.com

Post reply on HN
Print view View original

11–20 of 73 posts

Re: Contextual Retrieval

#11

Post by valstu »

cabidaher wrote: post #8
valstu wrote: post #6

We're doing something similar. We first chunk the documents based on h1,h2,h3 headings. Then we add headers in the beginning of the chunk as a context. As an imagenary example, instead of one chunk being: The usual dose for adults is one or two 200mg tablets or capsules 3 times a day. It is now something like: # Fever ## Treatment --- The usual dose for adults is one or two 200mg tablets or capsules 3 times a day. Th…

Did you experiment with different ways to format those included headers? Asking because I am doing something similar to that as well.

Nope, not yet. We have sticked with markdownish syntax so far.

Re: Contextual Retrieval

#12

Post by skeptrune »

resiros wrote: post #10
postalcoder wrote: post #9

To add some context, this isn't that novel of an approach. A common approach to improve RAG results is to "expand" the underlying chunks using an llm, so as to increase the semantic surface area to match against. You can further improve your results by running query expansion using HyDE[1], though it's not always an improvement. I use it as a fallback. I'm not sure what Anthropic is introducing here. I looked at the…

I think the innovation is using caching as so to make the cost of the approach manageable. The way they implemented it is that each time you create a chunk, you ask the llm to create an atomic chunk from the whole context. You need to do this for all tens of thousands of chunks in your data. This costs a lot. By caching the documents, you can spare costs

You could also just save the first outputted atomic chunk and store it then re-use it each time yourself. Easier and more consistent.

Re: Contextual Retrieval

#13

Post by postalcoder »

resiros wrote: post #10
postalcoder wrote: post #9

To add some context, this isn't that novel of an approach. A common approach to improve RAG results is to "expand" the underlying chunks using an llm, so as to increase the semantic surface area to match against. You can further improve your results by running query expansion using HyDE[1], though it's not always an improvement. I use it as a fallback. I'm not sure what Anthropic is introducing here. I looked at the…

I think the innovation is using caching as so to make the cost of the approach manageable. The way they implemented it is that each time you create a chunk, you ask the llm to create an atomic chunk from the whole context. You need to do this for all tens of thousands of chunks in your data. This costs a lot. By caching the documents, you can spare costs

Yup. Caching is very nice.. but the framing is weird. "Introducing" to me, connotes a product release, not a new tutorial.

Re: Contextual Retrieval

#14

Post by postalcoder »

skeptrune wrote: post #12
resiros wrote: post #10

Earlier quoted context omitted.

I think the innovation is using caching as so to make the cost of the approach manageable. The way they implemented it is that each time you create a chunk, you ask the llm to create an atomic chunk from the whole context. You need to do this for all tens of thousands of chunks in your data. This costs a lot. By caching the documents, you can spare costs

You could also just save the first outputted atomic chunk and store it then re-use it each time yourself. Easier and more consistent.

To be fair, that only works if you keep chunk windows static.

Re: Contextual Retrieval

#15

Post by TmpstsTrrctta »

jillesvangurp wrote: post #5
simonw wrote: post #4

My favorite thing about this is the way it takes advantage of prompt caching. That's priced at around 1/10th of what the prompts would normally cost if they weren't cached, which means that tricks like this (running every single chunk against a full copy of the original document) become feasible where previously they wouldn't have financially made sense. I bet there are all sorts of other neat tricks like this which…

You could do a lot of stuff with pre-calculating things for your embeddings. Why cache when you can pre-calculate. That brings into play a whole lot of things people commonly do as part of ETL. I come from a traditional search back ground. It's quite obvious to me that RAG is a bit of a naive strategy if you limit it to just using vector search with some off the shelf embedding model. Vector search simply isn't that…

I have experience in traditional search as well and I think this is doing some limiting of my imagination when it comes to vector search. In the post, I did like the introduction of the Contextual BM25 compared to other hybrid approaches then doing rrf.

For question answering, vector/semantic search is clearly a better fit in my mind, and I can see how the contextual models can enable and bolster that. However, because I’ve implemented and used so many keyword based systems, that just doesn’t seem to be how my brain works.

An example I’m thinking of is finding a sushi restaurant near me with availability this weekend around dinner time. I’d love to be able to search for this as I’ve written it. How I would search for it would be search for sushi restaurant, sort by distance and hope the application does a proper job of surfacing time filtering.

Conversely, this is mostly how I would build this system. Perhaps with a layer to determine user intention to pull out restaurant type, location sorting, and time filtering.

I could see using semantic search for filtering down the restaurants to related to sushi, but do we then drop back into traditional search for filtering and sorting? Utilize function calling to have the LLM parameterize our search query?

As stated, perhaps I’m not thinking of these the right way because of my experiences with existing systems, which I find seem to give me better results when well built

Re: Contextual Retrieval

#16

Post by _bramses »

The technique I find most useful is to implement a “linked list” strategy where a chunk has multiple pointers to the entry it is referenced by. This task is done manually, but the diversity of the ways you can reference a particular node go up dramatically.

Another way to look at it, comments. Imagine every comment under this post is a pointer back to the original post. Some will be close in distance, and others will be farther, due to the perception of the authors of the comments themselves. But if you assign each comment a “parent_id”, your access to the post multiplies.

You can see an example of this technique here [1]. I don’t attempt to mind read what the end user will query for, I simply let them tell me, and then index that as a pointer. There are only a finite number of options to represent a given object. But some representations are very, very, very far from the semantic meaning of the core object.

[1] - https://x.com/yourcommonbase/status/1833262865194557505

Re: Contextual Retrieval

#17

Post by postalcoder »

jillesvangurp wrote: post #5
simonw wrote: post #4

My favorite thing about this is the way it takes advantage of prompt caching. That's priced at around 1/10th of what the prompts would normally cost if they weren't cached, which means that tricks like this (running every single chunk against a full copy of the original document) become feasible where previously they wouldn't have financially made sense. I bet there are all sorts of other neat tricks like this which…

You could do a lot of stuff with pre-calculating things for your embeddings. Why cache when you can pre-calculate. That brings into play a whole lot of things people commonly do as part of ETL. I come from a traditional search back ground. It's quite obvious to me that RAG is a bit of a naive strategy if you limit it to just using vector search with some off the shelf embedding model. Vector search simply isn't that…

Graph RAG is very cool and outstanding at filling some niches. IIRC, Perplexity's actual search is just BM25 (based a lex fridman interview of the founder).

Re: Contextual Retrieval

#18

Post by jillesvangurp »

postalcoder wrote: post #17
jillesvangurp wrote: post #5

Earlier quoted context omitted.

You could do a lot of stuff with pre-calculating things for your embeddings. Why cache when you can pre-calculate. That brings into play a whole lot of things people commonly do as part of ETL. I come from a traditional search back ground. It's quite obvious to me that RAG is a bit of a naive strategy if you limit it to just using vector search with some off the shelf embedding model. Vector search simply isn't that…

Graph RAG is very cool and outstanding at filling some niches. IIRC, Perplexity's actual search is just BM25 (based a lex fridman interview of the founder).

Makes sense; perplexity is really responsive and fast usually.

I need to check out that interview with Lex Fridman.

Re: Contextual Retrieval

#19

Post by calufa »

postalcoder wrote: post #17
jillesvangurp wrote: post #5

Earlier quoted context omitted.

You could do a lot of stuff with pre-calculating things for your embeddings. Why cache when you can pre-calculate. That brings into play a whole lot of things people commonly do as part of ETL. I come from a traditional search back ground. It's quite obvious to me that RAG is a bit of a naive strategy if you limit it to just using vector search with some off the shelf embedding model. Vector search simply isn't that…

Graph RAG is very cool and outstanding at filling some niches. IIRC, Perplexity's actual search is just BM25 (based a lex fridman interview of the founder).

Do you have the link and the time in the video where he mentions it?

Re: Contextual Retrieval

#20

Post by visarga »

jillesvangurp wrote: post #5
simonw wrote: post #4

My favorite thing about this is the way it takes advantage of prompt caching. That's priced at around 1/10th of what the prompts would normally cost if they weren't cached, which means that tricks like this (running every single chunk against a full copy of the original document) become feasible where previously they wouldn't have financially made sense. I bet there are all sorts of other neat tricks like this which…

You could do a lot of stuff with pre-calculating things for your embeddings. Why cache when you can pre-calculate. That brings into play a whole lot of things people commonly do as part of ETL. I come from a traditional search back ground. It's quite obvious to me that RAG is a bit of a naive strategy if you limit it to just using vector search with some off the shelf embedding model. Vector search simply isn't that…

GraphRAG requires you define upfront the schema of entity and relation types. This works when you are in a known domain, but in general, when you want to just answer questions from a large reference, you don't know what you need to put in the graph.
Post reply on HN