Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Contextual Retrieval

anthropic.com

Post reply on HN
Print view View original

51–60 of 73 posts

Re: Contextual Retrieval

#51

Post by ValentinA23 »

TmpstsTrrctta wrote: post #15
jillesvangurp wrote: post #5

Earlier quoted context omitted.

You could do a lot of stuff with pre-calculating things for your embeddings. Why cache when you can pre-calculate. That brings into play a whole lot of things people commonly do as part of ETL. I come from a traditional search back ground. It's quite obvious to me that RAG is a bit of a naive strategy if you limit it to just using vector search with some off the shelf embedding model. Vector search simply isn't that…

I have experience in traditional search as well and I think this is doing some limiting of my imagination when it comes to vector search. In the post, I did like the introduction of the Contextual BM25 compared to other hybrid approaches then doing rrf. For question answering, vector/semantic search is clearly a better fit in my mind, and I can see how the contextual models can enable and bolster that. However, becau…

Another approach I saw is to build a conceptual graph using entity extraction and have the LLM suggest search paths through that graph to enhance the retrieval step. The LMM is fine-tuned on the conceptual graph for this specific task. Could work in your case, but you need to deal with an ontology that suits your use case, in other words it must already contain restaurant location, type of dishes served and opening hours.

Re: Contextual Retrieval

#52

Post by will-burner »

I wish they included the datasets they used for the evaluations. As far as I can tell, in appendix II they include some sample questions, answers, and golden chunks but they do not give the entire dataset or give an explicit information on exactly what the datasets are.

Does anyone know if the datasets they used for the evaluation are publicly available or if they give more information on the datasets than what's in appendix II?

There are standard publically available datasets for this type of evaluation, like MTEB (https://github.com/embeddings-benchmark/mteb). I wonder how this technique does on the MTEB dataset.

Re: Contextual Retrieval

#54

Post by quantadev »

thelastparadise wrote: post #31

Can someone explain simply how these benchmarks work? What exactly is a "failure rate" and how is it computed?

They simply ask the AI a question about a large document (or set of docs). It either gets the answer right or wrong. They count the number of hits and misses.

Re: Contextual Retrieval

#55

Post by KTibow »

spott wrote: post #43
bayesianbot wrote: post #26

Earlier quoted context omitted.

I was trying to do this using Prompt Caching like a month ago, but then noticed there's five minute maximum lifetime for the cached prompts - doesn't really work for my RAG needs (or probably most), where the queries would be ran during the next month or a year. I can't see any changes to that policy. Little surprised to see them talk about Prompt Caching relating to RAG.

They aren’t using the prompt caching on the query side, only on the embedding side… so you cache the document in the context window when ingesting it, but not during retrieval.

It seems a little odd to make multiple requests instead of using one request to create all the context for all the chunks.

Re: Contextual Retrieval

#56

Post by simonw »

davedx wrote: post #50
simonw wrote: post #4

My favorite thing about this is the way it takes advantage of prompt caching. That's priced at around 1/10th of what the prompts would normally cost if they weren't cached, which means that tricks like this (running every single chunk against a full copy of the original document) become feasible where previously they wouldn't have financially made sense. I bet there are all sorts of other neat tricks like this which…

Cost is one aspect, but what about ingest time? You’re adding significant processing time to your pipeline with this method right?

I expect most implementations of RAG don't mind this too much - if you're dealing with only a few hundred more pages of documents a day the ingestion time from using fancy tricks like this is going to be measured in minutes.

Re: Contextual Retrieval

#57

Post by skybrian »

visarga wrote: post #22
skybrian wrote: post #2

This sounds a lot like how we used to do research, by reading books and writing any interesting quotes on index cards, along with where they came from. I wonder if prompting for that would result in better chunks? It might make it easier to review if you wanted to do it manually.

The fundamental problem of both keyword and embedding based retrieval is that they only access surface level features. If your document contains 5+5 and you search "where is the result 10" you won't find the answer. That is why all texts need to be "digested" with LLM before indexing, to draw out implicit information and make it explicit. It's also what Anthropic proposes we do to improve RAG. "study your data before…

Makes sense. It seems after retrieval, both would be useful - both the exact quote and a summary of its context.

Re: Contextual Retrieval

#58

Post by passion__desire »

torginus wrote: post #25
skeptrune wrote: post #3

I'm not a fan of this technique. I agree the scenario they lay out is a common problem, but the proposed solution feels odd. Vector embeddings have bag-of-words compression properties and can over-index on the first newline separated text block to the extent that certain indices in the resulting vector end up much closer to 0 than they otherwise would. With quantization, they can eventually become 0 and cause you to…

Sorry random question - do vector dbs work across models? I'd guess no, since embeddings are models specific afaik, but that means that a vector db would lock you into using a single LLM and even within that, a single version, like Claude-3.5 Sonnet, and you couldn't move to 3.5 Haiku, Opus etc., never mind ChatGPT or Llama without reindexing.

Embedding is a transformation which allows us to find semantically relevant chunks from a catalogue given a query. Through some nearness criteria, you would retrieve "semantically relevant" chunks which along with query would be fed to LLMs and ask them to synthesize the best answer. Vespa docs are very great if you are thinking of building in this space. Retrieval part is independent of synthesis, hence it has its separate leaderboard on huggingface.

https://docs.vespa.ai/en/embedding.html

https://huggingface.co/spaces/mteb/leaderboard

Re: Contextual Retrieval

#59

Post by passion__desire »

valstu wrote: post #6

We're doing something similar. We first chunk the documents based on h1,h2,h3 headings. Then we add headers in the beginning of the chunk as a context. As an imagenary example, instead of one chunk being: The usual dose for adults is one or two 200mg tablets or capsules 3 times a day. It is now something like: # Fever ## Treatment --- The usual dose for adults is one or two 200mg tablets or capsules 3 times a day. Th…

I used to always wonder how do llms know whether a particular long article or audio transcript was written by say Alan Watts. Basically these kind of metadata annotation would be common while preparing training data for Llama models and so on. This could also be reason for the genesis for the argument that ChatGPT got slower in December. That "date" metadata would "inform" ChatGPT to be unhelpful.

Re: Contextual Retrieval

#60

Post by turing_complete »

underlines wrote: post #27

We build a corporate RAG for a government entity. What I've learned so far by applying an experimental A/B testing approach to RAG using RAGAS metrics: - Hybrid Retrieval (semantic + vector) and then LLM based Reranking made no significant change using synthetic eva-questions - HyDE decreased answer quality and retrieval quality severly when measured with RAGAS using synthetic eval-questions (we still have to do a RA…

What do you think of HippoRAG? Did you try it or plan to do?
Post reply on HN