Can someone explain simply how these benchmarks work?
What exactly is a "failure rate" and how is it computed?
Contextual Retrieval
31–40 of 73 posts
Re: Contextual Retrieval
#32I've been wondering for a while if having ElasticSearch as just another function to call might be interesting. If the LLM can just generate queries it's an easy deployment.
Re: Contextual Retrieval
#33Earlier quoted context omitted.
I think the innovation is using caching as so to make the cost of the approach manageable. The way they implemented it is that each time you create a chunk, you ask the llm to create an atomic chunk from the whole context. You need to do this for all tens of thousands of chunks in your data. This costs a lot. By caching the documents, you can spare costs
You could also just save the first outputted atomic chunk and store it then re-use it each time yourself. Easier and more consistent.
I don't understand how that helps here. They're not regenerating each chunk every time, this is about caching the state after running a large doc through a model. You can only do this kind of thing if you have access to the model itself, or it's provided by the API you use.
Re: Contextual Retrieval
#34I just took the time to read through all source code and docs. Nice ideas. I like to experiment with LLMs running on my local computer so I will probably convert this example to use the light weight Python library Rank-BM25 instead of Elastic Search, and a long context model running on Ollama. I wouldn’t have prompt caching though.
This example is well written and documented, easy to understand. Well done.
Re: Contextual Retrieval
#35> If your knowledge base is smaller than 200,000 tokens (about 500 pages of material)
I would prefer that anthropic just release their tokeniser so we don't have to make guesses.
Re: Contextual Retrieval
#36We build a corporate RAG for a government entity. What I've learned so far by applying an experimental A/B testing approach to RAG using RAGAS metrics: - Hybrid Retrieval (semantic + vector) and then LLM based Reranking made no significant change using synthetic eva-questions - HyDE decreased answer quality and retrieval quality severly when measured with RAGAS using synthetic eval-questions (we still have to do a RA…
Could you explain or link to explanations of all of the acronyms you’ve used in your comment?
Re: Contextual Retrieval
#37My favorite thing about this is the way it takes advantage of prompt caching. That's priced at around 1/10th of what the prompts would normally cost if they weren't cached, which means that tricks like this (running every single chunk against a full copy of the original document) become feasible where previously they wouldn't have financially made sense. I bet there are all sorts of other neat tricks like this which…
I follow your blog and read almost everything you write about Llms. Just curious (if you havent already written about it somewhere and I missed it), how much do you spend monthly, exploring all the various Llms and their features? (I think its a useful context for having a grasp of how much I would have to spend to keep up to date with the models out there and the latest features)
Most months I spend less than $10 total across the OpenAI, Anthropic and Google APIs - for the kind of stuff I do I’m just not racking up really high token counts.
I spend $20/month on ChatGPT plus and $20/month on Claude Pro. I get GitHub Copilot for free as an open source maintainer.
Re: Contextual Retrieval
#38I don't know anything about AI but I've always wished I could just upload a bunch of documents/books and the AI would perform some basic keyword searches to figure out what is relevant, then auto include that in the prompt.
It would help if you tried Notebooklm by Google. It does this, you can upload a document/PDF whatever, and ask questions. The model replies to you giving also a reference to your material
+1 Google’s NotebookLM is amazing. In addition to the functionality you mention, I tried loading the PDF for my entire Practical AI Programming with Clojure book and had it generate an 8 minute podcast that was very nuanced - to be honest, it seriously blew my mind how well it works. Here is a link to the audio file it automatically generated https://markwatson.com/audio/AIClojureBook.wav
NotebookLM is currently free to use and was so good I almost immediately started paying Google $20 a month to get access to their pro version of Gemini.
I still think the Groq APIs for open weight models are the best value for the money, but the way OpenAI, Google, Anthropic, etc. are productizing LLMs is very impressive.
Re: Contextual Retrieval
#39Earlier quoted context omitted.
Graph RAG is very cool and outstanding at filling some niches. IIRC, Perplexity's actual search is just BM25 (based a lex fridman interview of the founder).
Do you have the link and the time in the video where he mentions it?
Re: Contextual Retrieval
#40We build a corporate RAG for a government entity. What I've learned so far by applying an experimental A/B testing approach to RAG using RAGAS metrics: - Hybrid Retrieval (semantic + vector) and then LLM based Reranking made no significant change using synthetic eva-questions - HyDE decreased answer quality and retrieval quality severly when measured with RAGAS using synthetic eval-questions (we still have to do a RA…
Could you explain or link to explanations of all of the acronyms you’ve used in your comment?
It makes me chuckle a bit to see this kind of request in a tech forum, particularly when discussing advanced LLM-related topics.
This is akin to a HN comment asking someone to search the Internet for something on their behalf, while discussing search engine algorithms!