Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Tonic Validate Metrics – an open-source RAG evaluation metrics package

github.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Show HN: Tonic Validate Metrics – an open-source RAG evaluation metrics package

#11

Post by Ephil012 »

bratton88 wrote: post #9
Ephil012 wrote: post #2

Hi all, if anyone has any questions about the open source library, Joe and I will be around today to answer them.

no questions - just want to say thanks for sharing!

No problem! If you have any questions in the future, feel free to open a issue on Github. Also, we got a free UI for visualizing the metric logs (no paid tier or anything like that at the moment) that we'd love people to try out. https://validate.tonic.ai

Re: Show HN: Tonic Validate Metrics – an open-source RAG evaluation metrics package

#13

Post by Ephil012 »

capybara wrote: post #12

This is cool. What are your plans for supporting and building upon this going forward?

Right now, we are planning on adding more metrics in the future. We also have considered adding the ability of using vector embeddings instead of LLMs for analyzing the responses (which should bring some efficiency gains). Plus, we might add the ability to analyze the distribution of vectors in your vector db to compare it to the distribution of vectors for user queries.

We also have a UI for visualizing all of these metrics that is free at the moment with no paid tier at all. We are planning to add more features into that UI for better visualizations along with improving the ability to keep track of different versions of your LLM system for comparison.

Re: Show HN: Tonic Validate Metrics – an open-source RAG evaluation metrics package

#15

Post by d4rkp4ttern »

Related — are there any good end to end benchmark datasets for RAG? End to end meaning not just (context, question, answer) tuples (which ignores retrieval) but (Document , question, answer). I know NQ (Natural Questions) is one such dataset:

https://ai.google.com/research/NaturalQuestions

But I do t see this dataset mentioned much in RAG discussions.

Re: Show HN: Tonic Validate Metrics – an open-source RAG evaluation metrics package

#16

Post by joewferrara »

d4rkp4ttern wrote: post #15

Related — are there any good end to end benchmark datasets for RAG? End to end meaning not just (context, question, answer) tuples (which ignores retrieval) but (Document , question, answer). I know NQ (Natural Questions) is one such dataset: https://ai.google.com/research/NaturalQuestions But I do t see this dataset mentioned much in RAG discussions.

It's true that there are not a lot of datasets for benchmarking RAG. RAG applications are so tailored to the specific data being used as well as the use case, that a benchmark dataset is not useful across different RAG applications. The data used for a RAG application could be slack messages, technical documentation, insurance policies, internal company microsoft word documents, or a combination of these. For each of these different data source examples, the benchmark dataset would need to be very different.

We recommend that when building a RAG application, the developers build a benchmark dataset specifically tailored to the data being used for the RAG application, and the use case of the RAG application.

Re: Show HN: Tonic Validate Metrics – an open-source RAG evaluation metrics package

#17

Post by d4rkp4ttern »

joewferrara wrote: post #16
d4rkp4ttern wrote: post #15

Related — are there any good end to end benchmark datasets for RAG? End to end meaning not just (context, question, answer) tuples (which ignores retrieval) but (Document , question, answer). I know NQ (Natural Questions) is one such dataset: https://ai.google.com/research/NaturalQuestions But I do t see this dataset mentioned much in RAG discussions.

It's true that there are not a lot of datasets for benchmarking RAG. RAG applications are so tailored to the specific data being used as well as the use case, that a benchmark dataset is not useful across different RAG applications. The data used for a RAG application could be slack messages, technical documentation, insurance policies, internal company microsoft word documents, or a combination of these. For each of…

Well we could say the same about question answering or Info Retrieval, and even LLMs , yet there are plenty of benchmarks for these. The point of a benchmark is not that it covers all use cases, but that it is agreed upon to be a meaningful way to compare different approaches. I suppose I should dig into RAG papers accepted into ICLR/ICML/NeurIPS and look at their experiments section.
Post reply on HN