Congratulations on the launch!
Personally would love to see a rough estimates of the expected number of requests and tokens required to run tasks like synthetic data generation for different amounts of data. Though this is likely highly variable, would like to have a loose idea of possible incurred costs and execution time.
Show HN: Ragas – Open-source library for evaluating RAG pipelines
11–20 of 27 posts
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#12Congratulations on the launch! Personally would love to see a rough estimates of the expected number of requests and tokens required to run tasks like synthetic data generation for different amounts of data. Though this is likely highly variable, would like to have a loose idea of possible incurred costs and execution time.
Hey, this is a highly requested feature. We will be implementing it soon. Something like a rough estimate is what we are planning to do.
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#13Congratulations on the launch of Ragas! This looks like an incredibly valuable tool for the LLM community. As the library continues to evolve, it will be interesting to see how it adapts to handle the growing diversity of LLM architectures and use cases.
Yes, this is an interesting challenge we are also excited about.
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#14congrats on launching! i think my continuing struggle with looking at Ragas as a company/library rather than a very successful mental model is that the core of it is like 8 metrics ( https://github.com/explodinggradients/ragas/tree/main/src/ra... ) that are each 1-200 LOC. i can inline that easily in my app and retain full control, or model that in langchain or haystack or whatever. why is Ragas a library and a compa…
Thank you for asking this question. To answer this question, I will explain two directions of Ragas. The first one is the horizontal expansion of the library which involves features like - Giving you the ability to use any LLMs instantly without any hassle - Asynchronous evaluations, integrations with tracing tools, etc - Automatic support to adapt metrics to any language The second is vertical expansion or adding mo…
cool cool. so 1) will be a direct langchain competitor, and 2) is net new territory?
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#15Earlier quoted context omitted.
Thank you for asking this question. To answer this question, I will explain two directions of Ragas. The first one is the horizontal expansion of the library which involves features like - Giving you the ability to use any LLMs instantly without any hassle - Asynchronous evaluations, integrations with tracing tools, etc - Automatic support to adapt metrics to any language The second is vertical expansion or adding mo…
cool cool. so 1) will be a direct langchain competitor, and 2) is net new territory?
1) Horizondal expansion and support are core to every framework/library. This won't make us a competitor to LC, we actually use langchain-core to support many of these like supporting different LLMs.
2) We operate in a layer underneath the stack of evals and testing because we want to solve the problem from the ground up rather than building a fancy tracing tool, which comes later in the stack.
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#16Also check out DeepEval... our team has been using it for a while, and it's been working well for us because we can evaluate any LLMs, something this library doesn't seem to support (https://github.com/confident-ai/deepeval).
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#17Also check out DeepEval... our team has been using it for a while, and it's been working well for us because we can evaluate any LLMs, something this library doesn't seem to support ( https://github.com/confident-ai/deepeval ).
Hey, DeepEval is interesting. What do you mean by "evaluating any LLMs"?
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#18Great product and great progress.
The first step to build rage is always to evaluate.
Except all the current evaluations, cost and perf should also be part of evaluations
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#19congrats on launching! i think my continuing struggle with looking at Ragas as a company/library rather than a very successful mental model is that the core of it is like 8 metrics ( https://github.com/explodinggradients/ragas/tree/main/src/ra... ) that are each 1-200 LOC. i can inline that easily in my app and retain full control, or model that in langchain or haystack or whatever. why is Ragas a library and a compa…
> that are each 1-200 LOC. i can inline that easily in my app and retain full control
Isn't that true of most of langchain as well though?
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#20Congratulations on the launch! I was unable to use this library: I was trying to evaluate different non-OpenAI models and it consistently failed due to malformed JSONs coming from the model. Any thoughts about using different models? Is this just a langchain limitation?
Thanks for your feedback. We have tested Ragas on alternatives like Claude, Mixtral, Gemini, etc. Although we support all LLMs supported by Langchain, sadly many of the OSS models out of the box aren't capable of generating JSON-type output which is important for us to ensure reproducibility.
Any tips for Mixtral? That’s what we tried