This is nice, we've got more Open Source LLM Evaluation Libraries coming in more often.
We're using DeepEval (https://github.com/confident-ai/deepeval) currently. How is this different from that?
Show HN: Ragas – Open-source library for evaluating RAG pipelines
21–27 of 27 posts
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#22Congratulations on the launch! I was unable to use this library: I was trying to evaluate different non-OpenAI models and it consistently failed due to malformed JSONs coming from the model. Any thoughts about using different models? Is this just a langchain limitation?
Checkout this instead: https://github.com/confident-ai/deepeval
Also has native ragas implementation but supports all models.
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#23congrats on launching! i think my continuing struggle with looking at Ragas as a company/library rather than a very successful mental model is that the core of it is like 8 metrics ( https://github.com/explodinggradients/ragas/tree/main/src/ra... ) that are each 1-200 LOC. i can inline that easily in my app and retain full control, or model that in langchain or haystack or whatever. why is Ragas a library and a compa…
> that are each 1-200 LOC. i can inline that easily in my app and retain full control Isn't that true of most of langchain as well though?
I think it's true for any early-stage library/framework. The tradeoff is then you will have to keep maintaining it, add support to other LLMs if you change LLMs, etc. Then in the end OSS will be far ahead because by that time it will have smoothened its rough edges.
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#24Earlier quoted context omitted.
Thanks for your feedback. We have tested Ragas on alternatives like Claude, Mixtral, Gemini, etc. Although we support all LLMs supported by Langchain, sadly many of the OSS models out of the box aren't capable of generating JSON-type output which is important for us to ensure reproducibility.
Any tips for Mixtral? That’s what we tried
Hey, I would recommend checking out our PRs. There would be PRs that have modified some of the prompts to better suit Mixtral.
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#25This is nice, we've got more Open Source LLM Evaluation Libraries coming in more often. We're using DeepEval ( https://github.com/confident-ai/deepeval ) currently. How is this different from that?
Deepeval also uses Ragas underneath. They initially took a different approach by allowing uses to formulate test cases but we were focusing on RAGs only and creating metrics and features like synthetic test data generation for it. Now that we are doing good in the RAG category, we also want to expand to solve the greater challenge.
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#26Great product and great progress. The first step to build rage is always to evaluate. Except all the current evaluations, cost and perf should also be part of evaluations
could you elaborate on what you mean by perf?
cost we'll add soon
Re: Show HN: Ragas – Open-source library for evaluating RAG pipelines
#27Earlier quoted context omitted.
> that are each 1-200 LOC. i can inline that easily in my app and retain full control Isn't that true of most of langchain as well though?
I think it's true for any early-stage library/framework. The tradeoff is then you will have to keep maintaining it, add support to other LLMs if you change LLMs, etc. Then in the end OSS will be far ahead because by that time it will have smoothened its rough edges.
Or OSS will be going in a different direction then what you need, so if you are using it you'll either be stuck on an old version or you have to keep fighting around it. ML libraries in particular have this annoying habit of not being very backwards compatible over more than 2-3 years.