Viewing profile — shahules

My team works on automatic environment generation for RL post-training. One of our projects is using coding agents to build web clones for BUAs/CUAs. We tested Gemini, Claude Code,…

Nice, their training recipe seems unique.

After doing few experiments, I think that having Agents work on browser for all tasks wouldn't be best due to many factors like token cost, safety, etc. But browser/computer can be…

There are few agents like browser-use, skyvern etc that may provide this capability.

[dead]

We’re the team at Vibrant Labs (W24). We’ve been building envs for browser agents and quickly realized that existing benchmarks in this space didn’t capture the primary failure mod…

Most current web agent benchmarks focus on single-tab tasks (e.g., 'go to Gmail and star this email'). We found that frontier models that score highly on those tasks (like in WebAr…

It's an interesting article and I agree with some points you brought up here. But here are some of them to which I don't agree to 1. Evals are used throughout the article in the se…

Can't agree with you more my friend. Another point on a philosophical level is efficiency or optimization in life, which always focuses on tangible aspects and ignores the greater …

Deepeval also uses Ragas underneath. They initially took a different approach by allowing uses to formulate test cases but we were focusing on RAGs only and creating metrics and fe…

Hey, I would recommend checking out our PRs. There would be PRs that have modified some of the prompts to better suit Mixtral.

I think it's true for any early-stage library/framework. The tradeoff is then you will have to keep maintaining it, add support to other LLMs if you change LLMs, etc. Then in the e…

Hey, DeepEval is interesting. What do you mean by "evaluating any LLMs"?

1) Horizondal expansion and support are core to every framework/library. This won't make us a competitor to LC, we actually use langchain-core to support many of these like support…

Yes, this is an interesting challenge we are also excited about.

Hey, this is a highly requested feature. We will be implementing it soon. Something like a rough estimate is what we are planning to do.

Thanks for your feedback. We have tested Ragas on alternatives like Claude, Mixtral, Gemini, etc. Although we support all LLMs supported by Langchain, sadly many of the OSS models …

Thank you.

Thank you for asking this question. To answer this question, I will explain two directions of Ragas. The first one is the horizontal expansion of the library which involves feature…

Hey, thanks for trying out Ragas. As an open-source library, we are continuously improving from the feedback from the community which I see as our primary strength. I am sure that …

Ragas is an open-source library for evaluating and testing RAG and other LLM applications. Github: https://docs.ragas.io/en/stable/ , docs: https://docs.ragas.io/ . Ragas provides …

Hey, The cost depends on how many samples you would want to generate. We are finetuning smaller OSS models to tackle the cost factor of synthetic data generation which we will open…