Hello so i created agent detective tool, this tool answer question: which of the agents broke in system/workflow. I would really appreciate if community would try it out. https://github.com/Thomeras/agent_detective
New open sourced eval tool for building agents
1–5 of 5 posts
Re: New open sourced eval tool for building agents
#2how do you define quality broke? for me it broke when nothing broke. score from llm moved, >= 4 filter after it meant something else. gating on exit code would have passed the whole time.
Re: New open sourced eval tool for building agents
#3how do you define quality broke? for me it broke when nothing broke. score from llm moved, >= 4 filter after it meant something else. gating on exit code would have passed the whole time.
Yeah thats the real failure mode and it took me a while to name it too. A threshold isnt a test, it is a coordinate system that assumes a fixed origin. When the judge moves the filter keeps passing but it is selecting a different population and nothing throws because no invariant anywhere names that population.
So i define it as: quality broke when an observable moved without a change that authorizes it.
Thanks for trying it out i am really happy for feed back :D
Re: New open sourced eval tool for building agents
#4how do you define quality broke? for me it broke when nothing broke. score from llm moved, >= 4 filter after it meant something else. gating on exit code would have passed the whole time.
Yeah thats the real failure mode and it took me a while to name it too. A threshold isnt a test, it is a coordinate system that assumes a fixed origin. When the judge moves the filter keeps passing but it is selecting a different population and nothing throws because no invariant anywhere names that population. So i define it as: quality broke when an observable moved without a change that authorizes it. Thanks for t…
labels here too, about 100 articles with the band they should get. what i still cannot see is a move with no change to blame.
Re: New open sourced eval tool for building agents
#5Earlier quoted context omitted.
Yeah thats the real failure mode and it took me a while to name it too. A threshold isnt a test, it is a coordinate system that assumes a fixed origin. When the judge moves the filter keeps passing but it is selecting a different population and nothing throws because no invariant anywhere names that population. So i define it as: quality broke when an observable moved without a change that authorizes it. Thanks for t…
labels here too, about 100 articles with the band they should get. what i still cannot see is a move with no change to blame.
[dead]