71–80 of 96 posts
We've using auto-research for harnesses and it's surprisingly powerful. It's amazing how many problems are easy to spot and fix from traces. Few key things were required to get it working well: 1) let it read a heap of prod traces to spot real issues. 2) let it write it's own tools (example "loading context" goes from 20k tokens across 15 tool calls to 800 tokens and 1 call to session_context tool). 3) you need evals…
I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…
It only tries to solve the „quality“ aspect of that equation tho, at least for the code output.
But I think for actual evals, some heuristics about a coding agent session are needed. But since the input tasks are always different, it’s hard to make any KPI actually comparable.
Did the agent needing 3 rounds of feedback on a big task perform worse than the one that needed none, but only worked on a small change?
All this work for something we already solved 40+ years ago with ontologies and expert systems
Earlier quoted context omitted.
apologies, I should have clarified the 'better' claim - same task result (passed) - finished faster - fewer tokens, less cost - fewer requests for inference - fewer tool calls - less peak RAM
Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable? As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible. My CLAUDE.md, memories, and skills are all about either (a) adherence to project standa…
. When you want those instructions to be followed, mentiont them in your prompt. "Test this following procedures on docs/test.md". (I've found I don't have to do that extra instruction in al cases depending on model)
I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…
I literally hold a "retro" with my agent after a session. > Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in. It…
I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…
Since the version of opus has changed over time, so have the evaluations. I was going to constrain it, but since they edit the same files, I can alter those (which it uses as summaries and starting points).
I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…
I literally hold a "retro" with my agent after a session. > Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in. It…
"did you notice any inconsistencies or opportunities to clarify/consolidate parameters or functionality while you were working?"
Another strategy that has worked well for my MCP server is noticing a sequence of failed tool-calls, then returning a response that contains a request for the agent to provide feedback via a skill.