61–70 of 131 posts
> The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6's 17% strict pass rate in the original paper. a 41% improvement is not much higher? come on that's just doomer
Earlier quoted context omitted.
I think they neutered Fable. When it first came out it was indeed revolutionary. But what we have today, is not what we had before the ban.
Noticed that too. I wonder if these things just degrade over time, perhaps with the way it writes memories about my project as it goes
Edit: FWIW the paper the post quoted has repositories as slop baseline https://arxiv.org/html/2603.24755v1#S4.SS2
Nice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code. It's especially relevant now that models are good enough to solve ~most point-in-time problems. Some relevant but disconnected thoughts: - deterministic scores are so nice - wh…
> another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system State space of a system AND the way to make it accessible / visible to a model. Many times a model can work magic if it can "see" the state of a system in a way that suits it. That's why sometimes having a cli added to the environment seems like such a big unlock. Because that cli usually takes a comple…
I think the problem with just encoding a whole plan in a single markdown file is that it gets polluted really quickly (agents can stop adhering to instructions to keep it clean and conform to a specific structure), which makes it harder for agents to see which parts of the plan they should focus on. As such, I've reached a similar conclusion about giving the agent access to CLI tools to help them deal with this. To try to mitigate this, I've been getting Claude to develop for me a CLI tool that:
1. Scaffolds reusable, structured plan templates and a reusable workflow that structures how to tackle the plan, step by step.
2. Validates that the plan files still conform to the correct, parseable structure.
3. Parses and evaluates the plan files, to determine what the current state is and what the next valid transitions and states are according to the workflow, like a state machine, and outputs instructions and reminders for agents as to what they should do at each step of the workflow.
So far, I've been dogfooding the tool and it seems promising: I can leave Claude running for longer and it doesn't drift as much. However I haven't ran any benchmarks yet and I'm still not entirely happy with the state of the codebase ( https://github.com/nothingnesses/agent-scaffold ), so take this with a grain of salt.
That said, I'm also bullish on using agents with formal methods and proofs. Type checkers and compile-time checking in general are great because they surface errors early and with great specificity. So if you can encode your specifications with, e.g. dependent types, you can use the type-checker as a way to steer the agent when it goes wrong and gets off-track.
I’ve used SCB as part of my assessment of agent skills (superpowers, GSD etc) https://orcabot.com/labs/do-skills-improve-coding-agent-accu...
There is a small but growing community on discord for discussing SCB so if interested please join https://discord.gg/BrC4BA9sVj
Many people though are going to read the headline figures and think it means - say - Opus 5 is only a quarter the strength of a human coder.
What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.
> The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6's 17% strict pass rate in the original paper. a 41% improvement is not much higher? come on that's just doomer
If you tell me that 76% of your code is dogshit but that's it's a massive improvement over your previous 83%, I'm firing you.
Is this bad? I have McCabe complexity switched on in Ruff and find it a handy watermark for when something should be broken up into smaller, individually testable callables. Five times as many callables could make for much more readable and testable code.
To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.