Better Models: Worse Tools

> In case you are curious about Fable: I intentionally did not test it because I was not sure if the classifiers they are running might downgrade me to Opus silently.

Is this still a thing? I thought Anthropic walked back the silent downgrades so now all the different domains downgrade non-silently.

Claude Code downgrades loudly but I'm not sure what happens over API or with other harnesses, OpenRouter, etc.

> [...] newer Claude models sometimes call Pi’s edit tool with extra, invented fields in the nested edits[] array

> My strongest hypothesis is that this is not random deterioration but a training artifact. [...] Anthropic’s own client appears to expect and accept a fair amount of slop and repairs it, mostly silently

> If reinforcement learning happens in a harness like that, or a simulation of one, then slightly malformed tool calls can still complete the task and receive reward.

> Worse, the model may become very strongly adapted to the canonical Claude Code edit tool shape.

> Tool schemas are somewhere in the distribution and some shapes are close to what the model saw during post-training and some are far away.

Great article.

Interesting root cause hypothesis. Couldn't one simply strip the slop-handling from the RL env's harness to avoid this though?

I do agree on the walled garden being built here. Proprietary frontier models performing best in proprietary harnesses makes sense for Anthropic's interests.

The skills are very readable too, so you win a nice documentation for free. At the very least it's human readable machine instructions.

Hey, an article right up my alley! AI infrastructure/tools engineer here (hic-ai.com); my flagship product, HIC Mouse, is a precision-editing system for coding agents designed to work across a wide array of models and harnesses. Mouse provides 11 tools exposed via MCP for read-, find-, and edit-operations, using a coordinate-based schema (as well as exact and multiple string replacement), a Dialog Box inspect/refine/save/cancel changes functionality controlled by the agent to force staging and review of multi-operation or large edits before changes are written to disk, and extensive agent guidance mechanisms or guardrails to help the agent realize if it's about to do something potentially destructive or overly verbose.

I definitely think models may be trained to use particular popular harnesses or expect certain fields in the editing-tool or other tool schemas. Rather than trying to conform to (or force) one particular format, my approach instead is to design flexibly enough to handle a wide array of possible inputs and tool calls, but that also help the agent recover whenever its tool calls truly can't be salvaged and have to return etrors, and to auto-normalize results whenever reasonable to do so. It really does make a very dramatic difference (I wouldn't have bothered to launch if I thought it wasn't a meaningful advance) but anyway, just wanted to share my perspective given that I live and breathe this problem all day, every day.