We’re entering the era of AI trained by previous generations’ slop. It’s not surprising that it’s sloppy.
Better Models: Worse Tools
41–50 of 90 posts
Re: Better Models: Worse Tools
#42This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…
I've been trying to push for this perspective about the error messages of jj vcs. There's some push back from people that don't perceive that making tools work well with LLMs is also making tools work well with humans. (Obviously there's more nuance to the arguments than this one sided perspective).
Re: Better Models: Worse Tools
#43closed source harness + RL fine tuning on customer prompts on said harness is becoming a kind of economic moat (?)
Re: Better Models: Worse Tools
#44Earlier quoted context omitted.
So, are you saying that skills are not such a good tool for agents to learn, they still need tool-trial-and-error dance after injecting them? (I'm assuming each tool comes with its own skill.)
I do not need to waste tokens on skills, I use Claude Code hooks. Have a look at the TDD guard at https://codeleash.dev - the scripts/tdd_log.py arguments are pretty specific but it also has guidance in CLAUDE.md and lots of helpful error messages.
May I know when should skills be used over hooks and vice versa?
Re: Better Models: Worse Tools
#45This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…
I've built a library that makes creating rich feedback systems easier, check this out:
Re: Better Models: Worse Tools
#46I suspect this isn't a malfunction, but rather a deliberate measure designed to counter so-called distillation attacks.
Re: Better Models: Worse Tools
#47[flagged]
Re: Better Models: Worse Tools
#48This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…
same findings here, it'll doom loop without the proper error messaging. really expensive without error logging that gets propagated back to the agent
Re: Better Models: Worse Tools
#49This has been the case since the early days. Aider had a bunch of code to be very forgiving with formatting of tool calls (file editing in particular at first). It's just the nature of the beast. It surprises me that Pi doesn't have a lot of this kind of stuff built in too
Re: Better Models: Worse Tools
#50This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…
This maneuver requires you to anticipate all the edge cases or error messages beforehand which is practically not possible in many situations. The moment something unanticipated happens or the model changes its processing logic, the tool call system stops working just like any other deterministic program or tool.