Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Better Models: Worse Tools

lucumr.pocoo.org

Post reply on HN
Print view View original

51–60 of 90 posts

Re: Better Models: Worse Tools

#51

Post by thx67 »

8cvor6j844qw_d6 wrote: post #44
cadamsdotcom wrote: post #34

Earlier quoted context omitted.

I do not need to waste tokens on skills, I use Claude Code hooks. Have a look at the TDD guard at https://codeleash.dev - the scripts/tdd_log.py arguments are pretty specific but it also has guidance in CLAUDE.md and lots of helpful error messages.

May I know when should skills be used over hooks and vice versa?

Hooks are for doing AoP style wrapping of your interactions with the harness. Type /hook on the console see what is available. Have CC analyze your session and suggest converting part of your workflow to a hook, and then have it test it.

Re: Better Models: Worse Tools

#52

Post by big-chungus4 »

A better solution might be not to constrain the generation, but to remove invalid fields from the tool call in the assistant message. So on the next turn, the model receives chat history which contains it's tool call, but without extra arguments. You can do that in OpenAI chat completions / responses, not sure about Anthopic API.

There is still a downside, sometimes the model really wants to include an additional argument for whatever reason it reasoned towards, and it needs the error message to understand that the argument doesn't exist. Otherwise if the argument is manually removed from it's tool call, the model will think that it accidentally left out the argument and start retrying and might go into a loop.

Re: Better Models: Worse Tools

#54

Post by cadamsdotcom »

8cvor6j844qw_d6 wrote: post #44
cadamsdotcom wrote: post #34

Earlier quoted context omitted.

I do not need to waste tokens on skills, I use Claude Code hooks. Have a look at the TDD guard at https://codeleash.dev - the scripts/tdd_log.py arguments are pretty specific but it also has guidance in CLAUDE.md and lots of helpful error messages.

May I know when should skills be used over hooks and vice versa?

Hooks provide determinism.

Hooks can run code.

Hook code can be written in advance by the agent, runs in milliseconds, costs zero tokens, and gives the same result everytime.

Agents live at the boundary of codification; anything codifiable should be codified rather than run through an unpredictable machine. Hence, use hooks when you want determinism & predictability & certainty.

Examples: your stop hook could run tests against the code that’s just been written. Now, if your agent docs also tell your agent that the stop hook will run tests and there’s no need to run tests itself, then it’ll trigger a stop when it’s done instead of running tests itself. Just be sure to change the exit code to 2 and route the test failure output to stderr so Claude Code will show that output to the agent. Because the stop hook will fail over and over until tests pass, you just created a very simple guard that guarantees tests pass before you see the code - your agent can’t stop working without passing tests!

Re: Better Models: Worse Tools

#55

Post by lelanthran »

pyeri wrote: post #50
cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

This maneuver requires you to anticipate all the edge cases or error messages beforehand which is practically not possible in many situations. The moment something unanticipated happens or the model changes its processing logic, the tool call system stops working just like any other deterministic program or tool.

> This maneuver requires you to anticipate all the edge cases or error messages beforehand which is practically not possible in many situations. The moment something unanticipated happens or the model changes its processing logic, the tool call system stops working just like any other deterministic program or tool.

Not all; error messages are part of UX design, and the user error message should always give an error that indicates what the user can do to fix the problem.

If you cannot open a file for writing, don't just return "error: cannot open MyFile.txt", return "MyFile.txt: permission denied" (so user can request additional permissions from whoever), "MyFile.txt: no space left on device" (so user can free up some space), "Myfile.txt: file exists and is a directory" (So user can retry with a different name, or remove the directory, etc).

I think what is happening now is that, with so many of the agent-using pool of devs having never shipped to end-users before, they are surprised that their "program" (the tool) is being used wrong by the end-user (the LLM).

Those of us with battle-scars already expect the user to use it wrong and have learned that it's easier to tell the user how to fix the problem than to ask the user to read the manual/do it the correct way.

Re: Better Models: Worse Tools

#56

Post by pugio »

Pi is my daily driver. I noticed the same phenomenon, and had Claude analyze all my past transcripts for classes of 'edit' error. Built an extension which patches the edit tool to self-heal on the majority of those kinds of calls. It's not 100%, but it cuts down on the rejections quite a bit and saves a few round trips.

EDIT: It's still quite fascinating seeing the kinds of things the models keep trying to do. It almost seems like when a human has slightly off with their nervous system. The conscious brain wants to do one thing, but for some reason the signals aren't getting to the hands correctly.

Re: Better Models: Worse Tools

#58

Post by the_mitsuhiko »

big-chungus4 wrote: post #52

A better solution might be not to constrain the generation, but to remove invalid fields from the tool call in the assistant message. So on the next turn, the model receives chat history which contains it's tool call, but without extra arguments. You can do that in OpenAI chat completions / responses, not sure about Anthopic API. There is still a downside, sometimes the model really wants to include an additional arg…

That kills the KV cache unfortunately. In some models (Gemini) I also doesn’t work because there is a signature on the model messages.

Re: Better Models: Worse Tools

#59

Post by cadamsdotcom »

klntsky wrote: post #45
cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

I've built a library that makes creating rich feedback systems easier, check this out: https://tool2agent.org/

Okay, but I solved this with a print statement.

Re: Better Models: Worse Tools

#60

Post by cadamsdotcom »

pyeri wrote: post #50
cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

This maneuver requires you to anticipate all the edge cases or error messages beforehand which is practically not possible in many situations. The moment something unanticipated happens or the model changes its processing logic, the tool call system stops working just like any other deterministic program or tool.

Are we talking about the same thing?

If the agent uses the to incorrectly, validation fails.

If validation fails for ANY reason, print a message saying “here’s how to use it correctly”.

You don’t need to anticipate every misuse, just validate your inputs.

Post reply on HN