Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Better Models: Worse Tools

lucumr.pocoo.org

Post reply on HN
Print view View original

31–40 of 90 posts

Re: Better Models: Worse Tools

#31

Post by siwatanejo »

cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

So, are you saying that skills are not such a good tool for agents to learn, they still need tool-trial-and-error dance after injecting them? (I'm assuming each tool comes with its own skill.)

Re: Better Models: Worse Tools

#32

Post by fragmede »

resonious wrote: post #14
sestep wrote: post #12

> In case you are curious about Fable: I intentionally did not test it because I was not sure if the classifiers they are running might downgrade me to Opus silently. Is this still a thing? I thought Anthropic walked back the silent downgrades so now all the different domains downgrade non-silently.

Claude Code downgrades loudly but I'm not sure what happens over API or with other harnesses, OpenRouter, etc.

If I send an API call specifying model="Fable", is there a world where returning tokens not from Fable is anything but dishonest?

Re: Better Models: Worse Tools

#33

Post by CompleteSkeptic »

aetherspawn wrote: post #19

Surprised models still output tools as text when for ages we’ve been able to constrain the output at the inference engine level and constrain the model what tools, parameters etc are available Edit: found it, it’s called Grammar-Constrained Decoding (GCD)

constrained decoding tends to make models dumber - this is why it's rarely used

Re: Better Models: Worse Tools

#34

Post by cadamsdotcom »

siwatanejo wrote: post #31
cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

So, are you saying that skills are not such a good tool for agents to learn, they still need tool-trial-and-error dance after injecting them? (I'm assuming each tool comes with its own skill.)

I do not need to waste tokens on skills, I use Claude Code hooks.

Have a look at the TDD guard at https://codeleash.dev - the scripts/tdd_log.py arguments are pretty specific but it also has guidance in CLAUDE.md and lots of helpful error messages.

Re: Better Models: Worse Tools

#35

Post by sdesol »

siwatanejo wrote: post #31
cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

So, are you saying that skills are not such a good tool for agents to learn, they still need tool-trial-and-error dance after injecting them? (I'm assuming each tool comes with its own skill.)

> they still need tool-trial-and-error dance after injecting them?

It honestly depends on the model. For my pi-brains extension for pi

https://github.com/gitsense/pi-brains

I've found after the first hook injection they get it, but there are occasions it can forget, but since everything is driven by hooks, you can inject as often as needed.

The issue with skills is, they are a one time thing, so you really can't use skills to correct haviorial issues.

Re: Better Models: Worse Tools

#36

Post by jdiff »

aetherspawn wrote: post #19

Surprised models still output tools as text when for ages we’ve been able to constrain the output at the inference engine level and constrain the model what tools, parameters etc are available Edit: found it, it’s called Grammar-Constrained Decoding (GCD)

I imagine the challenge comes from recognizing that your model is trying to call a tool before it actually has and only constraining output then. Running a separate pass for an optionally-empty list of tools afterwards may work, but maybe constraining its output like that causes many spurious tool calls.

Re: Better Models: Worse Tools

#37

Post by hsaliak »

It's been clear for some time that model tool calling is heavily fit to a few common patterns, it's unsurprising that a tool call that looks the same or has the same name, but works differently, is falling back to priors and causing problems.

Things are not quite AGI yet; which is why people are now saying that intelligence is the harness + model, because the harness makes up for limitations in generalization.

Re: Better Models: Worse Tools

#38

Post by miketery »

aetherspawn wrote: post #19

Surprised models still output tools as text when for ages we’ve been able to constrain the output at the inference engine level and constrain the model what tools, parameters etc are available Edit: found it, it’s called Grammar-Constrained Decoding (GCD)

Some model providers when using json_schema: true (eg. with_structured_output), it does constrain the output.

Re: Better Models: Worse Tools

#39

Post by psadri »

cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

This will cause an extra round trip to the LLM. Which means more $ spent.

Re: Better Models: Worse Tools

#40

Post by sdesol »

psadri wrote: post #39
cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

This will cause an extra round trip to the LLM. Which means more $ spent.

Better a round trip than bad/incorrect results. Also, the cache should kick in so the cost will be minimal.
Post reply on HN