Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Better Models: Worse Tools

lucumr.pocoo.org

Post reply on HN
Print view View original

61–70 of 90 posts

Re: Better Models: Worse Tools

#61

Post by mrbungie »

pyeri wrote: post #50
cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

This maneuver requires you to anticipate all the edge cases or error messages beforehand which is practically not possible in many situations. The moment something unanticipated happens or the model changes its processing logic, the tool call system stops working just like any other deterministic program or tool.

Just add a --verbose flag that shows the stacktrace when there is an error. Then add a footer message when an error appears in non-verbose mode that invites the user/agent to use --verbose to get the full picture.

It obviously may end up in thousands of tokens burned through though (you can also fix that adding different levels of verbosity), but hopefully errors are not common.

Re: Better Models: Worse Tools

#62

Post by Bolwin »

siwatanejo wrote: post #31
cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

So, are you saying that skills are not such a good tool for agents to learn, they still need tool-trial-and-error dance after injecting them? (I'm assuming each tool comes with its own skill.)

Tools come with a tool description in json schema format, but yes your point stands, it is not enough for opus 4.8 which I've also noticed having tool call issues.

Re: Better Models: Worse Tools

#63

Post by embedding-shape »

psadri wrote: post #39
cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

This will cause an extra round trip to the LLM. Which means more $ spent.

So? What alternative do you suggest? Let the LLM get it wrong forever? Remove the tool? Automatically try to patch the syntax?

Almost no "solutions" in engineering/programming comes for free, one way or another, it's all a balancing act between different solutions with different tradeoffs. In this case, another request/response seems preferable to the other tradeoffs.

Re: Better Models: Worse Tools

#64

Post by klntsky »

cadamsdotcom wrote: post #59
klntsky wrote: post #45

Earlier quoted context omitted.

I've built a library that makes creating rich feedback systems easier, check this out: https://tool2agent.org/

Okay, but I solved this with a print statement.

My assumption was that it is often not convenient if you have a lot of logic. I used it internally, and the complexity of my use cases was barely enough to justify it too. But I've seen systems where it would definitely be a value unlock if I had to integrate LLM chatbots into them

Re: Better Models: Worse Tools

#65

Post by bazodedo »

Once models get better, we could avoid paying for a cache read on edit or write calls, and have the model assume they succeeded and not interrupt the stream to get output. We can then just parse the output and once we encounter such a silent toolcall execute it. With high probability its correct (glm in pi for me had 95% tool call success rate) and we can continue, else rewind. As a workaround, you dont want to use the provider feature that interrupts the stream after a tool call, but instead parse the reasoning. I tried this in pi and it kind of worked, but the model got confused about whether edits had been applied and in several runs either double checked or used the bash tool instead, negating any possible benefits.

Re: Better Models: Worse Tools

#66

Post by euiq »

cadamsdotcom wrote: post #26

This is easily solved with good error messages. Claude always gets the syntax wrong on my tool calls. So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool . The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson. To d…

Pi already emits good errors messages; I always see Claude Opus 4.8 correct itself in its next attempt when it gets a tool call wrong.

Re: Better Models: Worse Tools

#68

Post by galaxyLogic »

LLM can write programs in any programming language it knows about. So how about askinng it to write a shell-program that does the tool-calls on the client?

You might want to run in some kind of sandbox to prevent the LLM from taking over the world, security is an isssue. But apart from that why not make the LLM write shell-programs instead of relying on JSON etc. ? Shell-scripting is the language for controlling the OS.

Re: Better Models: Worse Tools

#69

Post by dragonwriter »

galaxyLogic wrote: post #68

LLM can write programs in any programming language it knows about. So how about askinng it to write a shell-program that does the tool-calls on the client? You might want to run in some kind of sandbox to prevent the LLM from taking over the world, security is an isssue. But apart from that why not make the LLM write shell-programs instead of relying on JSON etc. ? Shell-scripting is the language for controlling the…

Not always shell—Python, often a subset, is common—but a single sandboxed coding-running tool as the way to run all of the other things that would otherwise be their own top-level tools themselves seems an increasingly common approach.

Re: Better Models: Worse Tools

#70

Post by jeffreygoesto »

lelanthran wrote: post #55
pyeri wrote: post #50

Earlier quoted context omitted.

This maneuver requires you to anticipate all the edge cases or error messages beforehand which is practically not possible in many situations. The moment something unanticipated happens or the model changes its processing logic, the tool call system stops working just like any other deterministic program or tool.

> This maneuver requires you to anticipate all the edge cases or error messages beforehand which is practically not possible in many situations. The moment something unanticipated happens or the model changes its processing logic, the tool call system stops working just like any other deterministic program or tool. Not all; error messages are part of UX design, and the user error message should always give an error t…

So much this. I tell my juniors: To a beginner programmer, errors are 'the end'. They feel they did their best, it is not their fault and that is the error message they print. Experienced programmers know the user struggle, for them an error message is 'a beginning'. The first step of the user striving to solve the problem. They gave that command and they did not give it to fail. They (the users) still want to teach their goal.

Pro tip: Don't just print the return code, also print the call and it's arguments that failed, even without a stack trace.

Post reply on HN