Token savings are great, but I wonder whether the biggest long-term benefit is latency reduction rather then API cost,Faster feedback loops can easily outweigh a few cents saved per request
Code mode yields a 99.2% cost reduction in our systems
41–50 of 77 posts
Re: Code mode yields a 99.2% cost reduction in our systems
#42A maxim I’ve arrived at working w llm every day is “put deterministic things in code, non deterministic in llm”. I do wish the harnesses would be more helpful in this regard. For example I’ve seen tons of scheduled jobs that people wrote in Claude/copilot/etc that could easily have been scripts. They aren’t scripts because the author doesn’t know how to script and they stop at the point the llm does what they want. I…
The way I try to illustrate this to my peers, in the context of automating with llms, is to "do as much of the deterministic work as possible before and after involving an agent". Tbf this is largely a restatement of your comment;
Ie instead of
"hey agent, use the api to get X, then do unpredictable mutation Y to it, and email it to me",
it should be;
"Scheduled task runs code to get X then place it in workspace. Task then starts agent in workspace and is prompted to do mutation Y to it. Post-flight code then sends final product."
The difference here being that the pre/post flight stuff is called deterministically, rather than called by an llm.
Re: Code mode yields a 99.2% cost reduction in our systems
#43A maxim I’ve arrived at working w llm every day is “put deterministic things in code, non deterministic in llm”. I do wish the harnesses would be more helpful in this regard. For example I’ve seen tons of scheduled jobs that people wrote in Claude/copilot/etc that could easily have been scripts. They aren’t scripts because the author doesn’t know how to script and they stop at the point the llm does what they want. I…
Preach!
Dedicated Infrastructure > automation > ad hoc scripts > non deterministic llms from a dependability perspective alone. Using an LLM to do something that prettier could have done is analogous to using a 3d printer to nail up a fence.
Re: Code mode yields a 99.2% cost reduction in our systems
#44Earlier quoted context omitted.
A python script is now more iffy than a LLM going through JSON?
Yeah, of course. The MCP server tools do use Python in the backend but just look at the affordances it provided in the args -- named columns, a fixed match/replace API, a way to specify the range of rows. The LLM is not gonna create a little library with these affordances in an ephemeral REPL it's just gonna YOLO some pandas code that you hope did what you wanted and addressed the right columns, had the right types,…
I feel like I am having a fever dream. How is an LLM that is trained on endless examples of Python, a well defined language as well, going to generate worst code than some random JSON interface?
Re: Code mode yields a 99.2% cost reduction in our systems
#45Cool but isn’t everyone already doing this? Surprised this is what I saw at top of HN this morning.
Re: Code mode yields a 99.2% cost reduction in our systems
#46I am still a bit surprised that this is not more widely used. Normal inference + tool calls is effectively composition via serialized natural language, whereas Code Mode / Programmatic Tool Calling is using purpose built languages (aka programming languages) for composition and concurrency. It somewhat feels obvious to me that the latter is way more token and latency efficient.
In the world of AI hype, everything is to be learned again and we are going to see many many breakthroughs like this. You can see this shift already in the tone of conversation in HN for example. The basics of computer engineering is now novel approaches. We are discovering why programming languages exist, starting with a worst version of Cobal running on a very expensive and probabilistic VM. We will shoehorn every…
to be fair, people are also discovering why vaccines, democracy and human decency are desirable traits; so it's not just programming in the age of LLMs.
There's clearly a paradox that happens when people have to face things they treat as essentially magic because the magic has wiped away the pain and suffering that borne them. So too will be another age when people have to relearn everything the LLMs do for us.
Re: Code mode yields a 99.2% cost reduction in our systems
#47you lose the per-call approvals though. one script hitting ten tools is harder to gate than ten separate calls
yes, that is true. in fact we are thinking a lot about this, not on a toll based, but rather on a role based. giving access to specific tools to specific agents, how that works and evolves, and how that links w the humans using it
makes sense
Re: Code mode yields a 99.2% cost reduction in our systems
#48This quarter’s clankerisms: “Honest” and “vibes”. The tech here is cool; good reminder to push for more scripts in general from the agentic tools.
[deleted]
Re: Code mode yields a 99.2% cost reduction in our systems
#49you lose the per-call approvals though. one script hitting ten tools is harder to gate than ten separate calls
Then the agent runtime should be happening in a sandbox, where policy is enforced by a gateway external to it. Bound the agent's autonomy based on of the affects the agent's actions. Approvals should be made into a contract before the agent runs.
Re: Code mode yields a 99.2% cost reduction in our systems
#50Cool but isn’t everyone already doing this? Surprised this is what I saw at top of HN this morning.
I did not expect it either tbh :D
But I believe it's not that common as it seems, only one actually trying to push this is executor.sh for example