Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code mode yields a 99.2% cost reduction in our systems

agent-swarm.dev

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Code mode yields a 99.2% cost reduction in our systems

#51

Post by firasd »

shireboy wrote: post #5

A maxim I’ve arrived at working w llm every day is “put deterministic things in code, non deterministic in llm”. I do wish the harnesses would be more helpful in this regard. For example I’ve seen tons of scheduled jobs that people wrote in Claude/copilot/etc that could easily have been scripts. They aren’t scripts because the author doesn’t know how to script and they stop at the point the llm does what they want. I…

An absurd example is how harnesses like Codex, Claude code have no 'cut/paste this range of chars' affordance so if you want to move a block of code the LLM galaxy brain rewrites it in token space in the new file.

It ~can~ be done using standard bash utils but there just isn't a kind of standard pattern for 'move this text block via CLI tools' so the LLM and human both overlook it

Re: Code mode yields a 99.2% cost reduction in our systems

#52

Post by tarasyarema »

firasd wrote: post #38
tarasyarema wrote: post #22

Earlier quoted context omitted.

we have a way to see created scripts in a dashboard, and in fact they are not ad-hoc only, i.e. it can create "saved scripts" that it can re-use, iterate and so on. Also not exec happens in a sandboxed env, which means it might not have access to the fs by default (it has to agent-fs, which is a layer we built on top of s3 for agents)

Yeah I guess the dichotomy between 'tools' and 'code mode' isn't really as clear especially if the tools themselves are intelligent and truncate long results and have start_range, end_range type arguments. So the gist is about 'what part of the output needs to reach the LLM' and a 'code mode' approach can help orchestrate that just like good tools can

exactly, I love how the cloudflare article puts it in fact

Re: Code mode yields a 99.2% cost reduction in our systems

#53

Post by rmsaksida »

tarasyarema wrote: post #2

Hi there! We've been building https://github.com/desplega-ai/agent-swarm in the open for a while, and one of the things we wanted to do since the start is find how powerful it can be to push for a code mode like env for the swarm. We tried it and we managed to reduce up +90% of token costs by using swarm scripts (the code mode variant we implemented) for some of our schedules. This is game changer, as we are able to…

I'm sure there is some valuable information in the article, but the LLM-generated prose is unreadable.

Re: Code mode yields a 99.2% cost reduction in our systems

#54

Post by jonathanlydall »

We ultimately switched to this approach with great success for our own product. We stumbled on its success a little by chance.

We have a visual designer and for LLMs to interact with it we originally built a tool per manipulation operation type (e.g. add, edit, delete item - for various item types).

We actually already had a JavaScript API with corresponding .d.ts file for scripting inside our product and one of our clients asked that we also expose this as an MCP tool. I figured sure, should be quite quick and easy, and with Claude Code's help I managed to do it in a single afternoon.

We then found that the LLMs way preferred reaching for this tool, managing with it to get their tasks done with fewer mistakes along the way and in much smaller time frames.

After seeing this and doing some more validation (I've also read that Cloudflare article), we ditched the other tools completely and made a cheat sheet for the LLMs on how to use our API.

Because our API returns decent error messages including stack traces, even if the script fails the LLMs have no trouble making another script to fix their mistake and carrying on from where the error occurred.

In hindsight it really was hardly surprising as LLMs are already aiming to be as good as possible at coding and with JS being so popular I imagine it's particularly good at it.

Re: Code mode yields a 99.2% cost reduction in our systems

#55

Post by tarasyarema »

VulgarExigency wrote: post #37
elzbardico wrote: post #14

Earlier quoted context omitted.

Because it was probably written by an LLM. Use a LLM to summarize it.

Relevant Slavoj Zizek talk: https://www.youtube.com/watch?v=qj4cxK-nOTw

Love this guy, you should read the jokes book he has, pure gold

Re: Code mode yields a 99.2% cost reduction in our systems

#56

Post by firasd »

asdfsa32 wrote: post #44
firasd wrote: post #25

Earlier quoted context omitted.

Yeah, of course. The MCP server tools do use Python in the backend but just look at the affordances it provided in the args -- named columns, a fixed match/replace API, a way to specify the range of rows. The LLM is not gonna create a little library with these affordances in an ephemeral REPL it's just gonna YOLO some pandas code that you hope did what you wanted and addressed the right columns, had the right types,…

I feel like I am having a fever dream. How is an LLM that is trained on endless examples of Python, a well defined language as well, going to generate worst code than some random JSON interface?

Have you ever tried analyzing a CSV in an LLM? Maybe then you'd get what I'm saying

Heck for a lot of questions the LLM won't even write Pandas and just try to eyeball the arithmetic and be like "Yeah March 3 to March 7 values add up to..."

Re: Code mode yields a 99.2% cost reduction in our systems

#58

Post by agentdev001 »

firasd wrote: post #51
shireboy wrote: post #5

A maxim I’ve arrived at working w llm every day is “put deterministic things in code, non deterministic in llm”. I do wish the harnesses would be more helpful in this regard. For example I’ve seen tons of scheduled jobs that people wrote in Claude/copilot/etc that could easily have been scripts. They aren’t scripts because the author doesn’t know how to script and they stop at the point the llm does what they want. I…

An absurd example is how harnesses like Codex, Claude code have no 'cut/paste this range of chars' affordance so if you want to move a block of code the LLM galaxy brain rewrites it in token space in the new file. It ~can~ be done using standard bash utils but there just isn't a kind of standard pattern for 'move this text block via CLI tools' so the LLM and human both overlook it

Anecdotally, when I see coding agents preform this action- I see them using bash. IMO less tools is better, if the agent has a shell- so not having a dedicated cut/paste tool is good.

Re: Code mode yields a 99.2% cost reduction in our systems

#59

Post by eddythompson80 »

asdfsa32 wrote: post #20
firasd wrote: post #15

The problem is that when the LLM writes ad-hoc code how can you trust it? For example in my Liveclip MCP server I'm working on (not released yet) I have table manipulation type tools So for example a couple days ago Claude made a combined ranking score for my Youtube Shorts analytics -- it did all these tool calls in the same turn First it normalized the % values into numbers { "destination_col": "G", "key": "cinemas…

A python script is now more iffy than a LLM going through JSON?

The difference is a yolo’d script needs to be verified everytime it’s generated. So it makes sense to generate it and verify it once (a cli command or mcp call).

I think OP’s example is bad. As that (if I understand the example correctly as I don’t really know what liveclip is) sounds like something worth generating a script for once and reusing it. However, if you’re looking to book a flight, you would prefer an airline cli or mcp server or a nodejs script using puppeteer to automate the browser or even some airline sdk to buy a ticket?

What doesn’t make sense is generating a script to achieve a task without clear feedback loop for verifying the script correctness. In OPs example, there is no way for the LLM to very the correct final result since it’s unknown. It can “code review” the code looking for bugs, but bugs in scripts happen.

Re: Code mode yields a 99.2% cost reduction in our systems

#60

Post by subscribed »

firasd wrote: post #15

The problem is that when the LLM writes ad-hoc code how can you trust it? For example in my Liveclip MCP server I'm working on (not released yet) I have table manipulation type tools So for example a couple days ago Claude made a combined ranking score for my Youtube Shorts analytics -- it did all these tool calls in the same turn First it normalized the % values into numbers { "destination_col": "G", "key": "cinemas…

Mine is forced to always start with writing tests (unit, e2e, smoke), then the script (with eager/early fail modes). I have yet to see this approach to fail.

That's obviously a part of the broader picture Claude always runs inside bwrap sandbox, any work involving python/nodejs must happen in the dedicated devcontainer, standard operations are reused via Makefile, Ai can use dedicated account in the developed $thing for testing in live, hooks governing tool calling are brutal, but it works.

For me at least.

Post reply on HN