Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code mode yields a 99.2% cost reduction in our systems

agent-swarm.dev

Post reply on HN
Print view View original

61–70 of 77 posts

Re: Code mode yields a 99.2% cost reduction in our systems

#61

Post by Diogenesian »

asdfsa32 wrote: post #8
somnium_sn wrote: post #4

I am still a bit surprised that this is not more widely used. Normal inference + tool calls is effectively composition via serialized natural language, whereas Code Mode / Programmatic Tool Calling is using purpose built languages (aka programming languages) for composition and concurrency. It somewhat feels obvious to me that the latter is way more token and latency efficient.

In the world of AI hype, everything is to be learned again and we are going to see many many breakthroughs like this. You can see this shift already in the tone of conversation in HN for example. The basics of computer engineering is now novel approaches. We are discovering why programming languages exist, starting with a worst version of Cobal running on a very expensive and probabilistic VM. We will shoehorn every…

"You know, as impressive as these systems are, they sure are bafflingly stupid. I am wondering if perhaps there is more to human intelligence than technical mastery of human language."

Drew McDermott, 1976, "Artificial Intelligence Meets Natural Stupidity": https://dl.acm.org/doi/10.1145/1045339.1045340

  I wish to rail against a pervasive sloppiness in our thinking, the tendency to see in natural language a natural source of problems and solutions. Many researchers tend to talk as if an internal knowledge representation ought to be closely related to the "corresponding" English sentences; and that operations on the structure should resemble human conversation or "word problems". Because the fault here is a disregard for logic, it will be hard for my criticism to be logical and clear.
Note that 1976 had some cool AI, including theorem-provers and story-tellers (Knuth wrote a story-telling program, I think it's an exercise in TAoCP). Yet the underlying theory of intelligence was "intelligence is logical processing of symbolic information"; this is obviously inadequate, but the ideology of analytic philosophy is a helluva drug for a scientist. Likewise with the 21st century "intelligence is about correctly predicting things based on past information,"; this is obviously inadequate, but the ideology of big data technocracy is a helluva drug for a scientist.

Re: Code mode yields a 99.2% cost reduction in our systems

#62

Post by tarasyarema »

jonathanlydall wrote: post #54

We ultimately switched to this approach with great success for our own product. We stumbled on its success a little by chance. We have a visual designer and for LLMs to interact with it we originally built a tool per manipulation operation type (e.g. add, edit, delete item - for various item types). We actually already had a JavaScript API with corresponding .d.ts file for scripting inside our product and one of our…

yes, typed "sdk"s are key, and indeed they are so good at js

we did some iterations specially on the typing and error messages, as you mentioned, that help a lot in the reliability

Re: Code mode yields a 99.2% cost reduction in our systems

#63

Post by subscribed »

lyall wrote: post #17
tarasyarema wrote: post #2

Hi there! We've been building https://github.com/desplega-ai/agent-swarm in the open for a while, and one of the things we wanted to do since the start is find how powerful it can be to push for a code mode like env for the swarm. We tried it and we managed to reduce up +90% of token costs by using swarm scripts (the code mode variant we implemented) for some of our schedules. This is game changer, as we are able to…

There might be something valuable in this article, but if the you couldn't be bothered to write it yourself, why should we be bothered to read it?

Indeed it's the HN variant of "I asked ChatGPT and it said...."

I wish there was a way to tag this type of the posts, but at least my HN client allows to tag the authors.

Re: Code mode yields a 99.2% cost reduction in our systems

#65

Post by neongreen »

tarasyarema wrote: post #2

Hi there! We've been building https://github.com/desplega-ai/agent-swarm in the open for a while, and one of the things we wanted to do since the start is find how powerful it can be to push for a code mode like env for the swarm. We tried it and we managed to reduce up +90% of token costs by using swarm scripts (the code mode variant we implemented) for some of our schedules. This is game changer, as we are able to…

Look at https://randomlabs.ai/blog (Slate), they're exactly that and you can play with it

Re: Code mode yields a 99.2% cost reduction in our systems

#66

Post by vessenes »

tarasyarema wrote: post #32
vessenes wrote: post #9

This quarter’s clankerisms: “Honest” and “vibes”. The tech here is cool; good reminder to push for more scripts in general from the agentic tools.

thanks!

How’s the agent swarm working? Do you have like a sample startup you run using it?

Re: Code mode yields a 99.2% cost reduction in our systems

#68

Post by asdfsa32 »

firasd wrote: post #56
asdfsa32 wrote: post #44

Earlier quoted context omitted.

I feel like I am having a fever dream. How is an LLM that is trained on endless examples of Python, a well defined language as well, going to generate worst code than some random JSON interface?

Have you ever tried analyzing a CSV in an LLM? Maybe then you'd get what I'm saying Heck for a lot of questions the LLM won't even write Pandas and just try to eyeball the arithmetic and be like "Yeah March 3 to March 7 values add up to..."

And often -- without using code _openly_ or in thinking -- it is wrong because it is just a "guesstimate" at best due to the nature of how LLMs generate tokens.

Re: Code mode yields a 99.2% cost reduction in our systems

#69

Post by peterbell_nyc »

Yeah, this has been my biggest contention with the models everywhere paradigm. I will definitely use a supervisor pattern and advisor pattern for simpler models when I just want to throw something together interactively with Fable and a few sub-agents. But for anything I am doing repeatedly, I built a deterministic orchestrator to run the steps and I have a hard rule that the skills that run from my repo can only ever be a thin shims calling to my central system so that rather than having agents enrich a person or call an API or do research they fire off predefined multi-step deterministic plays that just might have models for classification generation summarization and/or review.

In addition to the reliability and cost benefits you also then get shared capacity broker capabilities so if you're only able to enrich so many people or only capable of doing so many CI runs if that's all done deterministically you can have some intelligent orchestration in your main server to manage that limited capacity across all your agents that are trying to use it at the same time.

Post reply on HN