Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

arxiv.org

Post reply on HN
Print view View original

41–50 of 98 posts

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#41

Post by NitpickLawyer »

jurgenburgen wrote: post #40
lucaspiller wrote: post #25

Earlier quoted context omitted.

I think it's more a consequence of pushing for the biggest valuation/IPO. Rumoured profits on inference are north of 70%. Taking SpaceX as an example, they have increased prices across all their consumer products over the past six months. But they definitely aren't short on money with Alphabet and Anthropic combined paying them over $2 billion per month. Microsoft/GitHub lost out here as they were just repacking othe…

> Rumoured profits on inference are north of 70%. Rumors are worth squat when they’re most likely put in motion by the people with a vested interest in this industry. Let’s talk about profits when there’s real data from the IPO documentation.

> Rumors are worth squat

You can make some educated guesses and find out some limits on inferencing cost by looking at 3rd party providers on platforms like openrouter. You can get some median cost /tok for a given model size. Then make some educated guesses on SotA model sizes, and you can get an estimate on pure cost of serving a model. Error bars and all that, of course. But still a range, with some limits.

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#42

Post by bob1029 »

> Furthermore, we observe that input tokens consistently constitute the largest share of consumption for an average of 53.9%

I'm seeing a ratio of around 10:1 in my usage. A vast majority of the tokens consumed are on the input side. The agent will often read a million tokens just to patch one line of code.

I think if you are seeing something closer to 1:1 or more on the output side, there is either a problem with the agent or the codebase is new / empty.

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#43

Post by deadbabe »

drivebyhooting wrote: post #4

In the past Google et al would hire engineers based on how well they could optimize the infrastructure. Maybe soon companies will look at how engineers can optimize the token efficiency of AI.

I know how to drop a company’s token costs to zero: treat tokens as a utility same as internet and make engineers pay for it.

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#44

Post by zozbot234 »

bob1029 wrote: post #42

> Furthermore, we observe that input tokens consistently constitute the largest share of consumption for an average of 53.9% I'm seeing a ratio of around 10:1 in my usage. A vast majority of the tokens consumed are on the input side. The agent will often read a million tokens just to patch one line of code. I think if you are seeing something closer to 1:1 or more on the output side, there is either a problem with th…

If input tokens dominate the cost to that extent, this implies that major gains are possible by making better use of caching. You could basically ask the model to do a one-time "compaction" step including a dump of the relevant portions of the code, and use that as the cached prefix for a large amount of "swarm" subagent calls.

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#45

Post by kolinko »

bob1029 wrote: post #42

> Furthermore, we observe that input tokens consistently constitute the largest share of consumption for an average of 53.9% I'm seeing a ratio of around 10:1 in my usage. A vast majority of the tokens consumed are on the input side. The agent will often read a million tokens just to patch one line of code. I think if you are seeing something closer to 1:1 or more on the output side, there is either a problem with th…

Did you experiment with giving agent better tools to navigate and document the codebase? Asts, language servers and so on?

A million tokens (not cached) sounds like a lot.

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#46

Post by bob1029 »

kolinko wrote: post #45
bob1029 wrote: post #42

> Furthermore, we observe that input tokens consistently constitute the largest share of consumption for an average of 53.9% I'm seeing a ratio of around 10:1 in my usage. A vast majority of the tokens consumed are on the input side. The agent will often read a million tokens just to patch one line of code. I think if you are seeing something closer to 1:1 or more on the output side, there is either a problem with th…

Did you experiment with giving agent better tools to navigate and document the codebase? Asts, language servers and so on? A million tokens (not cached) sounds like a lot.

The target codebase is very large. A million tokens is a drop in the proverbial bucket.

I still don't understand how caching helps me very much. I must be misunderstanding it because I thought the user's prompt (which is the biggest variable) necessarily sits prior to all of these token intensive tool calls. How can we cache the reading of codebase if the prefix is always moving?

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#47

Post by Phemist »

bob1029 wrote: post #46
kolinko wrote: post #45

Earlier quoted context omitted.

Did you experiment with giving agent better tools to navigate and document the codebase? Asts, language servers and so on? A million tokens (not cached) sounds like a lot.

The target codebase is very large. A million tokens is a drop in the proverbial bucket. I still don't understand how caching helps me very much. I must be misunderstanding it because I thought the user's prompt (which is the biggest variable) necessarily sits prior to all of these token intensive tool calls. How can we cache the reading of codebase if the prefix is always moving?

If an agent makes a tool call, the LLM provider will receive the full context again after the result of the tool call becomes available in order to decide the next move. Everything up to the point of the tool call being made will no longer change and could thus in theory be cached. If the agent makes a ton of tool calls, then for every tool call one should be hitting the cache an equal amount of times.

A new instruction by the user will be appended at the end if it done in the same conversation. Thus only has influence on the cacheability of the original agent prompt, but not of subsequent tool calls.

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#48

Post by mariusor »

satvikpendem wrote: post #10

Tokenomics is already a word used to describe cryptocurrency economics, not sure why they'd try to redefine it for AI even if a different sort of token is used.

Tokenomics had been already used by marijuana enthusiasts for a long time.

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#49

Post by alchemism »

satvikpendem wrote: post #10

Tokenomics is already a word used to describe cryptocurrency economics, not sure why they'd try to redefine it for AI even if a different sort of token is used.

cryptocurrency economics = cryptonomics

You're welcome! =)

Re: Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

#50

Post by monkeydust »

I have a MA system setup for personal use.

You give it a problem, you then refine that problem where a fast, cheaper model asks you questions which you answer to get a better input prompt. You then choose a MA strategy for example take problem break up to sections then final judge concludes or you do multi turn where agents debate then judge summarises debate.

The best approach is what I call 'all angles' where all these strategies run in parallel the final meta-judge synthesise the response - the most useful part of this which I recently added is a view to see the variance in each strategy.

Been using this for life stuff - housing search, schools, family challenges!

Perhaps I should make a video of it in action if people in HN community interested let me know.

Post reply on HN