Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are the costs of AI agents also rising exponentially? (2025)

tobyord.com

Post reply on HN
Print view View original

41–50 of 155 posts

Re: Are the costs of AI agents also rising exponentially? (2025)

#41

Post by tibbar »

zozbot234 wrote: post #40
thelastgallon wrote: post #15

> On many task lengths (including those near their plateau) they cost 10 to 100 times as much per hour. For instance, Grok 4 is at $0.40 per hour at its sweet spot, but $13 per hour at the start of its final plateau. GPT-5 is about $13 per hour for tasks that take about 45 minutes, but $120 per hour for tasks that take 2 hours. And o3 actually costs $350 per hour (more than the human price) to achieve tasks at its fu…

This effect is likely even larger when you consider that the raw cost per inferred token grows linearly with context, rather than being constant. So longer tasks performed with higher-context models will cost quadratically more. The computational cost also grows super-linearly with model parameter size: a 20B-active model is more than four times the cost of a 5B-active model.

Doesn't context cacheing mostly eliminate this problem? (I suppose for enough context the 90% discount is eventually a lot anyway)

Re: Are the costs of AI agents also rising exponentially? (2025)

#42

Post by zozbot234 »

tibbar wrote: post #41
zozbot234 wrote: post #40

Earlier quoted context omitted.

This effect is likely even larger when you consider that the raw cost per inferred token grows linearly with context, rather than being constant. So longer tasks performed with higher-context models will cost quadratically more. The computational cost also grows super-linearly with model parameter size: a 20B-active model is more than four times the cost of a 5B-active model.

Doesn't context cacheing mostly eliminate this problem? (I suppose for enough context the 90% discount is eventually a lot anyway)

Context caching is really storing the KV-cache for reuse. It saves running prefill for that part of the context, but tokens referencing that KV-cache will still cost more.

Re: Are the costs of AI agents also rising exponentially? (2025)

#45

Post by bitwize »

xantronix wrote: post #29
bitwize wrote: post #26

Earlier quoted context omitted.

Cloud can pay more for RAM until all the RAM producers withdraw from the consumer market, then prices will go back down. End users will still get access to RAM. The cloud terminal they purchase from Apple, Google, Samsung, or HP will have all the RAM it will ever need directly soldered onto it.

I was really fucking hoping we weren't at the part where "cloud terminals" doesn't seem farfetched and paranoid and yet here we are. Jesus Christ.

The next step, I think, will be a "cash for clunkers" program to permit people to trade in old computer hardware to the government—especially since operating systems that do not collect KYC data on their users will soon be illegal to operate.

Re: Are the costs of AI agents also rising exponentially? (2025)

#46

Post by chii »

keepamovin wrote: post #43

My expectation: demand going up, prices will rise, supply will saturate to the point of ubiquitous "utility" status, and prices will drop, probably a bell curve shape with sine-wave undulations along the way.

> supply will saturate

that depends on the ability to produce supply at a saturation rate.

It did work for internet backhaul links - ala, those dark fibres. However, i reckon those fibres are easier to manufacture than silicon chips.

I wonder if saturation is possible for ai capable chips.

Re: Are the costs of AI agents also rising exponentially? (2025)

#50

Post by jiggawatts »

zozbot234 wrote: post #34
jiggawatts wrote: post #32

Earlier quoted context omitted.

> Until there is some drastic new hardware For inference, there is already a 10x improvement possible over a setup based on NVIDIA server GPUs, but volume production, etc... will take a while to catch up. During inference the model weights are static, so they can be stored in High Bandwidth Flash (HBF) instead of High Bandwidth Memory (HBM). Flash chips are being made with over 300 layers and they use a fraction of t…

> Flash chips are being made with over 300 layers and they use a fraction of the power compared to DRAM. Isn't reading from flash significantly more power intensive than reading DRAM? Anyway, the overhead of keeping weights in memory becomes negligible at scale because you're running large batches and sharding a single model over large amounts of GPU's. (And that needs the crazy fast networking to make it work, you g…

For a given capacity of memory, Flash uses far less power than DRAM, especially when used mostly for reads.

> becomes negligible at scale

Nothing is negligible at scale! Both the cost and power draw of the HBMs is a limiting factor for the hyperscalers, to the point that Sam Altman (famously!) cornered the market and locked in something like 40% of global RAM production, driving up prices for everyone.

> sharding a single model over large amounts of GPUs

A single host server typically has 4-16 GPUs directly connected to the motherboard.

A part of the reason for sharding models between multiple GPUs is because their weights don't fit into the memory of any one card! HBF could be used to give each GPU/TPU well over a terabyte of capacity for weights.

Last but not least, the context cache needs to be stored somewhere "close" to the GPUs. Across millions of users, that's a lot of unique data with a high churn rate. HBF would allow the GPUs to keep that "warm" and ready to go for the next prompt at a much lower cost than keeping it around in DRAM and having to constantly refresh it.

Post reply on HN