Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Managing AI Coding Costs at Scale

databricks.com

Post reply on HN
Print view View original

71–80 of 269 posts

Re: Managing AI Coding Costs at Scale

#71

Post by extr »

the_sleaze_ wrote: post #28
biophysboy wrote: post #18

Earlier quoted context omitted.

Do you have tips for generating clean productive output per dollar?

in my humble experience it boils down to mastery. Are you at least conversational in the subject matter? You're gonna have a good time just by paying attention and adjusting your workflow. If you're getting a lot of back and forth with it, its asking a lot of planning type questions, stop, step back, rethink the whole feature, and start again from the beginning with everything more fleshed out. If you are in a brand…

This is a great point and I agree. My own productivity varies based on what part of the codebase I'm working on. If it's "been in there before" and I know the right questions to ask, I can one-shot a good design/improvement. If I'm spending 20-30 minutes asking Fable to "draw a diagram so I can understand" - probably less so. But notably, I CAN get there in a fraction of the time it would have taken before. You can general personalized onboarding docs to ~anything.

Re: Managing AI Coding Costs at Scale

#72

Post by darkwater »

nujabe wrote: post #36
extr wrote: post #32

Earlier quoted context omitted.

Disagree. I operate this way inside a multi-million line legacy codebase.

> I work at a small startup How does a “small startup” end up with a multi million line “legacy” codebase? Something not mathing

> How does a “small startup” end up with a multi million line “legacy” codebase?

Easy! The output of 6 months ago Opus! Which seemed so wonderful at the time.

Re: Managing AI Coding Costs at Scale

#73

Post by anon73044 »

extr wrote: post #49
dieselgate wrote: post #45

Earlier quoted context omitted.

> The average SWE costs $200/hr This is a pointless quibble but the hourly rate claim is not true--it's like ~$60 in the USA [0]. Maybe you meant at a specific Org but this is important context when comparing "pricing" between human and AI. [0] https://www.salaryexpert.com/salary/job/software-developer/u...

How is this not true? Taking a Senior SWE @ ~$200K, even just the base salary cost / 2080 working hours is $100/hr. Fully loaded employer cost + accounting for non-coding time gets you to upper 100s easily. Even for a junior making $100K, I have a hard time believe their time is worth less than $75/hr or so. Edit: Fine, "Senior" is not "Average". But naive salary is not the true numerator.

Company time != Pay rate, if you're working somewhere that's publicly traded check out "revenue per employee" metrics sometime.

Re: Managing AI Coding Costs at Scale

#74

Post by chris_money202 »

nujabe wrote: post #44
extr wrote: post #38

Earlier quoted context omitted.

Have you worked at many startups?

No, but not relevant. What is the point of working at a startup if you’re dealing with millions of lines of legacy code ? Isn’t the whole point of startups to create & innovate with a clean slate and modern tools?

I don't know if that's the whole point, but I agree with the sentiment, why would a startup be working in legacy code and where would that code come from if this is truly the start of something.

OP might just be working at a small software company or for one that broke from a bigger one and is now "startup" like?

Re: Managing AI Coding Costs at Scale

#75

Post by pdhborges »

extr wrote: post #49
dieselgate wrote: post #45

Earlier quoted context omitted.

> The average SWE costs $200/hr This is a pointless quibble but the hourly rate claim is not true--it's like ~$60 in the USA [0]. Maybe you meant at a specific Org but this is important context when comparing "pricing" between human and AI. [0] https://www.salaryexpert.com/salary/job/software-developer/u...

How is this not true? Taking a Senior SWE @ ~$200K, even just the base salary cost / 2080 working hours is $100/hr. Fully loaded employer cost + accounting for non-coding time gets you to upper 100s easily. Even for a junior making $100K, I have a hard time believe their time is worth less than $75/hr or so. Edit: Fine, "Senior" is not "Average". But naive salary is not the true numerator.

   I have a hard time believe their time is worth less than $75/hr or so.
In many places in Europe it is.

Re: Managing AI Coding Costs at Scale

#76

Post by ankitmathur »

lbriner wrote: post #17

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ball…

Something underlying a lot of this is that pricing models for enterprise coding tools have changed from seat-based to consumption-based pretty quickly, as AI usage has exploded. For months, engineers were able to use unlimited AI for no marginal cost, but that's changed quickly.

In addition, we're seeing people applying AI to more and more use cases, so token growth is very significant. Paired with consumption pricing, it's brought this problem to the forefront very quickly for lots of companies.

Re: Managing AI Coding Costs at Scale

#77

Post by Karrot_Kream »

extr wrote: post #61
Karrot_Kream wrote: post #48

Earlier quoted context omitted.

Something isn't clear about the size of your codebase here and the level of reliability your customers expect, as a reader of your comments. Clarity there will help. My observation has been: - Initial greenfield work by an LLM is fast and very effective with minimal or no human oversight. - Subsequent work ends up being over engineered and very verbose. Assumptions are made that aren't suited to the problem at hand (…

> unguided LLM usage Why aren't you guiding your LLM usage? Is that what I said - to spam it and not guide anything? Or to have a careful workflow where you agree on design and maximize your human judgement/leverage? > any state that's not explicitly being tested and verified in QA loops As opposed to before, when engineers perfectly reasoned about code behavior from first principals and QA was unnecessary?

> Why aren't you guiding your LLM usage? Is that what I said - to spam it and not guide anything? Or to have a careful workflow where you agree on design and maximize your human judgement/leverage?

You didn't say anything positively or negatively regarding this so I made an assumption that you were using the LLM relatively unguided (e.g. a bit of oversight, not the kind of thing that heavy code reviews used to involve pre-agents.) Feel free to add clarity on your actual usage loop.

> As opposed to before, when engineers perfectly reasoned about code behavior from first principals and QA was unnecessary?

In my experience, most engineers are quite good at reasoning about code behavior for non-QAed code paths. Obviously things fall through the cracks. But I've been in the ground floor of plenty of Big Techs in their early stages before agents and, yes, a lot of initial development had spotty test coverage and yet most of the engineers had good mental models of what was happening. It used to be a very valuable skill to wrap your head around a torrid piece of code with few or no tests but was nonetheless a core piece of your application. Conversely, agentic development can bring cognitive debt [1].

===

This isn't a fight. We aren't sparring over what's right and wrong. I'm just curious how other people use agents in their work as someone who is also now in a startup that uses LLM agents heavily and has no limitations on spend.

[1]: https://martinfowler.com/fragments/2026-02-09.html

Re: Managing AI Coding Costs at Scale

#78

Post by chris_money202 »

shay_ker wrote: post #69

how do any of these routing approaches handle kv cache misses? Devin Fusion is the only one that explicitly addresses this, though it does so by switching models during compaction (not sure this isn't still a cache miss though)

The kv cache is wiped as soon as you get your answer, cloud hosts are not going to hold the GPU memory for your entire session. You're probably referring to some agent level cache

Re: Managing AI Coding Costs at Scale

#79

Post by ankitmathur »

shay_ker wrote: post #69

how do any of these routing approaches handle kv cache misses? Devin Fusion is the only one that explicitly addresses this, though it does so by switching models during compaction (not sure this isn't still a cache miss though)

We're going to do a followup blog detailing our routing approach soon! In short, the router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before (this is the advantage of having a meta-harness that is actually planning all the sub-agents centrally)

Maintaining the cache is extremely, extremely important, so we're iterating fast but that's a major factor we track in the router's development. Couple things I'd look at:

1. The cache is generally reset after a compaction - this is the best time to make a switch if you want.

2. In many cases, the max duration of a cache is 1h, so if a session is being resumed after a long time, that's also a good time to re-assess the complexity.

We're iterating fast here and learning a lot! Definitely a lot to think about it in this area.

Re: Managing AI Coding Costs at Scale

#80

Post by therealdrag0 »

lbriner wrote: post #17

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ball…

On this planet?

They’re not saying they regret doing it, or that it was a mistake.

They’re just saying they’ve gained experience and have leveraged the tools to an extent their usage can be optimized.

Pretty standard business or life iteration.

Post reply on HN