Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Managing AI Coding Costs at Scale

databricks.com

Post reply on HN
Print view View original

271–280 of 287 posts

Re: Managing AI Coding Costs at Scale

#271

Post by Terretta »

aabdi wrote: post #245
Terretta wrote: post #232

Earlier quoted context omitted.

DISCLOSURE: I like Databricks. While cosplaying enterprise CTO, I've directed the purchase and heavy integration of their work for over a decade. That said, this type of post needs to be read with product marketing context in mind. > I think the most important aspect from it is the auto-routing On the contrary, in white paper studies auto-routing is shown to destroy the single largest token cost they found, curiously…

If that’s such a large problem then the clear solution is to do sticky sessions? What’s the problem then? Use a cheap ds4 or Luna and do the second model net net per one shot best case you save couple dollars per?

Exactly. There are plenty things they could suggest for that.

Anyway you'll likely get a more effective and up to date set of advice by skipping marketing pieces and pointing your favorite frontier model at /r/LocalLLaMA in deep research mode and asking it to synthesize the latest advice, or even try ideas out for itself and let you know what works for your own session history.

Re: Managing AI Coding Costs at Scale

#272

Post by swader999 »

drTobiasFunke wrote: post #252
swader999 wrote: post #236

Earlier quoted context omitted.

Not op, but we are at 2400 total points delivered over seven years. 1000 of those in the last six months. About 2-4 devs over that period, just two the last six months.

Noone is questioning the volume of llm output. My question is whether all those points delivered improved your product and software in any meaningful way, or do you now have 20x more code that noone understands with the same quality of software product?

Yeah it did. We've signed three new customers because of one. Another large feature was an integration that landed more. We rewrote the entire front end to modern stack. Used to be a fifteen year old react and backbone bFrankenstein app.

Re: Managing AI Coding Costs at Scale

#273

Post by [deleted] »

habosa wrote: post #120
lbriner wrote: post #17

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ball…

There really has never been another product priced like AI is being priced right now. Each of these things has been done before, but all of them together is new. 1. Insanely discounted starter plans. Claude $200/mo plan is like $5k-$8k of API rate usage. 2. Very limited cost visibility, they make it hard to figure out where you spent money (unless you're on the enterprise plan which is for people with unlimited money…

[deleted]

Re: Managing AI Coding Costs at Scale

#274

Post by [deleted] »

lukasco wrote: post #254

With self-driving agents, the costs stop being evident as you go, and show up after the fact. I've been building governors that slow the agents down, and then also look at odd things some of the harnesses do, such as constantly making mistakes in tool calls. But overall, it's quite a task, and you really have to decide what you are prioritizing for. Do I want my agents doing lots of work, or (in my case), leaving som…

[deleted]

Re: Managing AI Coding Costs at Scale

#275

Post by emanuelecella »

jgilias wrote: post #217
arendtio wrote: post #212

I think the important aspects are making individual developers responsible for their costs and restricting the most expensive models when costs are too high. That way, people have a chance to learn when they tend to use the most expensive models for trivial tasks.

Definitely. I can easily cut my individual AI spend by 50% at least. But I won’t, until someone tells me to. For three reasons: 1. Thinking about how to optimize AI spend is one more thing to think about when I have too many things to think about already; 2. If there’s even a minuscule chance that me selecting down the model results in a task getting delayed because for that particular thing the model happened to be…

You shouldn't have to. The gate should be automatic and not a decision you make per request but a hard cap enforced at the infrastructure layer before the call happens from your providers

Re: Managing AI Coding Costs at Scale

#276

Post by andrekandre »

dan_q wrote: post #224
lbriner wrote: post #17

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ball…

> On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs. The AI rollout has in large part been about giving management an opportunity to say "fuck you" to engineering. Engi…

  > The ROI is besides the point. It's about breaking the spirit of engineering labor with irrational decisions made outside of their control.
it may be where you are at (not disagreeing at all with your experience), but where i am the management really are 'true believers' so to speak... they are all-in and mesmerized by the tech (call it fomo or whatever)

Re: Managing AI Coding Costs at Scale

#277

Post by andrekandre »

chadash wrote: post #35
jgalt212 wrote: post #19

Earlier quoted context omitted.

> we started using AI tools and ended up spending millions per year This is how AWS made its fortune.

Not only this, but perhaps even more nefarious is that AWS gives lots of startups $100k+ in credits. This feels generous when you get it. In reality, it means that (unless you are in a compute intensive startup) you can go for months or years before you hit this, but by the time you do, you already have very solid monthly spend. Initially, you picked the Multi-ZA RDS db.t3.2xlarge instance because you figured "eh i h…

  > And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.
the best thing to do is avoid the first cigarette (but everyone at the office smokes, and if you want a promotion you'll smoke with them too)

Re: Managing AI Coding Costs at Scale

#278

Post by andrekandre »

adamddev1 wrote: post #243

> "Exponentially growing costs" I don't think people thing don't think enough about the exponentially growing costs with AI. The cost of tokens, the cost of bugs, the cost of technical debt, the cost of software bloat and slowdown, the cost on the RAM market, on the environment, the costs of the little lies and hallucinations that creep into our codebases, blogs, books, and search results. What seems like a manageabl…

  > What seems like a manageable margin suddenly starts growing and destroying everything
probably where the 1 trillion+ dollar valuations are coming from.... lots of thick saas margins they can siphon/skim from the whole industry.

Re: Managing AI Coding Costs at Scale

#279

Post by catfood »

krzyk wrote: post #220
catfood wrote: post #113

Earlier quoted context omitted.

>Auto-fix bugs and CI until green + thermonuclear review skill x3. Gotta love this loop, I have it running while I'm asleep all the time.

How much does it cost you?

Practically free with any of the subscription plans.

Re: Managing AI Coding Costs at Scale

#280

Post by ankitmathur »

Terretta wrote: post #232
imilev wrote: post #229

Great article, I think the most important aspect from it is the auto-routing. As humans laziness is in our nature, so having to think if the model is capable enough is not something that most ppl will do - resulting in trying out smaller models which failed our task and then just giving up and running on the bigger model all the time.

DISCLOSURE: I like Databricks. While cosplaying enterprise CTO, I've directed the purchase and heavy integration of their work for over a decade. That said, this type of post needs to be read with product marketing context in mind. > I think the most important aspect from it is the auto-routing On the contrary, in white paper studies auto-routing is shown to destroy the single largest token cost they found, curiously…

Hey! Thanks for the feedback! I work on many of these things at Databricks, so figured I'd chime in on this.

Firstly, while routing is important, simple things like observability into the token costs of various features, which can drive optimization of better default parameters, are low hanging fruit everyone should do.

1. As for the cache busting, we're very aware of this. Had said this in a thread above too, so copying it here. We're going to do a followup blog detailing our routing approach, but we're designing it to be cache aware. The router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before. This does have disadvantages too, but the cache is the dominating cost reduction force.

2. We do love Pi (we published this too: https://www.databricks.com/blog/benchmarking-coding-agents-d...), but we didn't mention it here because we're still in the process of making it available to devs internally. There's a lot that goes into this more than just cost (e.g. feature gaps vs. other common harnesses).

3. As for existing utilization dashboards, it's good that each tool has one, but we found that as we gave developers more freedom to use more tools, it was impossible to have a single source of truth. Without that, it's hard to really understand all-in AI spend and optimize it.

Post reply on HN