Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Managing AI Coding Costs at Scale

databricks.com

Post reply on HN
Print view View original

31–40 of 267 posts

Re: Managing AI Coding Costs at Scale

#31

Post by extr »

biophysboy wrote: post #18
extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

Do you have tips for generating clean productive output per dollar?

Keep the decision-making and execution separate. Use the high IQ models to chat about the design and make them drive subagents to do the actual work. "Chat" style threads are actually quite cheap. Where it gets expensive is having Fable 5 output thousands of lines of implementation where 95% of it was already overdetermined and there were only a few important judgement calls.

I actually have no doubt that I could replace my Opus 5 Low/Medium subagent profiles with Grok 4.5/GLM 5.2/Deepseek v4 Flash and perf would probably be pretty similar.

On top of that - highly recommend adding accurate cost counters to your statusline. You can't improve what you don't measure! (Or even have any intuition about).

Re: Managing AI Coding Costs at Scale

#32

Post by extr »

reqo wrote: post #21
extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targ…

Disagree. I operate this way inside a multi-million line legacy codebase.

Re: Managing AI Coding Costs at Scale

#33

Post by app13 »

extr wrote: post #26
RugnirViking wrote: post #20

Earlier quoted context omitted.

Do you have issues with performance at the moment? Right now I tend to find that it produces absolutely terrible design patterns and especially performance. I mean maybe I don't know exactly what area you're looking at but yeah for us we tend to find it's terrible wrt dB/caching/scaling and often any performance improvements it proposes end up actually shooting itself in the foot and being worse than before but it's…

Performance is better than ever. It's never been more practical to set up wildly complex synthetic test environments and measure perf wins. Plus the models will find every possible algorithmic/design improvement. It actually gives me quite an uncanny feeling, bulldozing over years of human optimization work with a newer, "perfect" design. Like bringing an AK-47 back to the middle ages.

I needed to thoroughly test rerankers on my companies rather unique corpus.

Opus and I wrote a parallelized test harness and labeled groundtruth in around 2 hours.

In 2022 that would've likely been all I did for a couple sprints

Re: Managing AI Coding Costs at Scale

#34

Post by axus »

platinumrad wrote: post #3

Careful. If you admit to using models that weren't trained by OpenAI or Anthropic then you might hauled in front of Congress: https://www.scmp.com/news/china/diplomacy/article/3362616/us...

Why would it matter if foreign companies analyzed DoorDash data? Pizza deliveries to the Pentagon is all I can come up with, but that's publicly available at https://www.pizzint.watch/

I would bet my entire Polymarket balance ($0) that some military contractors have already asked AIs on the public Internet to design software for them.

Re: Managing AI Coding Costs at Scale

#35

Post by chadash »

jgalt212 wrote: post #19
lbriner wrote: post #17

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ball…

> we started using AI tools and ended up spending millions per year This is how AWS made its fortune.

Not only this, but perhaps even more nefarious is that AWS gives lots of startups $100k+ in credits. This feels generous when you get it. In reality, it means that (unless you are in a compute intensive startup) you can go for months or years before you hit this, but by the time you do, you already have very solid monthly spend.

Initially, you picked the Multi-ZA RDS db.t3.2xlarge instance because you figured "eh i have credits anyway". Two years later, someone looks at this and says "hey, this is expensive and I bet we can do everything we need on a machine half the size". But then they think "if i downsize it and that works, i'll get a thumbs up emoji on a slack thread. If i downsize it and it causes problems, i'll draw the ire of the whole team. I better leave it alone." And the truth is... by the time your company hits the end of those credits, you're probably at the point where that savings isn't gonna do much. Or maybe you are out of business.

And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.

Re: Managing AI Coding Costs at Scale

#36

Post by nujabe »

extr wrote: post #32
reqo wrote: post #21

Earlier quoted context omitted.

IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targ…

Disagree. I operate this way inside a multi-million line legacy codebase.

> I work at a small startup

How does a “small startup” end up with a multi million line “legacy” codebase? Something not mathing

Re: Managing AI Coding Costs at Scale

#37

Post by extr »

pizza234 wrote: post #27
extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered. When I personally want tight code, I have to spend a considerable amount of time adjusting it manually: - It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating - I need to consider whether there are solutions base…

This was more true a few months ago but Fable has improved the situation considerably.

Also just remember - minimalist code looks and feels great but customers do not read your code. I have caught myself many times providing "corrections" to abstractions that were already ~fine, just not perfect. The average SWE costs $200/hr. Careful you don't burn $50 worrying about code that will likely be rewritten or can be better abstracted when that's actually needed.

Re: Managing AI Coding Costs at Scale

#38

Post by extr »

nujabe wrote: post #36
extr wrote: post #32

Earlier quoted context omitted.

Disagree. I operate this way inside a multi-million line legacy codebase.

> I work at a small startup How does a “small startup” end up with a multi million line “legacy” codebase? Something not mathing

Have you worked at many startups?

Re: Managing AI Coding Costs at Scale

#39

Post by pinkgolem »

sandeepkd wrote: post #14

I find this funny and interesting at some levels 1. Codex, Claude and others try to switch models being used at their level itself to manage the cost and outcomes 2. Now company like data bricks develops one more layer on the top of it to do the same task, of finding the base harness and applicable model Companies like Codex and Claude are focussing/investing heavily on to ensure that people are using their harness d…

Open ai is allowing subscription use, anthropic also paused the effort to stop subscription use.

Re: Managing AI Coding Costs at Scale

#40

Post by extr »

app13 wrote: post #33
extr wrote: post #26

Earlier quoted context omitted.

Performance is better than ever. It's never been more practical to set up wildly complex synthetic test environments and measure perf wins. Plus the models will find every possible algorithmic/design improvement. It actually gives me quite an uncanny feeling, bulldozing over years of human optimization work with a newer, "perfect" design. Like bringing an AK-47 back to the middle ages.

I needed to thoroughly test rerankers on my companies rather unique corpus. Opus and I wrote a parallelized test harness and labeled groundtruth in around 2 hours. In 2022 that would've likely been all I did for a couple sprints

Yes 100%. This morning I casually prompted Codex to drive the browser to complete extensive performance testing in-situ that would have literally been weeks of work before. Probably in reality it just wouldn't have been done, and performance guarantees would have been attempted up front via more careful design.

In this case the design was also AI generated, and there were limited wins to be found because the design was already superb.

Post reply on HN