Managing AI Coding Costs at Scale • Hacker News Board

Re: Managing AI Coding Costs at Scale

#231

Post by ianmarcinkowski » Sat, Aug 08, 2026, 12:30 PM UTC

mrlongroots wrote: post #197

extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

In my experience, code is a small fraction of the work. I'm in an infra team and for the last 2 weeks or so I've been trying to understand whether a particular workload will catch fire if a switch is flicked. I'm also new to the team so partly it is me wearing training wheels, familiarizing myself with the telemetry etc, but I will state that I'm not completely lousy at this stuff. No model in my experience can do an…

An AI maximalist on my team put up 2 pull requests with ~120-140 changed files this week. If they spent $200 on tokens to achieve this, we spent $2500-3500 in human salary and opportunity cost reviewing it.

Re: Managing AI Coding Costs at Scale

#232

Post by Terretta » Sat, Aug 08, 2026, 12:34 PM UTC

imilev wrote: post #229

Great article, I think the most important aspect from it is the auto-routing. As humans laziness is in our nature, so having to think if the model is capable enough is not something that most ppl will do - resulting in trying out smaller models which failed our task and then just giving up and running on the bigger model all the time.

DISCLOSURE: I like Databricks. While cosplaying enterprise CTO, I've directed the purchase and heavy integration of their work for over a decade.

That said, this type of post needs to be read with product marketing context in mind.

> I think the most important aspect from it is the auto-routing

On the contrary, in white paper studies auto-routing is shown to destroy the single largest token cost they found, curiously not shown in their opening graphic. As they put it in their own words near the end:

"simple tuning of… caching settings… 50% reduction in … costs, with no observed quality degradation…"

They also aren't showing the harness called ‘pi’ which generally tops results (not only with ‘open’ models), and is consistent with Anthropic's recent "works better when we don't stuff 100k system prompts into context" about Opus 5. Recent models do better with less jank trying to prescribe behavior.

So then one wonders: why is this post written to say you should use a meta router but no mention that's offset by the cache busting, and show developers cost savings but no mention that Anthropic and OpenAI both have developer-facing utilization dashes already?

Perhaps it's so engineers can show enterprise procurement why they want to buy exactly what Databricks happens to have for sale.

Re: Managing AI Coding Costs at Scale

#233

Post by ianmarcinkowski » Sat, Aug 08, 2026, 12:35 PM UTC

lazarie wrote: post #230

sashank_1509 wrote: post #173

I suspect that when it comes to hard complex software products, you’re better off ignoring agents and doing “trad coding”. What you lose in short term speed you gain in manageable complex codebases. If you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term. Now of course, there are products that just involve lots of code but are not actua…

I work on and maintain a 900k+ LoC codebase with mixed languages, runtimes, deployment models, cloud and airgapped ops and it's 99% written by AI. The difference I see the most between our approach and others is that we see the end-user product as an output of our context engineering internal machine, we focus on building that machine and we measure the output like madmen across all and any qualitative software metri…

I'm totally willing to believe that new orders of magnitude of software complexity are going to be possible with this kind of engineering. And also, my company brings in a lot of revenue off of 200k lines of code. My old startup brought in millions of revenue on ... very little code tbh.

Have you ever listened to a politician talk for an hour and you can't remember what they talked about?

If some VC wanted to fund my crazy ideas for building personalized private versions of the internet because people are sick of the nightmare, I would vibe code the hell out of that; but at my current job I have existing customers to keep happy and make sure their financial data doesn't get lost or modified.

Re: Managing AI Coding Costs at Scale

#234

Post by ianmarcinkowski » Sat, Aug 08, 2026, 12:40 PM UTC

Maybe these larger models are better at producing concise, internally-consistent, non-polluting (of the wider codebase) features.

I just spent 3 days reviewing the code of a vibe coding maximalist at work. They needed to touch 96 front-end files, ~50 back-end files. We now have `is_uk_company` spread throughout the entire codebase every time we need to output a date format. I would say 30% of the code can be replaced by a moderately more-wise approach of "you ain't gonna need this", 20% is actively polluting and/or harmful to the coherence of the feature.

I'm pretty sure this plan came from a braindump of a verbal meeting they had with our product team, using OpenAI's top models. There were 2 initial spreadsheets that were ABSOLUTELY INDECIPHERABLE, and now we have a new one labeled "FeatureX - Phase 2 to 4".

Just the date fields is a hilarious example of why I'm skeptical. Python (babel.date) and Javascript (the browser's locale) both have locale-aware date rendering. Using a library for country-based date and time parsing would have eliminated ~12 country-specific change sites... but instead we got `is_uk` and raw date format strings (`%d/%m/%y`) in-line inside already-enormous functions. Also, at a meta level, the backend code shouldn't usually be formatting the dates (data export may be an exception), we should be sending ISO dates and having the frontend format the code if anyone gives a crap.

I'm ~$500/day and I spent 3 days reviewing this stuff. Our CTO spent ~2 days this week reviewing this feature, I'm not sure what his daily rate would end up being. Just the date-formatting part cost me half a day... that's at least $250. A few hundred dollars on Xhigh tokens burned $2500-3500 in human review effort, and I was delayed from:

1. Getting an entire sentiment analysis feature deployed by helping our interns

2. Getting a load balancer solution working to automate a manual process

3. Improving our deterministic Cypress test suite

4. Getting a demo feature launched for our sales team that would bring in more money

Re: Managing AI Coding Costs at Scale

#235

Post by tosh » Sat, Aug 08, 2026, 1:02 PM UTC

> Using harnesses that are “less chatty” (more token efficient), or tuning existing harnesses to generate less token overhead.

agree, this works, undervalued!

look at minimal agents that protect the context window:

  - pi (https://github.com/earendil-works/pi)
  - smol (https://github.com/smol-env/smol)

some thoughts on the other tips (for coding):

1) stronger models are more token efficient for open ended tasks because at the limit …

  - stronger models can solve tasks that the weaker models can not solve
  - stronger models make fewer mistakes, compose things better (cli, abstractions, …)
  - navigate the code base better
  - are better at removing and simplifying the code base again

that of course is difficult to benchmark, so most attention goes to simple benchmarks that show cheaper models can get similar results on 'closed' tasks with easy to 'eval' results

2) dynamic request and task routing sounds great/obvious but is very very hard

  - to benefit from caching you don't want to switch model or inference endpoint
  - to _know_ a certain request can be routed to a weaker/cheaper model needs good context and a strong model to get right and often is still unknowable because the active coding session can go many ways and turn from trivial to challenging in a few turns, always in motion is the future, if you get it wrong you are back in the problem space of #1

using cheaper models and auto-routing do work well for 'closed' tasks where you have something repeatable and can evaluate whether a certain quality threshold is reached that you are comfortable with

for open ended coding sessions it is not so easy

that said: cheaper does not have to mean weaker, you want to look at the pareto frontier and stay up to date on new good models

there are many models like deepseek v4 flash and luna that are both cheaper and way better than most other models

Re: Managing AI Coding Costs at Scale

#236

Post by swader999 » Sat, Aug 08, 2026, 1:14 PM UTC

drTobiasFunke wrote: post #169

extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

Output of 3 or 4 2022 engineers? Its that your self assessment? Output as in number of lines of code?

Not op, but we are at 2400 total points delivered over seven years. 1000 of those in the last six months. About 2-4 devs over that period, just two the last six months.

Re: Managing AI Coding Costs at Scale

#237

Post by sbarre » Sat, Aug 08, 2026, 1:20 PM UTC

rustystump wrote: post #167

sbarre wrote: post #118

Earlier quoted context omitted.

Man, so many people in this thread just arguing pointless semantics, making weirdo absolutist (and incorrect) statements. Accept that other people may ascribe different meanings/interpretations to words than you, and that if your reading of their statement doesn't make sense to you, perhaps you are simply reading it wrong. Trying to hold someone else to your definition of words suits what purpose exactly? Are you jus…

This is an ultra cop out. There are standards in language that are not all “left means right for me so you cannot assume when i say right it is right and not left” This whole thread around loc is depressing. It speaks volumes of some peoples inexperience working on actual legacy code. Legacy code is not just age or size but that the technical foundation is dated in a fundamental way. A giant monolith running on a now…

> There are standards in language that are not all “left means right for me so you cannot assume when i say right it is right and not left”

Yes of course there are words that have very clear meaning, and that's not the case here.

"startup" and "legacy" - used in the context of this discussion - are not words like "left" and "right" where everyone has a universal agreed-upon understanding of their meaning.

Re: Managing AI Coding Costs at Scale

#238

Post by oblio » Sat, Aug 08, 2026, 1:23 PM UTC

roncesvalles wrote: post #155

oblio wrote: post #153

Earlier quoted context omitted.

> Since software is still a winner-takes-all market, the mass-production property of software doesn't really matter. Fairly sure software dev was always an iceberg. Most software and most software devs aren't working on horizontal software, but on vertical software, in cost centers. Sales for that kind of software don't scale as much.

Most vertical software is just horizontal software glued together

Horizontal open source modules... And making the horizontal modules doesn't employ many people, both because of FOSS and also because you only need so many reusable modules.

When we were making brick houses we needed a few brick factories with thousands of people for million of bricklayers.

Re: Managing AI Coding Costs at Scale

#239

Post by sbarre » Sat, Aug 08, 2026, 1:24 PM UTC

Karrot_Kream wrote: post #198

skinfaxi wrote: post #179

Earlier quoted context omitted.

> If you think "unguided" means "I typed a prompt into claude code and waited yolo" I don't know what to say but, you have a very different idea of what professionals do than I do. What exactly does "unguided" mean to you, then?

Not having human input in the loop, i.e. allowing agents to act without guidance. I understand the idea of having agents guide agents, but really how much do we gain when Sol scolds Fable?

> how much do we gain when Sol scolds Fable?

You might be surprised at how much benefit you get from the model council[0] pattern.

The common pattern I've seen is "build with Claude, review with GPT", or vice-versa.

It's been proven to be very effective.

0: https://www.google.com/search?q=model+council+pattern&udm=14

Re: Managing AI Coding Costs at Scale

#240

Post by tosh » Sat, Aug 08, 2026, 1:27 PM UTC

extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

for complex open ended coding tasks better models are better

(and mid-to-long-term, often also short-term end up cheaper than weaker models)

this might change soon if we are reaching a certain capability threshold

but right now that's still the case

unless you are working on throw-away trivial stuff where iteration speed and trying many speculative things might give you an edge