Live data from Hacker News

Managing AI Coding Costs at Scale

databricks.com

111–120 of 267 posts

Re: Managing AI Coding Costs at Scale

#111
post #61

Earlier quoted context omitted.

> unguided LLM usage Why aren't you guiding your LLM usage? Is that what I said - to spam it and not guide anything? Or to have a careful workflow where you agree on design and maximize your human judgement/leverage? > any state that's not explicitly being tested and verified in QA loops As opposed to before, when engineers perfectly reasoned about code behavior from first principals and QA was unnecessary?

> Why aren't you guiding your LLM usage? Is that what I said - to spam it and not guide anything? Or to have a careful workflow where you agree on design and maximize your human judgement/leverage? You didn't say anything positively or negatively regarding this so I made an assumption that you were using the LLM relatively unguided (e.g. a bit of oversight, not the kind of thing that heavy code reviews used to involv…

> You didn't say anything positively or negatively regarding this so I made an assumption that you were using the LLM relatively unguided

I feel like this statement betrays your lack of advanced experience coding with LLMs.

OP's elaboration of the steps they are going through (planning, agreeing on plan, getting one LLM to draft execution plan, approving it, then executing with a separate LLM, then reviewing/testing) made it super obvious to me that they are guiding their LLMs quite considerably as part of their work.

Anyone making blanket statements about LLMs producing garbage is just telling on themselves about not having proper SDLC practices in place.

Re: Managing AI Coding Costs at Scale

#113
post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

>Auto-fix bugs and CI until green + thermonuclear review skill x3.

Gotta love this loop, I have it running while I'm asleep all the time.

Re: Managing AI Coding Costs at Scale

#114
post #17

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ball…

Weirdly a lot of the come from company that sell Ai credits in some capacity, and who are also selling (or will soon) some kind of AI gateway or router

Re: Managing AI Coding Costs at Scale

#116

Earlier quoted context omitted.

Whatever you are making this year as SWE you'll be making less next year if the current trend in improvement of AI coding aids is going to be sustained. Think about it : programmers used to derive a lot of their value from the fact that it was a hard skill to acquire. My kids can now 'vibe code' stuff faster (and better looking) than what I could come up with as the beginnings of a design plan. And then I still need…

There's a massive difference between your kids vibe coding something and an engineer using AI to implement something. If you're unable to discern the difference, that's something to reflect on :)

It doesn't matter if GP is able to discern the difference, it matters if your CEO is forced to care about the difference.

Re: Managing AI Coding Costs at Scale

#117
post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

My experience is that your description works for a certain time, since you're knowledgeable of the codebase and can guide it. But after too many iterations with not hand-holding the llm, it quickly gets unwieldy.

Re: Managing AI Coding Costs at Scale

#118
post #32

Earlier quoted context omitted.

Disagree. I operate this way inside a multi-million line legacy codebase.

"startup" and "legacy codebase" are diametrically opposed concepts. And if you're saying (based on your other comments) that a 6 month window is enough to create a legacy codebase...that indicates a serious lack of experience or understanding as to what a legacy codebase is, or why they exist.

Man, so many people in this thread just arguing pointless semantics, making weirdo absolutist (and incorrect) statements.

Accept that other people may ascribe different meanings/interpretations to words than you, and that if your reading of their statement doesn't make sense to you, perhaps you are simply reading it wrong.

Trying to hold someone else to your definition of words suits what purpose exactly? Are you just trying to "win" ?

Re: Managing AI Coding Costs at Scale

#119
post #46

Earlier quoted context omitted.

Do you have issues with performance at the moment? Right now I tend to find that it produces absolutely terrible design patterns and especially performance. I mean maybe I don't know exactly what area you're looking at but yeah for us we tend to find it's terrible wrt dB/caching/scaling and often any performance improvements it proposes end up actually shooting itself in the foot and being worse than before but it's…

Are you using the SOTA models at very high reasoning during planning? IME that makes a LOT of a difference. I‘d also never let them just rip into the architecture, but always push back and ask for alternatives first. Once the overall plan is nailed, not that much can go wrong. Provided it’s a reasonable change set and not a 20k LOC PR.

fable or sol w/ very high both planning and execution, yeah. I feel the "push back" part is a big part of my job now (on every step, planning, execution, and review) yeah, but that feels pretty incompatible with the sorts of "just let it do what it wants" which other people seem to be claiming

Re: Managing AI Coding Costs at Scale

#120
post #17

There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ball…

There really has never been another product priced like AI is being priced right now. Each of these things has been done before, but all of them together is new.

1. Insanely discounted starter plans. Claude $200/mo plan is like $5k-$8k of API rate usage.

2. Very limited cost visibility, they make it hard to figure out where you spent money (unless you're on the enterprise plan which is for people with unlimited money).

3. Nobody, not even the model provider, knows what your request will cost before it returns. You're writing a blank check every time you hit enter.

4. When you run out you run out very suddenly and disruptively. It's very hard to tell a developer on the 28th of the month "sorry, code by hand until the 1st of next month" so you tend to grant exceptions.

5. The price is changing all the time. New models come in, old models come out, prices change, caching behavior changes, harnesses change, etc. The cost of doing a single task is not predictable even if the task does not change.

6. Basically no volume discounting. Anthropic offered us 2% off for committing to $1M+ per year at API rates.

I manage AI spend for my team at work and I try really hard to keep costs under control but it's absolutely herding cats. Much harder than any other spending I've ever had to manage at work.

Post reply on HN