“use lower cost models”
“use price controls”
Truly revolutionary stuff.
Managing AI Coding Costs at Scale
181–190 of 269 posts
Re: Managing AI Coding Costs at Scale
#182> nearly every company deploying AI tools at scale has hit the same wall: exponentially growing costs. That curve is unsustainable - left unchecked it will eventually overtake revenue.
But the question remains, AI hasn't shown any explosion in growth/revenue for most companies. The effects seem muted.
Re: Managing AI Coding Costs at Scale
#183I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…
The output yes, but do you produce the impact and value of 3 engineers? I have seen this workflow being toyed with too, and I find it to produce massively overengineered stuff that actual people don't really wanna use
He only spot checks 1000s loc diffs, so probably has no clue.
Re: Managing AI Coding Costs at Scale
#184Earlier quoted context omitted.
but I produce the output of 3 or 4 2022 engineers and probably at better quality. Possibly, but the output of a 2022 engineer is about 1/10th of the output of a 2010 engineer, so it's an extremely low bar.
also - as always with these claims there's no actual product / repo / whatever one could check. I would love to see what these tools create but outside slop there's never: This works, is in production, here's the code. Any day now.
I found an interesting project recently. As I was looking through the source something felt off. Turned out to be entirely LLM written. There was duplicated code everywhere, same function defined in dozens of files (same name, same intended behavior) but none of them would produce the same output for an input. Dead code all over the place. Over architected. Useless comments.
It was all generated in the last 4 months, so don’t come at me with the “but did they use a model from the last 6 months” nonsense.
Re: Managing AI Coding Costs at Scale
#185I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…
IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targ…
I disagree, the approach works well in a legacy project, since there are structures and standards that already exist, which them model can draw from (if you aren't more explicit about it in AGENTS.md)
Re: Managing AI Coding Costs at Scale
#186I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…
You have an unlimited budget, and you only spend $80/day? I’m up to $3k/week, and still expanding.
Re: Managing AI Coding Costs at Scale
#187I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…
I'm probably between $50-$200/day depending on the day; we also have effectively unlimited budget, though a lot of that is because Azure gives startups $150,000 in credits for 2 years, which we've wired up to a LiteLLM gateway & OpenCode. Without that I think our appetite would be more around $400/month/employee.
A lot of my high costs is because I just throw Sol at everything. If I were more selective and brought in Luna or v4 Flash every once in a while, I think I'd be more like ~$400/month. That's why I'm not aligned with the notion that "tokens are subsidized so that's why people are using so much": its not that I'll have to adjust to using less, its just that I'd need to think before I prompt a bit and be more judicious. I could easily see my raw token counts doubling or tripling in the coming months. I don't think that will change as subsidization subsides; though maybe lab revenue will; intelligence per dollar is getting cheaper every week. Its solely a function of adaptation to process, which takes time.
The productivity gains per token are the single most asymmetrical thing I've ever seen in engineering. The engineers on our team are pretty effective with tokens; easily that 2x-4x output as you're seeing, spending $20-$200/day. Some of our security folks have also started contributing more-and-more code, and they're on the other side: they'll spend hundreds a day running in circles, eventually producing these +/-30k loc pull requests that take ages to get merged and are littered with issues. They weren't writing much code before, so arguably they're more productive by some multiplier greater than 1, but I think the drag on the rest of the team, and potential issues with what they produce, has overall created a net-negative situation. Inversely, some other company functions have produced a few one-off websites for things like sales processes, and those have been a huge win. The asymmetry is wild. There's almost a valley of incoming skill where if you know nothing about code, you'll leverage it well; if you know just a little bit, it makes you super dangerous; if you know a lot, you're the biggest winner. Really difficult situation to navigate.
Re: Managing AI Coding Costs at Scale
#188There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ball…
There really has never been another product priced like AI is being priced right now. Each of these things has been done before, but all of them together is new. 1. Insanely discounted starter plans. Claude $200/mo plan is like $5k-$8k of API rate usage. 2. Very limited cost visibility, they make it hard to figure out where you spent money (unless you're on the enterprise plan which is for people with unlimited money…
I think my startup can help: https://unbiased.ai
Happy to give you (or anyone here) some trial credits if interested! Email address in my profile.
Re: Managing AI Coding Costs at Scale
#189Earlier quoted context omitted.
How is this not true? Taking a Senior SWE @ ~$200K, even just the base salary cost / 2080 working hours is $100/hr. Fully loaded employer cost + accounting for non-coding time gets you to upper 100s easily. Even for a junior making $100K, I have a hard time believe their time is worth less than $75/hr or so. Edit: Fine, "Senior" is not "Average". But naive salary is not the true numerator.
I hire contractors for a large enterprise in the US. The going rate is typically $85-$100/hr for a senior dev, depending on specialization. Lead-level maybe $120 for the right skill set.
[deleted]
Re: Managing AI Coding Costs at Scale
#190Earlier quoted context omitted.
This was more true a few months ago but Fable has improved the situation considerably. Also just remember - minimalist code looks and feels great but customers do not read your code. I have caught myself many times providing "corrections" to abstractions that were already ~fine, just not perfect. The average SWE costs $200/hr. Careful you don't burn $50 worrying about code that will likely be rewritten or can be bett…
Also, you can probably think about it like this: "Will I benefit from this code being minimalist before [date]", where [date] is whenever you think the agent will be good enough to come back and make the corrections you would make today.
I'd caution that some corrections become harder to make over time, rather than easier. A bad architecture now can become much harder to fix once other things have grown up around it.
Even as a self-contained unit, you can't step in the same river twice, and on [date] some important details may have seriously faded, both in terms of text that can be mined and also in terms of human "why did we do that" and "what was the reason we did it this way and not that way" etc.