[dead]
Managing AI Coding Costs at Scale
281–289 of 289 posts
Re: Managing AI Coding Costs at Scale
#282Earlier quoted context omitted.
We are 7 years into a cycle of people baselessly claiming that "AI" will soon develop superhuman abilities and might just take over the world, before killing us all. I don't think there's any way to take part in this discourse without making somewhat unsupported claims.
We haven’t had LLM agents for 7 years
They claimed these problems LONG before agents were the new thing. To my eye it started with the bullshit around GPT-2, and that was in 2019, 7 years ago.
Re: Managing AI Coding Costs at Scale
#283Maybe these larger models are better at producing concise, internally-consistent, non-polluting (of the wider codebase) features. I just spent 3 days reviewing the code of a vibe coding maximalist at work. They needed to touch 96 front-end files, ~50 back-end files. We now have `is_uk_company` spread throughout the entire codebase every time we need to output a date format. I would say 30% of the code can be replaced…
> Just the date fields is a hilarious example of why I'm skeptical. Python (babel.date) and Javascript (the browser's locale) both have locale-aware date rendering. Using a library for country-based date and time parsing would have eliminated ~12 country-specific change sites... but instead we got `is_uk` and raw date format strings (`%d/%m/%y`) in-line inside already-enormous functions. Also, at a meta level, the ba…
I've worked with my teammate to see why this is suboptimal and negotiated a lot of concessions. I didn't re-review the code before we pushed it to staging for QA to test over the weekend... I'm worried about what I'll find when I look on Monday.
Re: Managing AI Coding Costs at Scale
#284There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ball…
The issue is the growth rates can cause costs to drastically change quickly. If you have 1,000 employees and the average is spending $100/month you're at a $1.2M run rate. But suddenly a new model comes out that's twice as expensive, there are some changes to the harness (we found randomly Claude Code and other harnesses will make changes that drastically impact efficiency), and then maybe you have some organic user…
[flagged]
Re: Managing AI Coding Costs at Scale
#285Earlier quoted context omitted.
I found an interesting project recently. As I was looking through the source something felt off. Turned out to be entirely LLM written. There was duplicated code everywhere, same function defined in dozens of files (same name, same intended behavior) but none of them would produce the same output for an input. Dead code all over the place. Over architected. Useless comments. It was all generated in the last 4 months,…
4 months is ancient. Fable and Sol are a different animal.
You said the same thing about whatever model was released 4 months ago. You are a walking meme.
Re: Managing AI Coding Costs at Scale
#286Earlier quoted context omitted.
> I don't think this clear lack of ability is some conspiracy. Maybe currently not. But we will never be able to know, as models are undeterministic and benchmarks are kind of scams. When you cannot prove that something gets worse, rest assured companies will to it.
This requires something other than a free market, and for China to not exist. We're not there yet. There's plenty of competition to prevent this, for now, with AI spending being an incredibly hot issue.
Currently the bad quality of architecture and holistic view of models is probably explained by the fact that the models just cannot do it. Free markets and China will make sure of that, kind of. In reality Claude and Codex are better and provide much better tooling, so not much competition from free markets and self-hosted models from China, I think.
Then when competition really settles to monoploy or duopoly, like it always does in bigtech, it is really difficult to prove Anthropic and OpenAI enshittify their models. Same with Opus 4.6, that suddenly was worse when 4.7 came out.
Re: Managing AI Coding Costs at Scale
#287Earlier quoted context omitted.
This requires something other than a free market, and for China to not exist. We're not there yet. There's plenty of competition to prevent this, for now, with AI spending being an incredibly hot issue.
Currently the bad quality of architecture and holistic view of models is probably explained by the fact that the models just cannot do it. Free markets and China will make sure of that, kind of. In reality Claude and Codex are better and provide much better tooling, so not much competition from free markets and self-hosted models from China, I think. Then when competition really settles to monoploy or duopoly, like i…
> Currently the bad quality of architecture and holistic view of models is probably explained by the fact that the models just cannot do it. Free markets and China will make sure of that, kind of.
I'm not following. Free markets and China will make sure that models stay bad!? Why didn't these entities already stop the progression we've seen? What has been motivating them all this time that has preventing them from stopping?
Keep in mind this was all science fiction just a few years ago.
Re: Managing AI Coding Costs at Scale
#288Earlier quoted context omitted.
Man, so many people in this thread just arguing pointless semantics, making weirdo absolutist (and incorrect) statements. Accept that other people may ascribe different meanings/interpretations to words than you, and that if your reading of their statement doesn't make sense to you, perhaps you are simply reading it wrong. Trying to hold someone else to your definition of words suits what purpose exactly? Are you jus…
Yes lol. Of all things people are getting on me for it's the number of LoC x Years In Business of this startup. I don't fucking know, I didn't start the company and I wasn't here for several of those industrious years. Looking now it looks like we have slightly fewer LoC than that, I was counting some of the generated stuff. But who cares? The point is any codebase over a few years old with lots of customers and a bi…
Sounds like you simply don't know what you're talking about.
Re: Managing AI Coding Costs at Scale
#289Earlier quoted context omitted.
DISCLOSURE: I like Databricks. While cosplaying enterprise CTO, I've directed the purchase and heavy integration of their work for over a decade. That said, this type of post needs to be read with product marketing context in mind. > I think the most important aspect from it is the auto-routing On the contrary, in white paper studies auto-routing is shown to destroy the single largest token cost they found, curiously…
Hey! Thanks for the feedback! I work on many of these things at Databricks, so figured I'd chime in on this. Firstly, while routing is important, simple things like observability into the token costs of various features, which can drive optimization of better default parameters, are low hanging fruit everyone should do. 1. As for the cache busting, we're very aware of this. Had said this in a thread above too, so cop…
> Firstly, while routing is important, simple things like observability
STRONG agree. Most devs, if they can see their costs, try. Even just putting it in a starship prompt goes a long way!
Observability supports effective, responsible, and accountable use, "cost controls" (as typically enterprise defined) supports bean counters. Ensuring Observability is recognized as a valid Control™ should be on every engineering executive's TODO list.
> the cache is the dominating cost reduction force
TY for the rich reply. As you're very aware of the cache issue, it might have been nice to have the word "cache" more prominent on your "Key Techniques" image (currently Lower Cost Models, Smart Routing, Spend Controls, Context Optimization; cache tuning buried at the end) or higher in the body sections.
It was mostly the image that prompted (ha) me to call out the marketing driven angle.