Earlier quoted context omitted.
There is no evidence that Anthropic's revenue is 70B.
https://www.linkedin.com/posts/sytaylor_anthropic-is-now-tra...
[dead]
Managing AI Coding Costs at Scale
241–250 of 267 posts
Re: Managing AI Coding Costs at Scale
#242Great article, I think the most important aspect from it is the auto-routing. As humans laziness is in our nature, so having to think if the model is capable enough is not something that most ppl will do - resulting in trying out smaller models which failed our task and then just giving up and running on the bigger model all the time.
DISCLOSURE: I like Databricks. While cosplaying enterprise CTO, I've directed the purchase and heavy integration of their work for over a decade. That said, this type of post needs to be read with product marketing context in mind. > I think the most important aspect from it is the auto-routing On the contrary, in white paper studies auto-routing is shown to destroy the single largest token cost they found, curiously…
Do you know they are using pi or are you inferring it?
Re: Managing AI Coding Costs at Scale
#243> "Exponentially growing costs"
I don't think people thing don't think enough about the exponentially growing costs with AI. The cost of tokens, the cost of bugs, the cost of technical debt, the cost of software bloat and slowdown, the cost on the RAM market, on the environment, the costs of the little lies and hallucinations that creep into our codebases, blogs, books, and search results.
What seems like a manageable margin suddenly starts growing and destroying everything.
Re: Managing AI Coding Costs at Scale
#244I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…
In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered. When I personally want tight code, I have to spend a considerable amount of time adjusting it manually: - It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating - I need to consider whether there are solutions base…
I largely agree, but when this point is brought up I always feel the need to go one step further:
For the vast majority of software out there, this quality gap simply doesn't matter. Not everything is big tech, needing internet scale. Most software built by developers at non-tech companies are simple line-of-business apps used by a handful to a few dozen or maybe a few hundred people to do a thing a bit better than the Excel spreadsheet that was used before.
And AI has essentially automated the development of this kind of software. Or rather, it has dramatically increased the ability of skilled developers to produce larger amounts of these kinds of systems.
Excel hardest hit.
Re: Managing AI Coding Costs at Scale
#245Great article, I think the most important aspect from it is the auto-routing. As humans laziness is in our nature, so having to think if the model is capable enough is not something that most ppl will do - resulting in trying out smaller models which failed our task and then just giving up and running on the bigger model all the time.
DISCLOSURE: I like Databricks. While cosplaying enterprise CTO, I've directed the purchase and heavy integration of their work for over a decade. That said, this type of post needs to be read with product marketing context in mind. > I think the most important aspect from it is the auto-routing On the contrary, in white paper studies auto-routing is shown to destroy the single largest token cost they found, curiously…
If that’s such a large problem then the clear solution is to do sticky sessions?
What’s the problem then?
Use a cheap ds4 or Luna and do the second model net net per one shot best case you save couple dollars per?
Re: Managing AI Coding Costs at Scale
#246Earlier quoted context omitted.
In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered. When I personally want tight code, I have to spend a considerable amount of time adjusting it manually: - It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating - I need to consider whether there are solutions base…
I largely agree, but when this point is brought up I always feel the need to go one step further: For the vast majority of software out there, this quality gap simply doesn't matter. Not everything is big tech, needing internet scale. Most software built by developers at non-tech companies are simple line-of-business apps used by a handful to a few dozen or maybe a few hundred people to do a thing a bit better than t…
i don't think people writing this understand how computers work. there is no "Internet scale hard" problem.
its reliability, safety, efficiency problems. these problems exist at any scale.
ai sucks at it.
Re: Managing AI Coding Costs at Scale
#247Great article, I think the most important aspect from it is the auto-routing. As humans laziness is in our nature, so having to think if the model is capable enough is not something that most ppl will do - resulting in trying out smaller models which failed our task and then just giving up and running on the bigger model all the time.
[deleted]
Re: Managing AI Coding Costs at Scale
#248I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…
In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered. When I personally want tight code, I have to spend a considerable amount of time adjusting it manually: - It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating - I need to consider whether there are solutions base…
>> In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating
Which ones don't?
I used to play Wing Commander II. It was 30mb. Today, the software to adjust the colored lights on my keyboard requires literally a thousand times as much memory.
Re: Managing AI Coding Costs at Scale
#249This approach seems fundamentally predicated on being able to evaluate coding agents on your own code by having domain specific evals. With that knowledge, you can trust the routing logic is actually improving/maintaining perf while reducing costs. Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity I'm actually working on building gene…
The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to replicate. We found our own evals correlated loosely with public generic SWE benchmarks. In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing t…
Experimentation makes a lot of sense as agents need to be pleasant to work with, and power users are a willing and able cohort to do that differentiation. Agent capability is only part of the equation, as evidenced by the widespread frustration with the experience of working with Opus 5, despite it topping benchmarks and being objectively extremely powerful/competent.
However, as a self-admitted eval true believer, I remain bullish that publicly available eval capabilities will catch up to current model intelligence, and start to be able to capture more and more of the subtle components of the agent experience, such as interactivity and code maintainability. Meta and Scale have some evals in the interactivity space that seem pretty promising https://arxiv.org/abs/2606.29957 https://labs.scale.com/papers/swe-interact.
Human usability of the agent should definitely remain the top priority, but IMO we focus too much on the subjective, and not enough on the measurable/objective now, and I think/hope we'll see more companies move in the direction of Databricks
Re: Managing AI Coding Costs at Scale
#250Earlier quoted context omitted.
No surprise, LLM companies optimize for waste. More tokens, and more prompts means more revenue. Reminds of Google’s Prabhakar Raghavan story: deliberately making search worse [1] [1]: https://pluralistic.net/2024/04/24/naming-names/#prabhakar-r...
Or, more likely, it's that concise code requires a much deeper, wholistic, understanding that these models just are capable of yet. Same with a junior dev. They don't write long form spaghetti because they're trying to write more LOC. They do it because not doing it is hard , literally above their pay grade. I use LLM every day, but they're still completely awful at architecture. I don't think this clear lack of abil…
> I don't think this clear lack of ability is some conspiracy.
Maybe currently not. But we will never be able to know, as models are undeterministic and benchmarks are kind of scams. When you cannot prove that something gets worse, rest assured companies will to it.