Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Managing AI Coding Costs at Scale

databricks.com

Post reply on HN
Print view View original

131–140 of 268 posts

Re: Managing AI Coding Costs at Scale

#131

Post by dgellow »

Krei-se wrote: post #89
gamblor956 wrote: post #85

Earlier quoted context omitted.

but I produce the output of 3 or 4 2022 engineers and probably at better quality. Possibly, but the output of a 2022 engineer is about 1/10th of the output of a 2010 engineer, so it's an extremely low bar.

also - as always with these claims there's no actual product / repo / whatever one could check. I would love to see what these tools create but outside slop there's never: This works, is in production, here's the code. Any day now.

It’s crazy how we are like ~2y in this AI revolution and still do not have an answer to this question: can you show us the ROI? Where is the revolutionary software your team of agents created?

Re: Managing AI Coding Costs at Scale

#132

Post by mandeepj »

jvican wrote: post #105
ankitmathur wrote: post #67

Earlier quoted context omitted.

Omnigent and OpenRouter are different in the sense that OpenRouter is where you can go to call the actual model but Omnigent is intended to be the place where you go describe the high level task to be done, and work is farmed out to various harnesses and models. Those sandboxes can themselves be using OpenRouter for capacity! We're calling the layer coordinating harnesses "meta-harness'

Omnigent seems to compete more against Orca https://github.com/stablyai/orca They both went to be the Agent IDE layer, where you come with your tasks and everything is taken care of. I've been using Orca for a handful of tasks and have been largely enjoying it. My default barebones workflow is ghostty + zmx on ssh connections.

These tools casually like to claim they are orchestrators, but unfortunately, none of them are.

Re: Managing AI Coding Costs at Scale

#133

Post by pwendell »

behat wrote: post #112

Appreciate the detail in this and the previous post on creating internal benchmarks! Have you all attempted finetuning smaller OSS models on your repos for coding?

We do this for a lot of our customers (fine tuned to save cost when inference volume is high). Right now for internal coding we are using off-the-shelf models but we are considering fine tuning as well to squeeze more efficiency out.

Re: Managing AI Coding Costs at Scale

#134

Post by blcknight »

extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

$80 sounds extremely low for what you're describing - are you on API token plans?

I have had some $3,000 token days - even without Fable. I don't see how this is sustainable.

My personal 20x plans get so much usage for so cheap. The consumer subsidies are crazy, but alas I can't use them for work.

Re: Managing AI Coding Costs at Scale

#135

Post by dgellow »

What I take from this is that models are already commoditized, and it’s pretty clear nobody has a moat: routing for the models, they can be swapped whenever new models are released, AI labs will have to continue to run on the treadmill non stop or be replaced. Long term I cannot imagine that business will be high margin. Routing for the harness, so anything that differentiate a provider vs another isn’t exposed to the user and isn’t too relevant.

One more datapoint for the thesis that OpenAI and anthropic aren’t viable, sustainable businesses, and cannot justify their $1T valuation and the level of compute commitment (reminder that OpenAI committed to >$750B in infra spending for 2030)

Re: Managing AI Coding Costs at Scale

#136

Post by OrangeDelonge »

dgellow wrote: post #135

What I take from this is that models are already commoditized, and it’s pretty clear nobody has a moat: routing for the models, they can be swapped whenever new models are released, AI labs will have to continue to run on the treadmill non stop or be replaced. Long term I cannot imagine that business will be high margin. Routing for the harness, so anything that differentiate a provider vs another isn’t exposed to th…

Do you think Anthropic or OpenAI will eventually try to crack down on routing harnasses? Provide a more vertically integrated experience? They are already trying ro ship hardware products.

Re: Managing AI Coding Costs at Scale

#137

Post by ajcp »

extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

Only spending $80 a day on Opus 5/Fable 5/GPT 5.6 Sol feels very low. I'll roll through a couple hundred dollars worth of credits a day with those models, the vast majority of which would be on non-coding tasks, and it's still a huge cost savings over me or my team having to do these things manually, if we'd even be able to do them at all.

But that's also why it's now easy to justify the cost of an Nvidia or Intel inference server with Kimi K3 locked and loaded :)

Re: Managing AI Coding Costs at Scale

#138

Post by eitally »

pizza234 wrote: post #27
extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered. When I personally want tight code, I have to spend a considerable amount of time adjusting it manually: - It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating - I need to consider whether there are solutions base…

As a business user, the same thing is true for non-code documents. The biggest exertion is reducing the excessive slop down to concise, clear points.

Re: Managing AI Coding Costs at Scale

#139

Post by jknoepfler »

extr wrote: post #49
dieselgate wrote: post #45

Earlier quoted context omitted.

> The average SWE costs $200/hr This is a pointless quibble but the hourly rate claim is not true--it's like ~$60 in the USA [0]. Maybe you meant at a specific Org but this is important context when comparing "pricing" between human and AI. [0] https://www.salaryexpert.com/salary/job/software-developer/u...

How is this not true? Taking a Senior SWE @ ~$200K, even just the base salary cost / 2080 working hours is $100/hr. Fully loaded employer cost + accounting for non-coding time gets you to upper 100s easily. Even for a junior making $100K, I have a hard time believe their time is worth less than $75/hr or so. Edit: Fine, "Senior" is not "Average". But naive salary is not the true numerator.

I hire contractors for a large enterprise in the US. The going rate is typically $85-$100/hr for a senior dev, depending on specialization. Lead-level maybe $120 for the right skill set.

Re: Managing AI Coding Costs at Scale

#140

Post by extr »

Karrot_Kream wrote: post #125
sbarre wrote: post #111

Earlier quoted context omitted.

> You didn't say anything positively or negatively regarding this so I made an assumption that you were using the LLM relatively unguided I feel like this statement betrays your lack of advanced experience coding with LLMs. OP's elaboration of the steps they are going through (planning, agreeing on plan, getting one LLM to draft execution plan, approving it, then executing with a separate LLM, then reviewing/testing)…

Planning, agreeing on a plan, separating planning and implementation LLM, using separate review LLMs, these are all table stakes. This isn't "guidance" if you're getting paid to write software. If you think "unguided" means "I typed a prompt into claude code and waited yolo" I don't know what to say but, you have a very different idea of what professionals do than I do. I find for my own work that I need to read the…

> "unguided" means "I typed a prompt into claude code and waited yolo"

Yes, this is literally what that means.

Post reply on HN