Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Managing AI Coding Costs at Scale

databricks.com

Post reply on HN
Print view View original

61–70 of 281 posts

Re: Managing AI Coding Costs at Scale

#61

Post by extr »

Karrot_Kream wrote: post #48
extr wrote: post #38

Earlier quoted context omitted.

Have you worked at many startups?

Something isn't clear about the size of your codebase here and the level of reliability your customers expect, as a reader of your comments. Clarity there will help. My observation has been: - Initial greenfield work by an LLM is fast and very effective with minimal or no human oversight. - Subsequent work ends up being over engineered and very verbose. Assumptions are made that aren't suited to the problem at hand (…

> unguided LLM usage

Why aren't you guiding your LLM usage? Is that what I said - to spam it and not guide anything? Or to have a careful workflow where you agree on design and maximize your human judgement/leverage?

> any state that's not explicitly being tested and verified in QA loops

As opposed to before, when engineers perfectly reasoned about code behavior from first principals and QA was unnecessary?

Re: Managing AI Coding Costs at Scale

#62

Post by extr »

nujabe wrote: post #44
extr wrote: post #38

Earlier quoted context omitted.

Have you worked at many startups?

No, but not relevant. What is the point of working at a startup if you’re dealing with millions of lines of legacy code ? Isn’t the whole point of startups to create & innovate with a clean slate and modern tools?

No, actually. The point is to build a profitable business.

Re: Managing AI Coding Costs at Scale

#63

Post by grigri907 »

the_sleaze_ wrote: post #28
biophysboy wrote: post #18

Earlier quoted context omitted.

Do you have tips for generating clean productive output per dollar?

in my humble experience it boils down to mastery. Are you at least conversational in the subject matter? You're gonna have a good time just by paying attention and adjusting your workflow. If you're getting a lot of back and forth with it, its asking a lot of planning type questions, stop, step back, rethink the whole feature, and start again from the beginning with everything more fleshed out. If you are in a brand…

I appreciate this non-judgmental description of what it's like to approach a topic/technology from a newcomer's perspective. Thanks!

Re: Managing AI Coding Costs at Scale

#65

Post by ankitmathur »

wxw wrote: post #12

> Rapidly adopting newer, more efficient models delivers the largest cost wins of any technique. I think the more interesting lever is the fourth they mention: token efficiency. > By the time costly LLM inference occurs, the user's initial statement accounts for only a negligible fraction of the data fed into the AI system, meaning costs are dominated by context the user did not explicitly include. I think there’s st…

100% - there's a lot to learn from traces from real-life sessions with coding tools! For example, I found it pretty eye-opening to see how wide the distribution of tasks truly is. There's also subtle things like how a poorly designed MCP API surface can cause a massive amount of token waste from the model just iterating on finding the right way to call it.

Re: Managing AI Coding Costs at Scale

#66

Post by extr »

nujabe wrote: post #41
extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

> essentially unlimited AI spend budget > I probably spend like $80 a day This doesn’t sound like “unlimited”, I spend more than this out of pocket per day and I have a strict budget.

It's a fair point, it's not truly unlimited and I do wonder how that would change my workflow. I can definitely imagine if I was inside Anthropic or OAI with unlimited "fast" tokens, you would be more tempted to hand over even more of this process. I completely understand why they talk about "graph engineering" and such, my entire workflow above could be a graph and I could try to increase my leverage even further. Realistically though I am bounded by product decision making, not code output right now.

Re: Managing AI Coding Costs at Scale

#67

Post by ankitmathur »

mandeepj wrote: post #54
mjuarez wrote: post #15

Earlier quoted context omitted.

It seems Databricks open-sourced it a while ago: https://www.databricks.com/blog/introducing-omnigent-meta-ha... https://github.com/omnigent-ai/omnigent

Omniagent looks quite similar to OpenRouter ( https://openrouter.ai/ )

Omnigent and OpenRouter are different in the sense that OpenRouter is where you can go to call the actual model but Omnigent is intended to be the place where you go describe the high level task to be done, and work is farmed out to various harnesses and models. Those sandboxes can themselves be using OpenRouter for capacity!

We're calling the layer coordinating harnesses "meta-harness'

Re: Managing AI Coding Costs at Scale

#68

Post by chis »

It’s funny how different everyone’s experience is with this stuff. To me the diminishing returns are more around not going crazy with prototyping or running with xmax thinking all the time. I haven’t found it hard to stay under the usage limit of one $200/mo Claude and one $200/mo Codex subscription.

If my company told me yeah we’ve decided you don’t get Fable or Opus 5 because it’s too pricey, you gotta use GLM whatever, I’d be displeased.

Re: Managing AI Coding Costs at Scale

#69

Post by shay_ker »

how do any of these routing approaches handle kv cache misses? Devin Fusion is the only one that explicitly addresses this, though it does so by switching models during compaction (not sure this isn't still a cache miss though)
Post reply on HN