Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Managing AI Coding Costs at Scale

databricks.com

Post reply on HN
Print view View original

191–200 of 268 posts

Re: Managing AI Coding Costs at Scale

#191

Post by pcox »

pianopatrick wrote: post #165
roncesvalles wrote: post #148

Earlier quoted context omitted.

Since software is still a winner-takes-all market, the mass-production property of software doesn't really matter. In such markets, what you produce is either worth nothing or worth millions of dollars. For as long as it's the case that well-constructed code (with or without LLM help) is more likely to be in the latter category, the economics of software don't really change. Even before LLMs, you could've commissione…

I think part of the reason software was winner take most was the difficulty of making software. I remember hearing a story that in the past movies were so technically difficult to make that any movie that got made had a good chance to be a profitable hit. But as movies got cheaper to make, more movies got made. Nowadays movie studio execs have to really calculate out the audience and expected revenue for any new movi…

> I think part of the reason software was winner take most was the difficulty of making software.

That might be part of it, but I think it also has to do with the reality of replicating and scaling. Hardware or physical goods simply don’t scale like digital goods. There can be hundreds of knock-off physical products that have lower quality and lower cost but serve 90% of the same purpose, because physical capacity for raw materials, construction, labor, shipping, etc. have scaling limits in each market and economy. Digital goods are just so much easier to replicate and scale, so it often doesn’t make sense to buy software at lower quality and lower price if it doesn’t do most of the job. There are still limits of course, and different from physical goods, but I think this is a key reason why software is seen as winner-take-all.

Re: Managing AI Coding Costs at Scale

#192

Post by ipaddr »

throw-the-towel wrote: post #116
alfalfasprout wrote: post #109

Earlier quoted context omitted.

There's a massive difference between your kids vibe coding something and an engineer using AI to implement something. If you're unable to discern the difference, that's something to reflect on :)

It doesn't matter if GP is able to discern the difference, it matters if your CEO is forced to care about the difference.

CEOs will always want someone who knows to implement so we're safe from kids vibe coding their way in but in a short while it becomes AI who knows who is managing less expensive AI.

Re: Managing AI Coding Costs at Scale

#193

Post by tfehring »

extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

I'm also at a startup. My workflow is similar but I have Fable 5 xhigh drive the whole thing: it gets Codex CLI installed in its environment with an API key, and it's instructed to delegate ~everything to Codex and review its work, especially for code quality/conciseness. Fable delegates to Sol or Luna (fast mode) xhigh/max depending on the task - I think Luna xhigh on fast mode is basically a Pareto improvement over Sol medium.

Re: Managing AI Coding Costs at Scale

#194

Post by pianopatrick »

pcox wrote: post #191
pianopatrick wrote: post #165

Earlier quoted context omitted.

I think part of the reason software was winner take most was the difficulty of making software. I remember hearing a story that in the past movies were so technically difficult to make that any movie that got made had a good chance to be a profitable hit. But as movies got cheaper to make, more movies got made. Nowadays movie studio execs have to really calculate out the audience and expected revenue for any new movi…

> I think part of the reason software was winner take most was the difficulty of making software. That might be part of it, but I think it also has to do with the reality of replicating and scaling. Hardware or physical goods simply don’t scale like digital goods. There can be hundreds of knock-off physical products that have lower quality and lower cost but serve 90% of the same purpose, because physical capacity fo…

Yes, but there are other digital goods, like music, movies and books which are not quite as hard to make as software. In those you have a hit driven market dynamic with lots of niches instead of winner take all.

I.e. because software was hard to make and complex to copy you would tend to have "natural monopolies" that were hard to compete with. Who wants to try to build a new desktop OS to compete with Windows? Or a web browser from scratch? Or a new search engine? Etc.

Those and other pieces of software were complex and hard to make. The cost to copy and compete was very high. So one winner took most because that winner was the company who could figure that software out.

But as we can see with Kimi Work and other such things, software is now much easier to copy. Let's say it took $1 billion to make a copycat piece of software with people but now takes $100 million or $10 million with AI. Suddenly a copy and compete tactic makes much more sense than before

For example, with AI it might make financial sense to build a Chinese Native OS instead of Windows. Similarly for Russia, Iran, the EU, and a whole bunch of other places. All of a sudden, Windows might not be the winner take most OS, we might have lots of Operating Systems, with smaller markets and lower profits, which require much more careful financial analysis to stay profitable.

This would be just like Movies, TV Shows, Books or Music. When something works, people relentlessly copy it and different regions put their own spin on the idea. After Iron Man succeeded we had so many super hero movies. Etc. So there is not really a winner take most dynamic in these other digital products. Software may be moving that way

Re: Managing AI Coding Costs at Scale

#195

Post by willsmith72 »

extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

> I probably spend like $80 a day

Wait what? I don't understand these numbers. I spend $1k/day

Your story about being told to use AI for everything I was expecting you to be well over that

Re: Managing AI Coding Costs at Scale

#196

Post by williamstein »

thelastgallon wrote: post #182

> nearly every company deploying AI tools at scale has hit the same wall: exponentially growing costs. That curve is unsustainable - left unchecked it will eventually overtake revenue. But the question remains, AI hasn't shown any explosion in growth/revenue for most companies. The effects seem muted.

If only one company had AI and nobody else did, then that one company would be expected to start seeing an explosion in growth/revenue. As it is, everybody has access to AI, so extensive use of AI should just result in keeping up with the competition. The ROI to spending on AI is "not dying".

Re: Managing AI Coding Costs at Scale

#197

Post by mrlongroots »

extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

In my experience, code is a small fraction of the work.

I'm in an infra team and for the last 2 weeks or so I've been trying to understand whether a particular workload will catch fire if a switch is flicked. I'm also new to the team so partly it is me wearing training wheels, familiarizing myself with the telemetry etc, but I will state that I'm not completely lousy at this stuff.

No model in my experience can do anything remotely comparable to the work "what happens to the workload if this switch is flicked" needs. They can't even design a reliable quick experiment to answer what cast should be applied to the binary trace_id in table A for the join to table B to work. They will happily do something idiotic and then conclude that the join does not work.

Re: Managing AI Coding Costs at Scale

#198

Post by Karrot_Kream »

skinfaxi wrote: post #179
Karrot_Kream wrote: post #125

Earlier quoted context omitted.

Planning, agreeing on a plan, separating planning and implementation LLM, using separate review LLMs, these are all table stakes. This isn't "guidance" if you're getting paid to write software. If you think "unguided" means "I typed a prompt into claude code and waited yolo" I don't know what to say but, you have a very different idea of what professionals do than I do. I find for my own work that I need to read the…

> If you think "unguided" means "I typed a prompt into claude code and waited yolo" I don't know what to say but, you have a very different idea of what professionals do than I do. What exactly does "unguided" mean to you, then?

Not having human input in the loop, i.e. allowing agents to act without guidance. I understand the idea of having agents guide agents, but really how much do we gain when Sol scolds Fable?

Re: Managing AI Coding Costs at Scale

#199

Post by lilbigdoot »

nomel wrote: post #177
frevib wrote: post #56

Earlier quoted context omitted.

No surprise, LLM companies optimize for waste. More tokens, and more prompts means more revenue. Reminds of Google’s Prabhakar Raghavan story: deliberately making search worse [1] [1]: https://pluralistic.net/2024/04/24/naming-names/#prabhakar-r...

Or, more likely, it's that concise code requires a much deeper, wholistic, understanding that these models just are capable of yet. Same with a junior dev. They don't write long form spaghetti because they're trying to write more LOC. They do it because not doing it is hard , literally above their pay grade. I use LLM every day, but they're still completely awful at architecture. I don't think this clear lack of abil…

Personal anecdote: I spent a few days hacking on my compiler to remove 1k lines of code (about 15% of total code) while preserving behavior

I was only able to do that after I had solved multiple related problems in different places and started introducing subtle bugs by accident / had difficulty detecting all edge cases

I've noticed whenever I use LLMs they introduce the same kind of thing but at much smaller scales than I would. They often suggest solving the wrong problem when I prompt them to diagnose specific bugs too. Usually opting for a shortcut that introduces its own issues and ironically calling the proper direction "too complex" when it's really not.

Re: Managing AI Coding Costs at Scale

#200

Post by gabrigoo »

wxw wrote: post #12

> Rapidly adopting newer, more efficient models delivers the largest cost wins of any technique. I think the more interesting lever is the fourth they mention: token efficiency. > By the time costly LLM inference occurs, the user's initial statement accounts for only a negligible fraction of the data fed into the AI system, meaning costs are dominated by context the user did not explicitly include. I think there’s st…

[dead]
Post reply on HN