Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Managing AI Coding Costs at Scale

databricks.com

Post reply on HN
Print view View original

251–260 of 269 posts

Re: Managing AI Coding Costs at Scale

#251

Post by bonoboTP »

frevib wrote: post #56
pizza234 wrote: post #27

Earlier quoted context omitted.

In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered. When I personally want tight code, I have to spend a considerable amount of time adjusting it manually: - It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating - I need to consider whether there are solutions base…

No surprise, LLM companies optimize for waste. More tokens, and more prompts means more revenue. Reminds of Google’s Prabhakar Raghavan story: deliberately making search worse [1] [1]: https://pluralistic.net/2024/04/24/naming-names/#prabhakar-r...

Bullshit. You can be cynical, it's fine but this is just nonsense. People have so much demand for coding that there is no need to make it waste tokens. People are eager to implement more features, do more testing, more platforms, more file formats, bla bla. There is absolutely no incentive to waste tokens. They can barely serve the demanded tokens anyway. There is in fact incentive to save tokens, so subsidized subscriptions don't consume as much.

Re: Managing AI Coding Costs at Scale

#252

Post by drTobiasFunke »

swader999 wrote: post #236
drTobiasFunke wrote: post #169

Earlier quoted context omitted.

Output of 3 or 4 2022 engineers? Its that your self assessment? Output as in number of lines of code?

Not op, but we are at 2400 total points delivered over seven years. 1000 of those in the last six months. About 2-4 devs over that period, just two the last six months.

Noone is questioning the volume of llm output. My question is whether all those points delivered improved your product and software in any meaningful way, or do you now have 20x more code that noone understands with the same quality of software product?

Re: Managing AI Coding Costs at Scale

#253

Post by patrickmay »

ianmarcinkowski wrote: post #234

Maybe these larger models are better at producing concise, internally-consistent, non-polluting (of the wider codebase) features. I just spent 3 days reviewing the code of a vibe coding maximalist at work. They needed to touch 96 front-end files, ~50 back-end files. We now have `is_uk_company` spread throughout the entire codebase every time we need to output a date format. I would say 30% of the code can be replaced…

> Just the date fields is a hilarious example of why I'm skeptical. Python (babel.date) and Javascript (the browser's locale) both have locale-aware date rendering. Using a library for country-based date and time parsing would have eliminated ~12 country-specific change sites... but instead we got `is_uk` and raw date format strings (`%d/%m/%y`) in-line inside already-enormous functions. Also, at a meta level, the backend code shouldn't usually be formatting the dates (data export may be an exception), we should be sending ISO dates and having the frontend format the code if anyone gives a crap.

Did you reject the PR or is that not a career-enhancing approach in your organization?

Re: Managing AI Coding Costs at Scale

#254

Post by lukasco »

With self-driving agents, the costs stop being evident as you go, and show up after the fact. I've been building governors that slow the agents down, and then also look at odd things some of the harnesses do, such as constantly making mistakes in tool calls.

But overall, it's quite a task, and you really have to decide what you are prioritizing for. Do I want my agents doing lots of work, or (in my case), leaving some of my subscriptions available for me to do work.

Re: Managing AI Coding Costs at Scale

#255

Post by roncesvalles »

sevenzero wrote: post #218
roncesvalles wrote: post #148

Earlier quoted context omitted.

Since software is still a winner-takes-all market, the mass-production property of software doesn't really matter. In such markets, what you produce is either worth nothing or worth millions of dollars. For as long as it's the case that well-constructed code (with or without LLM help) is more likely to be in the latter category, the economics of software don't really change. Even before LLMs, you could've commissione…

>Since software is still a winner-takes-all market, the mass-production property of software doesn't really matter. This could not be further from the truth. It might be true if the software works without supervisors in a B2C scenario where everything is self explanatory and nothing can go wrong, movie streaming services for example, and even these have tons of variants and multiple big players racking up billions in…

"Winner-takes-all" in the software context is never meant literally. It's more like, an oligarchy of a few winners will emerge out of hundreds of contenders.

It's not like restaurants where for example you can open a "Larry's Pizzeria" and still build a valuable business despite a Domino's nearby.

Re: Managing AI Coding Costs at Scale

#257

Post by desbo »

“agentic coding has measurably improved every velocity metric we track and, in some teams, driven an order-of-magnitude gains in output”

Of course it has. But is that a good enough measure of value? What about the business and customer outcomes from all the extra lines of code and closed JIRA tickets? Was it worth it?

Re: Managing AI Coding Costs at Scale

#258

Post by z0mghii »

ianmarcinkowski wrote: post #231
mrlongroots wrote: post #197

Earlier quoted context omitted.

In my experience, code is a small fraction of the work. I'm in an infra team and for the last 2 weeks or so I've been trying to understand whether a particular workload will catch fire if a switch is flicked. I'm also new to the team so partly it is me wearing training wheels, familiarizing myself with the telemetry etc, but I will state that I'm not completely lousy at this stuff. No model in my experience can do an…

An AI maximalist on my team put up 2 pull requests with ~120-140 changed files this week. If they spent $200 on tokens to achieve this, we spent $2500-3500 in human salary and opportunity cost reviewing it.

You need to change the way you think about reviews

Re: Managing AI Coding Costs at Scale

#259

Post by myegorov »

salmonfamine wrote: post #102

I think there is a lot of dev cope in this thread. My workflow is very simple: 1. develop requirements for code change 2. take manual notes for implementation, maybe use LLM for some discovery/investigation 3. present notes to frontier LLM 4. develop implementation plan (bulk of work) 5. let LLM rip 6. review diff, manually fixing/refactoring code as necessary, sometimes prompting for revisions 7. get automated LLM r…

My workflow as well. If I had to gauge the direction of the wind, this produces the work of somewhere between 1-2 pre-AI engineers. My own code quality probably went down, along with my focus and the depth of understanding. Bug rates have increased. I don't have any stats to prove this.

Re: Managing AI Coding Costs at Scale

#260

Post by myegorov »

redhale wrote: post #244
pizza234 wrote: post #27

Earlier quoted context omitted.

In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered. When I personally want tight code, I have to spend a considerable amount of time adjusting it manually: - It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating - I need to consider whether there are solutions base…

I largely agree, but when this point is brought up I always feel the need to go one step further: For the vast majority of software out there, this quality gap simply doesn't matter. Not everything is big tech, needing internet scale. Most software built by developers at non-tech companies are simple line-of-business apps used by a handful to a few dozen or maybe a few hundred people to do a thing a bit better than t…

> For the vast majority of software out there, this quality gap simply doesn't matter.

You don't mean the majority of paid for software can behave non-deterministically? That's the value add of the human in the loop.

Post reply on HN