Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Managing AI Coding Costs at Scale

databricks.com

Post reply on HN
Print view View original

101–110 of 268 posts

Re: Managing AI Coding Costs at Scale

#101

Post by steve_adams_86 »

app13 wrote: post #33
extr wrote: post #26

Earlier quoted context omitted.

Performance is better than ever. It's never been more practical to set up wildly complex synthetic test environments and measure perf wins. Plus the models will find every possible algorithmic/design improvement. It actually gives me quite an uncanny feeling, bulldozing over years of human optimization work with a newer, "perfect" design. Like bringing an AK-47 back to the middle ages.

I needed to thoroughly test rerankers on my companies rather unique corpus. Opus and I wrote a parallelized test harness and labeled groundtruth in around 2 hours. In 2022 that would've likely been all I did for a couple sprints

I encounter this regularly and it still feels weird.

That sense that you did something better in a few days than you would have in a month 5 years ago. It's like buying a table saw for wood working.

One crazy thing I think about often is how there are so many correctness and testing harnesses that would have taken weeks to build in the past so we simply never would have. We'd just do our best then wait and see what comes to the surface. This is a huge part of what makes it possible to actually make better software with LLMs in my opinion. It isn't just 'LLM codes better than I ever could' (that's often untrue still) but 'LLM enables me to make assertions about the program to degrees that would have been absurdly impractical in the past'. It's huge

Re: Managing AI Coding Costs at Scale

#102

Post by salmonfamine »

I think there is a lot of dev cope in this thread.

My workflow is very simple:

1. develop requirements for code change

2. take manual notes for implementation, maybe use LLM for some discovery/investigation

3. present notes to frontier LLM

4. develop implementation plan (bulk of work)

5. let LLM rip

6. review diff, manually fixing/refactoring code as necessary, sometimes prompting for revisions

7. get automated LLM review

8. get human review

this reliably produces the work of 2-3 pre-AI senior engineers with a lower bug rate, equivalent performance, robust edge-case consideration, etc.

Does the LLM produce over-engineered solutions? All the time. I stop it from doing that, or manually fix it myself.

Does the LLM always adhere to the best system design? No, not at all. I often have to guide its design into a better, north-star aligned one.

I don't just sit in front of my terminal and say, "Ok Claude, build the app." It is a very iterative process, and not without its potential pitfalls.

But it is very, very productive.

Re: Managing AI Coding Costs at Scale

#103

Post by vonneumannstan »

reqo wrote: post #21
extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targ…

Wow how many years of experience with Claude Code and Codex do you have? lol

Re: Managing AI Coding Costs at Scale

#104

Post by slopinthebag »

extr wrote: post #37
pizza234 wrote: post #27

Earlier quoted context omitted.

In our team's experience, the product of agents is generally The Homer (1). It does work, but it's vastly overengineered. When I personally want tight code, I have to spend a considerable amount of time adjusting it manually: - It needs to be trimmed down. In my experience, at least one agent I use struggles to produce minimalist designs, and it's very frustrating - I need to consider whether there are solutions base…

This was more true a few months ago but Fable has improved the situation considerably. Also just remember - minimalist code looks and feels great but customers do not read your code. I have caught myself many times providing "corrections" to abstractions that were already ~fine, just not perfect. The average SWE costs $200/hr. Careful you don't burn $50 worrying about code that will likely be rewritten or can be bett…

The average SWE makes $400k a year? Are you being serious?

Re: Managing AI Coding Costs at Scale

#105

Post by jvican »

ankitmathur wrote: post #67
mandeepj wrote: post #54

Earlier quoted context omitted.

Omniagent looks quite similar to OpenRouter ( https://openrouter.ai/ )

Omnigent and OpenRouter are different in the sense that OpenRouter is where you can go to call the actual model but Omnigent is intended to be the place where you go describe the high level task to be done, and work is farmed out to various harnesses and models. Those sandboxes can themselves be using OpenRouter for capacity! We're calling the layer coordinating harnesses "meta-harness'

Omnigent seems to compete more against Orca https://github.com/stablyai/orca They both went to be the Agent IDE layer, where you come with your tasks and everything is taken care of. I've been using Orca for a handful of tasks and have been largely enjoying it. My default barebones workflow is ghostty + zmx on ssh connections.

Re: Managing AI Coding Costs at Scale

#106

Post by hunterpayne »

nujabe wrote: post #90
gamblor956 wrote: post #87

Earlier quoted context omitted.

"startup" and "legacy codebase" are diametrically opposed concepts. And if you're saying (based on your other comments) that a 6 month window is enough to create a legacy codebase...that indicates a serious lack of experience or understanding as to what a legacy codebase is, or why they exist.

exactly. Usually legacy code forms when people lose context and confidence in parts of the codebase due to staff turnover etc and ppl avoid touching or enhancing those parts for long periods. Six months is a short time to accrue that much tech debt, its enough time where most of the people who created that "legacy" are probably still around. As you said indicates bigger problems.

So basically any LLM codebase of sufficient size is immediately legacy.

Re: Managing AI Coding Costs at Scale

#107

Post by sarchertech »

extr wrote: post #62
nujabe wrote: post #44

Earlier quoted context omitted.

No, but not relevant. What is the point of working at a startup if you’re dealing with millions of lines of legacy code ? Isn’t the whole point of startups to create & innovate with a clean slate and modern tools?

No, actually. The point is to build a profitable business.

How long has your startup been around? I’ve worked at plenty of startups over the past 20 years. Including one that was still calling themselves a startup 10 years out. The org I work at now was a startup before my tech giant employer acquired them. We have a very bloated and very profitable 8 year old codebase that is barely 500k LOC.

I’ve never seen a startup with a multi million line legacy codebase.

Re: Managing AI Coding Costs at Scale

#108

Post by hunterpayne »

vonneumannstan wrote: post #103
reqo wrote: post #21

Earlier quoted context omitted.

IME this works until it does not. This approach works well at the beginning of a greenfield project, but at the same time because it is so easy to add features, you will likely ship something that is way too over engineered. And that complexity will not amortize over next increments and will more likely lead to the entire project being a black box only fully understood by AI. However a more careful use of AI for targ…

Wow how many years of experience with Claude Code and Codex do you have? lol

The job requires 10 years of those technologies ;)

Re: Managing AI Coding Costs at Scale

#109

Post by alfalfasprout »

jacquesm wrote: post #95
j-bos wrote: post #50

Earlier quoted context omitted.

> The average SWE costs $200/hr. And this is how I find out I'm woefully underpaid.

Whatever you are making this year as SWE you'll be making less next year if the current trend in improvement of AI coding aids is going to be sustained. Think about it : programmers used to derive a lot of their value from the fact that it was a hard skill to acquire. My kids can now 'vibe code' stuff faster (and better looking) than what I could come up with as the beginnings of a design plan. And then I still need…

There's a massive difference between your kids vibe coding something and an engineer using AI to implement something. If you're unable to discern the difference, that's something to reflect on :)

Re: Managing AI Coding Costs at Scale

#110

Post by K3UL »

extr wrote: post #16

I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fa…

The output yes, but do you produce the impact and value of 3 engineers? I have seen this workflow being toyed with too, and I find it to produce massively overengineered stuff that actual people don't really wanna use
Post reply on HN