Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The current state of LLM-driven development

blog.tolki.dev

Post reply on HN
Print view View original

171–180 of 244 posts

Re: The current state of LLM-driven development

#171

Post by SkyPuncher »

deadbabe wrote: post #162
SkyPuncher wrote: post #148

Earlier quoted context omitted.

> The last 20%, while possible to attain, is ultimately not worth it for the amount of time you spend in context hells. You can just do it yourself faster. I'm arguing that there's a skill that has to be learned in order to break through this. As you start in a new code base, you should be quick to jump in when you hit that 20%. But, as you spend more time in it, you learn how to avoid the same "context hell" issues…

It’s not incredibly powerful, it’s incrementally powerful. Getting the first 80% via LLM is already the incredible power. A sufficiently skilled developer should be able to handle the rest with ease. It is not worth doing anything unnatural in an effort to chase down the last 20%, you are just wasting time and atrophying skills. If you can get full 95% in some one shot prompts, great. But don’t go chasing waterfallls…

No, it actually has an exponential growth type of effect on productivity to be able to push it to the boundary more.

I’m making this a bit contrived, but I’m simplifying it to demonstrate the underlying point.

When an LLM is 80% effect, I’m limits to doing 5 things in parallel since I still need to jump in 20% of the time.

When an LLM is 90% effect, I can do 10 things at once. When it’s 95%, 20 things. 99%, 100 things.

Now, obviously I can’t actually juggle 10 or 20 things at once. However, the point is there are actually massive productivity gains to be had when you can reduce your involvement in a task from 20% to, even 10%. You’re effectively 2x as productive.

Re: The current state of LLM-driven development

#172

Post by skydhash »

Tainnor wrote: post #165
skydhash wrote: post #147

Earlier quoted context omitted.

I'll bite. What are those things that they are good for? And consistently so?

As someone who leans more towards the side of LLM-sceptiscism, I find Sonnet 4 quite useful for generating tests, provided I describe in enough detail how I want the tests to be structured and which cases should be tested. There's a lot of boilerplate code in tests and IMO because of that many developers make the mistake of DRYing out their test code so much that you can barely understand what is being tested anymore…

Isn’t tests supposed to be premises (ensure initial state is correct), compute (run the code), and assertions (verify the result state and output). If your test code is complex, most of it should be moved into harness and helpers functions. Writing more complex code isn’t particularly useful.

Re: The current state of LLM-driven development

#173

Post by Jensson »

SkyPuncher wrote: post #171
deadbabe wrote: post #162

Earlier quoted context omitted.

It’s not incredibly powerful, it’s incrementally powerful. Getting the first 80% via LLM is already the incredible power. A sufficiently skilled developer should be able to handle the rest with ease. It is not worth doing anything unnatural in an effort to chase down the last 20%, you are just wasting time and atrophying skills. If you can get full 95% in some one shot prompts, great. But don’t go chasing waterfallls…

No, it actually has an exponential growth type of effect on productivity to be able to push it to the boundary more. I’m making this a bit contrived, but I’m simplifying it to demonstrate the underlying point. When an LLM is 80% effect, I’m limits to doing 5 things in parallel since I still need to jump in 20% of the time. When an LLM is 90% effect, I can do 10 things at once. When it’s 95%, 20 things. 99%, 100 thing…

Do you understand what parallel means? Most LLM responds in seconds, there is no parallel work for you to do there.

Or do you mean you are using long running agents to do tasks and then review those? I haven't seen such a workflow be productive so far.

Re: The current state of LLM-driven development

#174

Post by myhf »

JimDabell wrote: post #121
hiAndrewQuinn wrote: post #73

>I made a CLI logs viewers and querier for my job, which is very useful but would have taken me a few days to write (~3k LoC) I recall The Mythical Man-Month stating a rough calculation that the average software developer writes about 10 net lines of new, production-ready code per day. For a tool like this going up an order of magnitude to about 100 lines of pretty good internal tooling seems reasonable. OP sounds a…

A lot of the Mythical Man-Month is timeless, but for a stat like that, it really is worth bearing in mind the book was written half a century ago about developers working on 1970s mainframes.

Yeah, I think that metric has grown to about 20 lines per day using 2010s-era languages and methods. So maybe we could think of LLM usage as an attempt to bring it back down to 10 per day.

Re: The current state of LLM-driven development

#175

Post by dogcomplex »

hn_throwaway_99 wrote: post #168
thefourthchime wrote: post #159

Earlier quoted context omitted.

It entirely depends on the exposure and reliability the code needs. Some code is just a one-off to show a customer what something might look like. I don't care at all how well the code works or what it looks like for something like that. Rapid prototyping is a valid use case for that. I have also written a C++ code that has to have a runtime of years, meaning there can be absolutely no memory leaks or bugs whatsoever…

> It entirely depends on the exposure and reliability the code needs. Ahh, sweet summer child, if I had a nickel for every time I've heard "just hack something together quickly, that's throwaway code", that ended up being a critical lynchpin of a production system - well, I'd probably have at least like a buck or so. Obviously, to emphasize, this kind of thing happens all the time with human-generated code, but LLMs…

> Ahh, sweet summer child, if I had a nickel for every time I've heard "just hack something together quickly, that's throwaway code", that ended up being a critical lynchpin of a production system - well, I'd probably have at least like a buck or so.

Because this is the first pass on any project, any component, ever. Design is done with iterations. One can and should throw out the original rough lynchpin and replace it with a more robust solution once it becomes evident that it is essential.

If you know that ahead of time and want to make it robust early, the answer is still rarely a single diligent one-shot to perfection - you absolutely should take multiple quick rough iterations to think through the possibility space before settling on your choice. Even that is quite conducive to LLM coding - and the resulting synthesis after attacking it from multiple angles is usually the strongest of all. Should still go over it all with a fine toothed comb at the end, and understand exactly why each choice was made, but the AI helps immensely in narrowing down the possibility space.

Not to rag on you though - you were being tongue in cheek - but we're kidding ourselves if we don't accept that like 90% of the code we write is rough throwaway code at first and only a small portion gets polished into critical form. That's just how all design works though.

Re: The current state of LLM-driven development

#176

Post by dogcomplex »

throwawaybob420 wrote: post #140

Judging from all the comments here, it’s going to be amazing seeing the fallout of all the LLM generated code in a year or so. The amount of people who seemingly relish the ability to stop thinking and let the model generate giant chunks of their code base, is uh, something else lol.

lol yep we've never had codebases hacked together by juniors before running major companies in production - nope, never

Re: The current state of LLM-driven development

#177

Post by KallDrexx »

dogcomplex wrote: post #175
hn_throwaway_99 wrote: post #168

Earlier quoted context omitted.

> It entirely depends on the exposure and reliability the code needs. Ahh, sweet summer child, if I had a nickel for every time I've heard "just hack something together quickly, that's throwaway code", that ended up being a critical lynchpin of a production system - well, I'd probably have at least like a buck or so. Obviously, to emphasize, this kind of thing happens all the time with human-generated code, but LLMs…

> Ahh, sweet summer child, if I had a nickel for every time I've heard "just hack something together quickly, that's throwaway code", that ended up being a critical lynchpin of a production system - well, I'd probably have at least like a buck or so. Because this is the first pass on any project, any component, ever. Design is done with iterations. One can and should throw out the original rough lynchpin and replace…

I would love to work at the places you have been where you are given enough time to throw out the prototype and do it properly. In my almost 20 years of professional experience this has never been the case and prototype and exploratory code has only been given minimal polishing time before reaching production and in use state.

Re: The current state of LLM-driven development

#178

Post by deadbabe »

SkyPuncher wrote: post #171
deadbabe wrote: post #162

Earlier quoted context omitted.

It’s not incredibly powerful, it’s incrementally powerful. Getting the first 80% via LLM is already the incredible power. A sufficiently skilled developer should be able to handle the rest with ease. It is not worth doing anything unnatural in an effort to chase down the last 20%, you are just wasting time and atrophying skills. If you can get full 95% in some one shot prompts, great. But don’t go chasing waterfallls…

No, it actually has an exponential growth type of effect on productivity to be able to push it to the boundary more. I’m making this a bit contrived, but I’m simplifying it to demonstrate the underlying point. When an LLM is 80% effect, I’m limits to doing 5 things in parallel since I still need to jump in 20% of the time. When an LLM is 90% effect, I can do 10 things at once. When it’s 95%, 20 things. 99%, 100 thing…

I’d bet you don’t even have 2 or 3 things to do at once, much less 100. So it’s pointless to chase those types of coverages.

Re: The current state of LLM-driven development

#179

Post by KallDrexx »

simonw wrote: post #48
troupo wrote: post #23

Earlier quoted context omitted.

> I have never heard anybody successfully using LLMs say this before. Most of what I've learned from talking to people about their workflows is counterintuitive and subtle. Because for all our posturing about being skeptical and data driven we all believe in magic. Those "counterintuitive non-trivial workflows"? They work about as well as just prompting "implement X" with no rules, agents.md, careful lists etc. Becau…

The problem with your argument here is that you're effectively saying that developers (like myself) who put effort into figuring out good workflows for coding with LLMs are deceiving themselves, and are effectively wasting their time. Either I've wasted significant chunks of the past ~3 years of my life or you're missing something here. Up to you to decide which you believe. I agree that it's hard to take solid measu…

I'm not the OP and I"m not saying you are wrong, but I am going to point out that the data doesn't necessarily back up significant productivity improvements with LLMs.

In this video (https://www.youtube.com/watch?v=EO3_qN_Ynsk) they present a slide by the company DX that surveyed 38,880 developers across 184 organizations, and found the surveyed developers claiming a 4 hour average time savings per developer per week. So all of these LLM workflows are only making the average developer 10% more productive in a given work week, with a bunch of developers getting less. Few developers are attaining productivity higher than that.

In this video by stanford researchers actively researching productivity using github commit data for private and public repositories (https://www.youtube.com/watch?v=tbDDYKRFjhk) they have a few very important data points in there:

1. There's zero correlation they've found between how productive respondants claim their productivity is and how it's actually measured, meaning people are poor judges of their own productivity numbers. This does refute the claims on the previous point I made but only if you assume people are wildly more productive then they claim on average.

2. They have been able to measure actual increase in rework and refactoring commits in the repositories measured as AI tools become more in use in those organizations. So even with being able to ship things faster, they are observing increase number of pull requests that need to fix those previous pushes.

3. They have measured that greenfield low complexity systems have pretty good measurements for productivity gains, but once you get more towards higher complexity systems or brownfield systems they start to measure much lower productivity gains, and even negative productivity with AI tools.

This goes hand in hand with this research paper: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... which had experienced devs in significant long term projects lose productivity when using AI tools, but also completely thought the AI tools were making them even more productivity.

Yes, all of these studies have their flaws and nitpicks we can go over that I'm not interested in rehashing. However, there's a lot more data and studies that show AI having very marginal productivity boost compared to what people claim than vice versa. I'm legitimately interested in other studies that can show significant productivity gains in brownfield projects.

Re: The current state of LLM-driven development

#180

Post by troupo »

azan_ wrote: post #156
troupo wrote: post #66

Earlier quoted context omitted.

The "learning curve" is reading "experts opinion" on the ever-changing set of magical rituals that may or may not work but trust us it works.

No, you do not need to trust anyone, you can just verify what works and what doesn't, it's very easy.

Indeed. And it's extremely easy to verify my original comment: https://news.ycombinator.com/item?id=44849887
Post reply on HN