Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codegen is not productivity

antifound.com

Post reply on HN
Print view View original

71–80 of 103 posts

Re: Codegen is not productivity

#71

Post by ip26 »

ChicagoDave wrote: post #68
greggyb wrote: post #64

Earlier quoted context omitted.

The post is about using LOC as a metric when making any sort of point about AI. Nowhere do I suggest someone shouldn't use it, nor that they should expect negative results if they opt to.

No one I’ve ever worked with in 40 years has ever seriously used loc as a measurement of progress or success. I honestly don’t know where this comes from.

That’s the odd psychosis here. Everyone knew loc was a terrible measure. But perhaps the instinctual pull was always there, and now that you can generate halfway coherent tens of thousands of loc in hours, our sensibilities are overwhelmed.

Re: Codegen is not productivity

#72

Post by greggyb »

demorro wrote: post #66
greggyb wrote: post #45

Earlier quoted context omitted.

I didn't set out to teach you anything, change your behavior, or give you practical takeaways, so it's a rant (: Emotions can be expressed with citations. I am fully on board with gen AI representing a paradigm shift in software development. I tried to be careful not to take a stance on other debates in the larger conversation. I just saw too many people talking about how much code they're generating as proof stateme…

I guess I over-diagnosed your stance, apologies. I wonder if you have a take on measuring productivity in light of the potential difficulty of achieving good outcomes across the general population? You mention in the second appendix (which I skipped on my first read), that you are a rather experienced LLM user, with experiences in all the harnesses and context management which are touted as "best practice" nowadays.…

Measuring productivity in software development is a hard problem, beyond the typical categorizations used in computer science. Unfortunately, I think my best answer is to go read the book I linked in the conclusion: https://link.springer.com/chapter/10.1007/978-1-4842-4221-6_...

That is an unsatisfying answer. I can point to anecdotes that suggest AI is hurting productivity or improving it, but those don't make an argument. And the extremes on either side make it very difficult to consider. How do you weigh "An LLM deleted my production database" against "I built a business on the back of AI-assisted software"?

I think we have to wait and see. And we should revisit questions of cost and value continuously, not just about LLMs, but generally in life. Most of my motivation (though not an overwhelming majority) around using LLMs right now is a mix of curiosity and wanting to avoid the fate of the steam shovel.

Re: Codegen is not productivity

#73

Post by demorro »

jwilliams wrote: post #50
demorro wrote: post #28

Earlier quoted context omitted.

Would you entertain the idea that "work was never the bottleneck", or even "building products was never the bottleneck"? We need to address Jevons' Paradox somehow.

I love Jevons’ paradox too, but if we apply it here don’t we still end up with more software? Definitely would entertain -- I do agree with your framing. I just think the article undersells the impact of fast+cheap codegen. Lowering the cost of implementation will (has) expose new bottlenecks elsewhere. But imho many of those bottlenecks probably weren’t worth serious investment in solving before. The codegen change…

I think that's where a heck of a lot of the frustration on this topic is coming from. Some engineers claim to have solved the code generation issue well enough that it hasn't been the bottleneck in their local environment, and have been trying to pivot to widening the new bottlenecks for a while now, but have been confounded by organisational dynamics.

To see the other bottlenecks starting to be taken seriously now, but (if I'm to be petulant) all the "credit" of solving the code bottleneck being taken by LLM systems, it's painful, especially when you are in a local domain where the code gen bottleneck doesn't matter very much and hasn't for a long time.

I suspect engineers that managed to solve the code generation bottlenecks are compulsive problem solvers, which exacerbates the issue.

That isn't to say there are some domains where it still does matter, although I'm dubious that LLM codegen is the best solve, but I am not dubious that it is at least a solve.

Re: Codegen is not productivity

#74

Post by slopinthebag »

It's so difficult to quantify productivity over an entire field, especially when it's so vast. Chris Lattner recently concluded this about LLM tooling [0]:

> AI systems can internalize the textbook knowledge of a field and apply it coherently at scale. AI can now reliably operate within established engineering practice. This is a genuine milestone that removes much of the drudgery of repetition and allows engineers to start closer to the state of the art.

This matches my experience, there is a lot of code that we probably should not need to write and rewrite anymore but still do because this field has largely failed at deriving complete and reusable solutions to trivial problems - there is a massive coordination problem that has fragmented software across the stack and LLMs provide one way of solving it by generating some of the glue and otherwise trivial but expensive and unproductive interop code required.

But the thing about productivity is that it's not one thing and cannot be reduced to an anecdote about a side-project, or a story about how a single company is introducing (or mandating) AI tooling, or any single thing. Being able to generate a bunch of code of varying quality and reliability is undeniably useful, but there are simply too many factors involved to make broad sweeping claims about an entire industry based on a tool that is essentially autocomplete on crack. Thus it's not surprising that recent studies have not validated the current hype cycle.

[0] https://www.modular.com/blog/the-claude-c-compiler-what-it-r...

Re: Codegen is not productivity

#75

Post by jwpapi »

ip26 wrote: post #69
ChicagoDave wrote: post #61

I continue to jump into these discussions because I feel like these upvoted posts completely miss what’s happening… - guardrails are required to generate useful results from GenAI. This should include clear instructions on design patterns, testing depth, and iterative assessments. - architecture decision records are one useful way to prevent GenAI from being overly positive. - very large portions of code can be compl…

No one should care about actual code ever again. It’s ephemeral. Caveat: it still works best in a codebase that is already good. So while any one line of code is ephemeral, how is the overall codebase trending? Towards a bramble, or towards a bonsai? If the software is small and not mission critical, it doesn’t matter if it becomes a bramble, but not all software is like that.

I think it works great in codebases that are good, but I think it will degrade the quality of the codebase compared to what it was before.

A good codebase depends on the business context, but in my case its an agile one that can react to discovered business cases. I’ve written great typed helpers that practically allow me to have typed mongo operators for most cases. It makes all operations really smooth. AI keeps finding cretaive ways of avoiding my implementations and over time there are more edge cases, thin wrappers, lint ignore comments and other funny exceptions. Whilst I’m losing the guarantees I built...

Re: Codegen is not productivity

#76

Post by greggyb »

ChicagoDave wrote: post #68
greggyb wrote: post #64

Earlier quoted context omitted.

The post is about using LOC as a metric when making any sort of point about AI. Nowhere do I suggest someone shouldn't use it, nor that they should expect negative results if they opt to.

No one I’ve ever worked with in 40 years has ever seriously used loc as a measurement of progress or success. I honestly don’t know where this comes from.

Yes, but it comes up in conversations of LLMs a lot. Thus, the rant in question. I think we are in agreement, or at least we lack disagreement, because that is the only stance I endeavored to take in the post.

Re: Codegen is not productivity

#77

Post by galbar »

This article describes the body of knowledge I was taught when I joined the industry and parallels with my experience with and thoughts about AI.

I have come to the realization that most people in the industry don't know this body of knowledge, or even that it exists.

I'm now seeing the same people trying to solve their ineffectiveness with AI.

I don't know what to think about this situation. My intuition hints at it not being good.

Re: Codegen is not productivity

#78

Post by slopinthebag »

ChicagoDave wrote: post #61

I continue to jump into these discussions because I feel like these upvoted posts completely miss what’s happening… - guardrails are required to generate useful results from GenAI. This should include clear instructions on design patterns, testing depth, and iterative assessments. - architecture decision records are one useful way to prevent GenAI from being overly positive. - very large portions of code can be compl…

> - No one should care about actual code ever again. It’s ephemeral. The role of software engineering is now molding features and requirements into functional results. Choosing Rust, C#, Java, or Typescript might matter depending on the domain, but then you stop caring and focus on measuring success.

I think this has always been the case. "Bad programmers worry about the code. Good programmers worry about data structures and their relationships." Perhaps you mean that they shouldn't worry about structures & relationships either but I think that is a fools errand. Although to be fair neither of those need to be codified in the code itself, but ignore those at your own peril...

Re: Codegen is not productivity

#79

Post by felipellrocha »

nubg wrote: post #27

For me it's simple: 1. Assume you're to work on product/feature X. 2. If God were to descend and give you a very good, reality-tested spec: 3. Would you be done faster? Of course, because as every AI doomer says, writing code was never the bottleneck!!1! 4. So the only bottleneck is getting to the spec. 5. Guess what AI can help you with as well, because you can iterate out multiple versions with little mental effort…

Man, if this were true we’d see a crazy, massive explosion of quality products being written, and launched. While we see some use, i don’t perceive an acceleration. In fact, i see a lot of trivial bugs being deployed to prod.

Re: Codegen is not productivity

#80

Post by slopinthebag »

nubg wrote: post #27

For me it's simple: 1. Assume you're to work on product/feature X. 2. If God were to descend and give you a very good, reality-tested spec: 3. Would you be done faster? Of course, because as every AI doomer says, writing code was never the bottleneck!!1! 4. So the only bottleneck is getting to the spec. 5. Guess what AI can help you with as well, because you can iterate out multiple versions with little mental effort…

4 doesn't follow from 3
Post reply on HN