Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI can code, but it can't build software

bytesauna.com

Post reply on HN
Print view View original

61–70 of 185 posts

Re: AI can code, but it can't build software

#61

Post by cadamsdotcom »

manwe150 wrote: post #58
CjHuber wrote: post #44

Earlier quoted context omitted.

Can you maybe give an example you’ve encountered of an algorithm or a data structure that LLMs cannot handle well? In my experience implementing algorithms from a good comprehensive description and keeping track of data models is where they shine the most.

Converting an algorithm implementation from recursive to iterative: it got the concept broadly right, but was quite bad at making the logic actually match up, often refusing to fix mistakes or reverting fixes two edits later. Still a positive experience though, since it was fixable issues and reduced the amount of tedious copies I had to type

Did you have it write tests and give it the ability to iterate & validate its implementation without you in the loop?

Anything less is setting it up for failure...

Re: AI can code, but it can't build software

#62

Post by simonw »

prmph wrote: post #56
crazygringo wrote: post #38

Earlier quoted context omitted.

> The people saying LLM can code are hard for me to understand. Just today, I spent an hour documenting a function that performs a set of complex scientific simulations. Defined the function input structure, the outputs, and put a bunch of references in the body to function calls it would use. I then spent 15 minutes explaining to the free version of ChatGPT what the function needs to do both in scientific terms and…

> documenting a function that performs a set of complex scientific simulations. The example you gave sounds like the problem is deterministic, even if composed of many moving parts. That's one way of looking at complexity. When I talk about complex problems I'm not just talking about intricate problems. I'm talking about problems where the "problem" is design, not just implementing a design, and that is where LLMs st…

For the problems like that I consider my role to be the expert designer. I figure out a the design, then get the LLM to write the code and the tests for me.

It is a lot faster at typing than I am.

Re: AI can code, but it can't build software

#63

Post by prmph »

CjHuber wrote: post #44
prmph wrote: post #27

Earlier quoted context omitted.

> LLMs are remarkably good at writing code. Just this past weekend, I've designed and written code (in Typescript) that I don't think LLMs can even come close to writing in years. I have a subscription to a frontier LLM, but lately I find myself using like 25% of the time. At a certain level the software architecture problems I'm solving, drawing upon decades of understanding about maintainable, performant, and verif…

Can you maybe give an example you’ve encountered of an algorithm or a data structure that LLMs cannot handle well? In my experience implementing algorithms from a good comprehensive description and keeping track of data models is where they shine the most.

Example (expanding on [1]): I want to design a strongly typed fluent API interface to some role/permissions based authorization engine functionality. Even knowing how to shape the fluent interface so that is powerful but intuitive, as strongly typed as possible but also and maintainable, is a deep art.

One reason I know LLM can't come close to my design is this: I've written something that works (that a typical senior engineer might write), but this not enough. I have evaluated it critically (drawing on my experience with long lived software), rewritten it again to better meet the targets above, and repeated this process several times. I don't know what would make an LLM go: now that kind of works, but is this the most intuitive, well typed, and maintainable design that there could be?

1. https://news.ycombinator.com/item?id=45728183

Re: AI can code, but it can't build software

#64

Post by manwe150 »

cadamsdotcom wrote: post #61
manwe150 wrote: post #58

Earlier quoted context omitted.

Converting an algorithm implementation from recursive to iterative: it got the concept broadly right, but was quite bad at making the logic actually match up, often refusing to fix mistakes or reverting fixes two edits later. Still a positive experience though, since it was fixable issues and reduced the amount of tedious copies I had to type

Did you have it write tests and give it the ability to iterate & validate its implementation without you in the loop? Anything less is setting it up for failure...

Yes, but it got 99% of those then got stuck on why the others made no sense to it

Re: AI can code, but it can't build software

#65

Post by bloppe »

carlosjobim wrote: post #55
bloppe wrote: post #29

Earlier quoted context omitted.

> I'm paying much less attention for quality now. After all, why bother when AI produce working code? I hear this so much. It's almost like people think code quality is unrelated to how well the product works. As though you can have 1 without the other. If your code quality is bad, your product will be bad. It may be good enough for a demo right now, but that doesn't mean it really "works".

> If your code quality is bad, your product will be bad. Why? Modern hardware power allow for extremely inefficient code, so even if some code runs a thousand times slower because it's badly programmed it will still be so fast that it seems instant. For the rest of the stuff, it has no relevance for the user of the software what the code is doing inside of the chip, as long as the inputs and outputs function as they…

Sure. Everyone remembers from Algorithms 101 that a constant multiple ("a thousand times slower") is irrelevant. What matters is the scalability. Something that's O(n) will always scale better than something that O(n^2), even if the thing that's O(n) has 1000x overhead per unit.

But that's just a small piece of the puzzle. I agree that the user only cares about what the product does and not how the product works, but the what is always related to how, even if that relationship is imperceptible to the user. A product with terrible code quality will have more frequent and longer outages (because debugging is harder), and it will take longer for new features to be added (because adding things is harder). The user will care about these things.

Re: AI can code, but it can't build software

#66

Post by thegrim33 »

And here I am, using AI twice within the last 12 hours, to ask it two questions about an extremely well used, extremely well documented, physics library, and both times having it return to me sample code which makes use of library methods which don't exist. When I tell it this, I get the "Oh, you're so right to point that out!" response, and get new code returned, which still just blatantly doesn't work.

Re: AI can code, but it can't build software

#67

Post by falcor84 »

echelon wrote: post #60
roxolotl wrote: post #13

Earlier quoted context omitted.

One of the interesting corollaries of the title is that this can also be true of humans. Being able to code is not the same as being a software engineer. It never has been.

We're also finding this true with media generation. AI video is an incredible tool, but it can't make movies. It's almost as if all of these models are an exoskeleton for people that already know what they're doing. But you still need an expert in the loop.

> but it can't make movies.

To me this appears to be a very time-dependent assertion. 5 years ago, AI couldn't generate a good movie frame. 2 years ago, AI couldn't generate a good shot, but now in 2025, AI can generate a not-too-shabby scene. If capabilities continue improving at this rate (e.g. as they have with AI being able to generate full musical albums), I wouldn't bet against AI being able to generate a decent feature film in the next decade. It might take longer until it's the sort of thing that we'd present in festivals, but I just don't a clear barrier any more.

Looking at it from another perspective, if an AI driven task currently requires "an expert in the loop" to navigate things by offering the appropriate prompts, evaluating and iterating on the AI generated content, then there's nothing clear to stop us from training the next generation of AI to include that expert's competency.

Taking it into full extrapolation mode, the thing that current generation AIs really don't have is the human experience that leads to a creative drive, but once we have robotic agents among us, these would arguably be able start gathering "experiences" that they could then mine to write and produce "their own" stories.

Re: AI can code, but it can't build software

#68

Post by simonw »

DougWebb wrote: post #50
crazygringo wrote: post #38

Earlier quoted context omitted.

> The people saying LLM can code are hard for me to understand. Just today, I spent an hour documenting a function that performs a set of complex scientific simulations. Defined the function input structure, the outputs, and put a bunch of references in the body to function calls it would use. I then spent 15 minutes explaining to the free version of ChatGPT what the function needs to do both in scientific terms and…

> Just today, I spent an hour documenting a function that performs a set of complex scientific simulations. Defined the function input structure, the outputs, and put a bunch of references in the body to function calls it would use. So that's... math. A very well defined problem, defined very well. Any decent programmer should be able to produce working software from that, and it's great that ChatGPT was able to help…

> Only a tiny subset of software development projects are like that though.

Right: the majority of software development is things like "build a REST API for these three database tables" or "build a contact form with these four fields" or "write unit tests for this new function" or "update my YAML CI configuration to run this extra command".

Re: AI can code, but it can't build software

#69

Post by abhishekismdhn »

Even the code quality is often quite poor. At the same time, not using critical thinking can have serious consequences for those who treat AI as more than an explorer or companion. You might think that with AI, the number of highly skilled developers would increase but it could be quite the opposite. Code is just a medium; developers are paid to solve problems, not to write code. But writing code is still important as it refines your thoughts and sharpens your problem-solving skills.

The human brain learns through mistakes, repetition, breaking down complex problems into simpler parts, and reimagining ideas. The hippocampus naturally discards memories that aren’t strongly reinforced.. so if you rely solely on AI, you’re simply not going to remember much.

Re: AI can code, but it can't build software

#70

Post by simonw »

Bukhmanizer wrote: post #48
simonw wrote: post #4

This is a good headline. LLMs are remarkably good at writing code. Writing code isn't the same thing as delivering working software. A human expert needs to identify the need for software, decide what the software should do, figure out what's feasible to deliver, build the first version (AI can help a bunch here), evaluate what they've built, show it to users, talk to them about whether it's fit for purpose, iterate…

> the big question is if experienced product management types can pick up enough coding technical literacy to work like this without programmers I have a strong opinion that AI will boost the importance of people with “special knowledge” more than anyone else regardless of role. So engineers with deep knowledge of a system or PMs with deep knowledge of a domain.

That sounds right to me.
Post reply on HN