I this the only way to escape this trap is by developing better LLMs in the future. The rapid rate at which new AI-generated code is produced means that humans will no longer be able to review it all.
That's why you have other LLMs review.
Comprehension debt: A ticking time bomb of LLM-generated code
91–100 of 352 posts
Re: Comprehension debt: A ticking time bomb of LLM-generated code
#92This was a pre-existing problem, even if reliance on LLMs is making it worse. Naur ( https://gwern.net/doc/cs/algorithm/1985-naur.pdf ) called it "theory building": > The death of a program happens when the programmer team possessing its theory is dissolved. A dead program may continue to be used for execution in a computer and to produce useful results. The actual state of death becomes visible when demands for modi…
> The actual state of death becomes visible when demands for modifications of the program cannot be intelligently answered. Yeah but we can ask an LLM to read the code and write documentation, if that happens.
Good documentation also contains the "why" of the code, ie why it is the way it is and not one of the other possible ways to write the same code. That is information inherently not present in the code, and there would be no way for a LLM to figure it out after the fact.
Also, no "small" program is ever at risk of dying in the sense that Naur describes it. Worst case, you can re-read the code. The problem lies with the giant enterprise code bases of the 60s and 70s where thousands of people have worked on it over the years. Even if you did have good documentation, it would be hundreds of pages and reading it might be more work than just reading the code.
Re: Comprehension debt: A ticking time bomb of LLM-generated code
#93At my first programmer job, a large majority of the code was written by a revolving door of interns allowed to push to main with no oversight. Much of the codebase was unknown and irreplaceable, which meant it slowly degraded and crumbled over the years. Even way back then, everyone knew the entire project was beyond salvage and needed to be rewritten from scratch.
Well they kept limping along with that mess for another ten years while the industry sprinted ahead. The finally released a new product recently, but I don't think anyone cares because everyone else did it better five years ago
Re: Comprehension debt: A ticking time bomb of LLM-generated code
#94This was a pre-existing problem, even if reliance on LLMs is making it worse. Naur ( https://gwern.net/doc/cs/algorithm/1985-naur.pdf ) called it "theory building": > The death of a program happens when the programmer team possessing its theory is dissolved. A dead program may continue to be used for execution in a computer and to produce useful results. The actual state of death becomes visible when demands for modi…
I think this is part of the reason why I've had a bit more success with AI Coding than some of my colleagues. My pre-llm workflow was to rapidly build a crappy version of something so that I could better understand it, then rework it (even throw away to the prototype) to build something I now know how I want to handle. I've found even as plenty of thought leaders talk about this general approach (rapid prototyping, continuous refactoring, etc ...) that many engineers are resistant and want to think through the approach and then build it "right." Or alternatively just whip something out and don't throw it away, but rather toil on fixes to the their crappy first pass.
With AI this loop is much easier. It is cheap to even build 3 parallel implementations of something and maybe another where you let the system add whatever capability it thinks would be interesting. You can compare and use that to build much stronger "theory of the program" with requirements, where the separation of concerns are, how to integrate with the larger system. Then having AI build that, with close review of the output (which takes much less time if you know roughly what should be being built) works really well.
Re: Comprehension debt: A ticking time bomb of LLM-generated code
#95So many of these concepts only make sense under the assumption that AI will not get better and humans will continue to pour over code by hand. They won't. In a year or two these will be articles that get linked back to similar to "Is the internet just a fad?" articles of the late 90s.
I disagree. They Not every technological advance "improves" at an exponential rate.
The issue is that LLMs don't "understand." They merely copy without contributing original thought or critical thinking. This is why LLMs can't handle complicated concepts in codebases.
What I think we'll see in the long run is:
(Short term) Newer programming models that target LLMs: IE, describe what you want the computer to do in plain English, and then the LLM will allow users to interact with the program in a more conversational manner. Edit: These will work in "high tolerance" situations where small amounts of error is okay. (Think analog vs digital, where analog systems tend to tolerate error more gracefully than digital systems.)
(Long term) Newer forms of AI that "understand." These will be able to handle complicated programs that LLMs can't handle today, because they have critical thinking and original thought.
Re: Comprehension debt: A ticking time bomb of LLM-generated code
#96When was the golden age, when everybody understood how everything worked?
Re: Comprehension debt: A ticking time bomb of LLM-generated code
#97This was a pre-existing problem, even if reliance on LLMs is making it worse. Naur ( https://gwern.net/doc/cs/algorithm/1985-naur.pdf ) called it "theory building": > The death of a program happens when the programmer team possessing its theory is dissolved. A dead program may continue to be used for execution in a computer and to produce useful results. The actual state of death becomes visible when demands for modi…
Fully agree.
I was once on a project where all the original developers suddenly disappeared and it was taken over by a new team. All institutional knowledge had been lost.
We spent a ridiculous amount of time trying to figure out the original design. Introduced quite a few bugs until it was better understood. But also fixed a lot of design issues after a much head bashing.
By the end, it had been mostly rewritten and extended to do things not originally planned.
But the process was painful.
Re: Comprehension debt: A ticking time bomb of LLM-generated code
#98A friend was recently telling me about an LLM'd PR he was reviewing submitted by a largely non-technical manager where the feature from the outside entirely appeared to work, but actually investigating the thousands of lines of generated code, it was instead hacking their response cache system to appear to work without actually updating anything on the backend. It took a ton of effort on his part to convince his mana…
Where are these non-technical engineering managers and how did they stay in the business?
I haven't seen a truly non-technical manager in over 15 years.
Re: Comprehension debt: A ticking time bomb of LLM-generated code
#99This was a pre-existing problem, even if reliance on LLMs is making it worse. Naur ( https://gwern.net/doc/cs/algorithm/1985-naur.pdf ) called it "theory building": > The death of a program happens when the programmer team possessing its theory is dissolved. A dead program may continue to be used for execution in a computer and to produce useful results. The actual state of death becomes visible when demands for modi…
Debt has always existed, and "LLMs is making it worse" Yes, I think point is, LLM's are making it 'a-lot' worse. And then compounding that will be in 10 years when no Senior Devs were being created, so nobody will be around to fix it. Extreme of course, there will be dev's, they'll just be under-water, piled on with trying to debug the LLM stuff.
>they'll just be under-water, piled on with trying to debug the LLM stuff.
So in that theory the senior devs of those days will still be able to command large salaries if they know their stuff, in specific how to untangle the mess of LLM code.
Re: Comprehension debt: A ticking time bomb of LLM-generated code
#100Earlier quoted context omitted.
Just like how in a year or two we will have fully self-driving cars, right? The last percentage point for something to get just right are the hardest, why are you so sure that the flaws in LLMs will be gone in such a short time frame?
We do have fully self-driving cars. You can go to a number of American cities and take a nap in the backseat of one while it drives you around safely.
[deleted]