Comprehension debt: A ticking time bomb of LLM-generated code

Curious about the mechanics here — when you say the model was ‘trained on our code base’, was that an actual fine-tune of the weights (e.g. LoRA/adapter or full SFT), or more of a retrieval/indexing setup where the model sees code snippets at inference? Always interested in how teams distinguish between the two.

People always wring their hands that operating at a new, higher level of abstraction with destroy people's ability to think and reason.

But people still think and reason just fine, but now at a higher level that gives them greater power and leverage.

Do you feel like you're missing something when you "cook for yourself" but you didn't you didn't plant and harvest the vegetables, raise and butcher the protein, forge the oven, or generate the gas or electricity that heats it?

You also didn’t write the CPU microcode or the compiler that turns your code into machine language.

When you cook or code, you're already operating on top of a very tall stack of abstractions.

Offshore coding practices in the 2010's is the same thing as LLM. Id take LLM over offshore 10/hr devs any day of the week...

> I.e., understanding a code base, the algorithms, etc.?

The big problem is that LLMs do not *understand* the code you tell them to "explain". They just take probabilistic guesses about both function and design.

Even if "that's how humans do it too", this is only the first part of building an understanding of the code. You still need to verify the guess.

There's a few limitations using LLMs for such first-guessing: In humans, the built up understanding feeds back into the guessing, as you understand the codebase more, you can intuit function and design better. You start to know patterns and conventions. The LLM will always guess from zero understanding, relying only on the averaged out training data.

A following effect is that which bunderbunder points out in their reply: while LLMs are good at identifying algorithms, mere pattern recognition, they are exceptionally bad at world-modelling the surrounding environment the program was written in and the high level goals it was meant to accomplish. Especially for any information obtained outside the code. A human can run a git-blame and ask what team the original author was on, an LLM cannot and will not.

This makes them less useful for the task. Especially in any case where you intent to write new code; Sure, it's great that the LLM can give basic explanations about a programming language or framework you don't know, but if you're going to be writing code in it, you'd be better off taking the opportunity to learn it.

I'm so glad someone has finally described the phenomena so well.

"Comprehension debt" is a perfect description for the thing I've been the most concerned about with AI coding.

One I got past the Dunning-Kruger phase and started really looking at what was being generated, I ran into this comprehension issue.

With a human, even a very junior one, you can sort of "get in the developer's head". You can tell which team member wrote which code and what they were thinking at the time. This leads to a narrative, or story of execution which is mostly comprehensible.

With the AI stuff, it's just stochastic parrot stuff. It may work just fine, but there will be things like random functions that are never called, hundreds or thousands of lines of extra code to do very simple things. References to things that don't exist, and never have.

I know this stuff can exist in human code bases too - but generally I can reason about why. "Oh, this was taken out for this issue and the dev forgot to delete it".

I can track it, even if it's poor quality.

With the AI stuff, it's just randomly there. No idea why, if it is used, was ever used, makes sense, is extra fluff or brilliant.

It takes a lot of work to figure out.