Comprehension debt: A ticking time bomb of LLM-generated code • Hacker News Board

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#311

Post by int_19h » Wed, Oct 01, 2025, 12:30 AM UTC

blackhaj7 wrote: post #222

alexpotato wrote: post #156

Was listening to the Dwarkesh Patel podcast recently and the guest (Agustin Lebron) [0] mentioned the book "A Deepness In The Sky" by Vernor Vinge [1]. I started reading it and a key plot point is that there is a computer system that is thousands of years old. One of the main characters has "cold sleeped" for so long that he's the only one who knows some of the hidden backdoors. That legacy knowledge is then used to…

Sounds great - thanks for the recommendation. Looks like it is the second in a trilogy. Can you just dive in or did you read the first book before?

The first two books can be treated largely as standalone works. They do technically take place in the same broad universe, but said universe is basically divided into FTL and non-FTL zones with vastly different societies in each (for obvious reasons), and the non-FTL societies aren't even aware of this boundary. "Fire upon the Deep" is set mostly in the FTL zone, with the boundary itself being a major plot point. "Deepness in the Sky" is set entirely in the non-FTL zone, and the lack of FTL is a major plot point there.

Chronologically, DitS takes place before FotD. But there is exactly one character in common between the two books, and while he is a major character in both, none of the events of DitS are relevant to the story in FotD (which makes sense since FotD was written first).

So it's really largely a matter of preference as to which one to read first. I would say that FotD has more action and, for the lack of better term, "weirdness" in the setting; while DitS is more slow-paced, with more character development and generally more fleshed-out characters, and explores its themes deeper. But both books have plenty for your mind to chew on.

All in all I think FotD is an easier read, and DitS is a more rewarding one, but this is all very subjective.

One upside to the books being decoupled as much as they are is that whichever one you start with, you get a complete story, so even if you're a completionist you can disregard the other book if you don't like the first one.

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#312

Post by int_19h » Wed, Oct 01, 2025, 12:43 AM UTC

redsymbol wrote: post #136

low_tech_punk wrote: post #118

Most programmers don't understand the low level assembly or machine code. High level language becomes the layer where human comprehension and collaboration happens. LLM is pushing that layer towards natural language and spec-driven development. The only *big* difference is that high level programming languages are still deterministic but natural language is not. I'm guessing we've reached an irreducible point where t…

There is another big difference: natural languages have ambiguity baked in. If a programming language has any ambiguity in how it can be parsed, that is rightly considered a major bug. But it's almost a feature of natural languages, allowing poetry, innuendo, and other nuanced forms of communication.

There are constructed languages that preserve the expressivity of natural human languages but without the implicit ambiguity, though; most notably, Loglan and its successor Lojban. If you read Golden Age sci-fi, Loglan sometimes shows up there specifically in this role - e.g. "Moon is a Harsh Mistress":

> By then Mike had voder-vocoder circuits supplementing his read-outs, print-outs, and decision-action boxes, and could understand not only classic programming but also Loglan and English, and could accept other languages and was doing technical translating—and reading endlessly. But in giving him instructions was safer to use Loglan. If you spoke English, results might be whimsical; multi-valued nature of English gave option circuits too much leeway.

For those unfamiliar with it, it's not that Lojban is perfectly unambiguous. It's that its design strives to ensure that ambiguity is always deliberate by making it explicit.

The obvious problem with all this is that Lojban is a very niche language with a fairly small corpus, so training AI on it is a challenge (although it's interesting to note that existing SOTA models can read and write it even so, better than many obscure human languages). However, Lojban has the nice property of being fully machine parseable - it has a PEG grammar. And, once you parse it, you can use dictionaries to construct a semantic tree of any Lojban snippet.

When it comes to LLMs, this property can be used in two ways. First, you can use structured output driven by the grammar to constrain the model to output only syntactically valid Lojban at any point. Second, you can parse the fully constructed text once it has been generated, add semantic annotations, and feed the tree back into the model to have it double-check that what it ended up writing means exactly what it wanted to mean.

With SOTA models, in fact, you don't even need the structured output - you can just give them parser as a tool and have them iterate. I did that with Claude and had it produce Lojban translations that, while not perfect, were very good. So I think that it might be possible, in principle, to generate Lojban training data out of other languages, and I can't help but wonder what would happen if you trained a model primarily on that; I suspect it would reduce hallucinations and generally improve metrics, but this is just a gut feel. Unfortunately this is a hypothesis that requires a lot of $$$ to properly test...

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#313

Post by [deleted] » Wed, Oct 01, 2025, 12:44 AM UTC

alexpotato wrote: post #261

mock-possum wrote: post #245

Earlier quoted context omitted.

His description of learning to be a programmer in that far future era was fun too, iirc there was just so much ‘legacy code’, like practically infinite libraries and packages to perform practically any function - that ‘coding’ was mostly a matter of finding the right pieces and wiring them together. Knowing the nuances of these existing pieces and the subtlety of their interpretation was the skill.

100% Another great example: In Fire Upon the Deep, due to the delay in communications between star systems, everyone use a descendant of Usenet.

[deleted]

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#314

Post by int_19h » Wed, Oct 01, 2025, 12:48 AM UTC

ModernMech wrote: post #234

rAum wrote: post #219

Earlier quoted context omitted.

lol you should absolutely merge it and go with it in such cases, just collect evidence first to have enough deniability and enjoy the show. You can tell a child not to do the thing over and over or just accept it will very quickly learn for their life that touching hot oven is not a smart thing to do. With so much AI hype induced brainrot seems for certain individuals the only antitode is to make them feel direct con…

That's only an option when it's not you who will have to clean up the mess.

Even if it's not you, it's still ethically questionable. It won't be the manager in question who will have to deal with the fallout, even in the long run - it will be another software engineer. It would ultimately contribute to this:

https://www.scottsmitelli.com/articles/altoids-by-the-fistfu...

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#315

Post by int_19h » Wed, Oct 01, 2025, 12:50 AM UTC

OutOfHere wrote: post #126

donatj wrote: post #71

A friend was recently telling me about an LLM'd PR he was reviewing submitted by a largely non-technical manager where the feature from the outside entirely appeared to work, but actually investigating the thousands of lines of generated code, it was instead hacking their response cache system to appear to work without actually updating anything on the backend. It took a ton of effort on his part to convince his mana…

I would report the manager to the CTO or CEO or business owners/investors.

You mean, the very people who keep doubling down on investments into AI combined with layoffs? You'd go and tell them that this thing that they signed off on, pitched to others, and thus are ultimately responsible for if it fails in a way that cannot be denied or covered up, is not working.

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#316

Post by int_19h » Wed, Oct 01, 2025, 12:52 AM UTC

ToValueFunfetti wrote: post #15

carlmr wrote: post #6

Earlier quoted context omitted.

>However, code that is well-designed by humans tends to be easier to understand than LLM spaghetti. Additionally you may have institutional knowledge accessible. I can ask a human and they can explain what they did. I can ask an LLM, too and they will give me a plausible-sounding explanation of what they did.

I can't speak for others, but if you ask me about code I wrote >6 months ago, you'll also be stuck with a plausible-sounding explanation. I'll have a better answer than the LLM, but it will be because I am better at generating plausible-sounding explanations for my behavior, not because I can remember my thought processes for months.

This is where stuff like git history often comes in handy. I cannot always reliably explain why some code was the way it is when looking at a single diff of my own from years ago, but give me the history of that file and the issue tracker where I can look up references from commits and see the comments etc, and I can reconstruct it with very high degree of certainty.

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#317

Post by int_19h » Wed, Oct 01, 2025, 12:56 AM UTC

mcoliver wrote: post #189

The counterpoint to this is that LLMs cannot only write code, they can comprehend it! They are incredibly useful for getting up to speed on a new code base and transferring comprehension from machine to human. This of course spans all job functions and is still immature in its accuracy but rapidly approaching a point where people with an aptitude for learning and asking the right questions can actually have a decent…

This would be great if said comprehension is reliable. But I've seen tools designed to "understand" and document repos hallucinate many times, often coming up with a plausible but completely wrong explanation of how things actually work, or, even more subtly, of why they work the way they work.

And while I could catch that because I wrote the code in question and know the answers to those questions, others do not have that benefit. The notion that someone new to the codebase - especially a relatively unexperienced dev - would have AI "documentation" as a starting point is honestly quite terrifying, and I don't see how it could possibly end with anything other than garbage out.

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#318

Post by int_19h » Wed, Oct 01, 2025, 1:03 AM UTC

abraxas wrote: post #178

dns_snek wrote: post #25

Earlier quoted context omitted.

> As the economics change, how sustainable is the current cost/benefit equilibrium of high-quality code "High quality code"? The standard today is "barely functional", if we lower the standards any further we will find ourselves debating how many crashes a day we're willing to live with, and whether we really care about weekly data loss caused by race conditions.

And if that's what's economically beneficial then it shall be. Unfortunately.

https://www.scottsmitelli.com/articles/altoids-by-the-fistfu...

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#319

Post by int_19h » Wed, Oct 01, 2025, 1:12 AM UTC

torben-friis wrote: post #9

injidup wrote: post #3

Shouldn't you be getting the LLM to also generate test cases to drive the code and also enforce coding standards on the LLM to generate small easily comprehensible software modułes with high quality inline documentation. Is this something people are doing?

The problem is similar to that of journalism vs social media hoaxes. An llm-assisted engineer writes code faster than a careful person can review. Eventually the careful engineers get ran over by the sheer amount of work to check, and code starts passing reviews when it shouldn’t. It sounds obvious, that careless work is faster than careful one, but there are psychological issues in play - expectation by management o…

> expectation by management of ai as a speed multiplier

In many cases, it's more than expectation. For top management especially, these are the people who have signed off on massive AI spending on the basis that it will improve productivity. Any evidence to the contrary is not just counter to their expectations - it's a giant flashing neon sign screaming "YOU FUCKED UP". So of course organizations run by those people are going to pretend that everything is fine, for as long as anything works at all.

And then the other side of this is the users. Who have already been conditioned to shrug at crappy software because we made that the norm, and because the tech market has so many market-dominant players or even outright monopolies in various niches that users often don't have a meaningful choice. Which is a perfect setup for slowly boiling the frog - even if AI is used to produce sloppy code, the frog is already used to hot water, and already convinced that there's no way out of the pot in any case, so if it gets hotter still they just rant about it but keep buying the product.

Which is to say, it is a shitshow, but it's a shitshow that can continue for longer than most engineers have emotional capacity to sustain without breaking down. In the long term, I expect AI coding in this environment to act as a filter: it will push out all the people who care about quality and polish out of the industry, and reward those who treat clicking "approved" on AI slop as their real job description.

Re: Comprehension debt: A ticking time bomb of LLM-generated code

#320

Post by int_19h » Wed, Oct 01, 2025, 1:20 AM UTC

blindriver wrote: post #145

No, I 100% don't think it will happen. LLMs have made the value of content worth precisely zero. Any content can be duplicated with a prompt. That means code is also worth precisely zero. It doesn't matter if humans can understand the code, what matters is if the LLM can understand the code and make modifications. As long as the LLM can read the code and adjust it based on the prompt, what happens on the inside doesn…

But how do you know that it's "fixed", if you don't understand the code?

You can have functional tests, sure, but if there's one thing that LLMs (and AI in general) is good at, it's finding unconventional ways to game metrics.