Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verification debt: the hidden cost of AI-generated code

fazy.medium.com

Post reply on HN
Print view View original

41–50 of 103 posts

Re: Verification debt: the hidden cost of AI-generated code

#41

Post by ahsisjb »

fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

> Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO

Replace AI written with “cheap dev written” and think about why that isn’t already true.

The bottleneck is a competent dev understanding a project. Always has been.

Another fundamental flaw is you can’t trust LLMs. It’s fundamentally impossible compared to the way you trust a human. Humans make mistakes. LLMs do not. Anything “wrong” they do is them working exactly as designed.

Re: Verification debt: the hidden cost of AI-generated code

#42

Post by hirvi74 »

johngossman wrote: post #12
apical_dendrite wrote: post #11

Earlier quoted context omitted.

I used gemini to look up a relative with a connection to a famous event. The relative himself is obscure, but I have some of his writings and I've heard his story from other relatives. Gemini fabricated a completely false narrative about my relative that was much more exciting than what actually happened. I spent a bunch of time looking at the sources that Gemini supplied trying to verify things and although the sour…

Yup. I've had Gemini create fake citations to papers. I've also had it hallucinate the contents of paywalled papers, so I know I can't trust anything it writes, though I am getting better at using it recursively to verify things.

I am certain I read article that was posted on YN a month or so ago about some researchers that were caught using false citations in their research.

If I remember correctly, some group used an AI tool to sniff for AI citations in other's works. What I remember most was how abhorrent some of the sources that the AI sniffer caught. One of the citation's authors was literally cited as "FirstName LastName" -- didn't even sub in a fake name lol.

Edit: I found the OP:

https://news.ycombinator.com/item?id=46720395

Re: Verification debt: the hidden cost of AI-generated code

#43

Post by jldugger »

Verification debt has always been present, we just now feel an acute need for it, because we do it wrong.

Clause and friends represent an increase in coders, without any corresponding increase in code reviewers. It's a break in the traditional model of reviewing as much code as you submit, and it all falls on human engineers, typically the most senior.

Well, that model kinda sucked anyways. Humans are falliable and Ironies of Automation lays bare the failure modes. We all know the signs: 50 comments on a 5 line PR, a lonely "LGTM" on the 5000 line PR. This is not responsible software engineering or design; it is, as the author puts it, a big green "I'm accountable" button with no force behind it.

It's probably time for all of us on HN to pick up a book or course on TLA+ and elevate the state of software verification. Even if Claude ends up writing TLA+ specs too, at least that will be a smaller, simpler code base to review?

Re: Verification debt: the hidden cost of AI-generated code

#44

Post by dwaltrip »

gregoryl wrote: post #38
lowsong wrote: post #8

Earlier quoted context omitted.

> AI is actually better getting those built as long as you clean it up afterwards I've never seen a quick PoC get cleaned up. Not once. I'm sure it happens sometimes, but it's very rare in the industry. The reality is that a PoC usually becomes "good enough" and gets moved into production with only the most perfunctory of cleanup.

The key to every quick POC having a short life, is a reliance on manual work outside of the engineering team.

This is genius.

Re: Verification debt: the hidden cost of AI-generated code

#45

Post by logicchains »

sarchertech wrote: post #34
fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

Translating from a natural language spec to code involves a truly massive amount of decision making. For a non trivial program, 2 implementations of the same natural language spec will have thousands of observable differences. Where we are today, that is agents require guardrails to keep from spinning out, there is no way to let agents work on code autonomously that won’t end up with all of those observable differenc…

>For a non trivial program, 2 implementations of the same natural language spec will have thousands of observable differences.

If they're not defined in the spec then these differences shouldn't matter, they're just implementation details. And if they do matter, then they should be included in the spec; a natural language spec that doesn't specify some things that should be specified is not a good spec.

Re: Verification debt: the hidden cost of AI-generated code

#46

Post by suzzer99 »

afro88 wrote: post #24
apical_dendrite wrote: post #16

Earlier quoted context omitted.

Yes, because he can't answer basic questions about the code. He was hired because we needed a contractor quickly and he and his company represented to us that he was a lot more experienced than he actually is.

Will you get rid of him? It sounds like he's wasting a lot of your time

Or... is apical_dendrite just circling the wagons, scared of AI taking his job?

/management thoughts

Re: Verification debt: the hidden cost of AI-generated code

#47

Post by suzzer99 »

lowsong wrote: post #8
hnthrow0287345 wrote: post #4

This still seems like technical debt to me. It's just debt with a much higher compounding interest rate and/or shorter due date. Credit cards vs. traditional loans or mortgages. >And six months later you discover you’ve built exactly what the spec said — and nothing the customer actually wanted. That's not a developer problem, it's a PM/business problem. Your PM or equivalent should be neck deep in finding out what t…

> AI is actually better getting those built as long as you clean it up afterwards I've never seen a quick PoC get cleaned up. Not once. I'm sure it happens sometimes, but it's very rare in the industry. The reality is that a PoC usually becomes "good enough" and gets moved into production with only the most perfunctory of cleanup.

One trick for avoiding this is to use artifacts in the PoC that no self-respecting developer would ever allow in production. I use html tables in PoCs because front-end devs hate them - with old-school properties like cellpadding that I know will get replaced.

I also name everything DEMO__ so at least they'll have to go through the exercise or renaming it. Although I've had cases where they don't even do that lol. But at least then you know who's totally worthless.

Re: Verification debt: the hidden cost of AI-generated code

#48

Post by scuff3d »

gowld wrote: post #28
scuff3d wrote: post #22

Earlier quoted context omitted.

This sums up the inherent friction between hype and reality really well. CEOs and hype men want you to believe that LLMs can replace everyone. In 6 months you can give them the keys to the kingdom and they'll do a better job running your company then you did. No more devs. No more QA. No more pesky employees who needs crazy stuff like sleep, and food, and time off to be a human. Then of course we run face first into…

> CEOs and hype men want you to believe that LLMs can replace everyone. > they'll do a better job running your company SWEs are the ones running the company. CEOs are.

I was being hyperbolic to make a point. Not literal.

Re: Verification debt: the hidden cost of AI-generated code

#49

Post by sarchertech »

logicchains wrote: post #45
sarchertech wrote: post #34

Earlier quoted context omitted.

Translating from a natural language spec to code involves a truly massive amount of decision making. For a non trivial program, 2 implementations of the same natural language spec will have thousands of observable differences. Where we are today, that is agents require guardrails to keep from spinning out, there is no way to let agents work on code autonomously that won’t end up with all of those observable differenc…

>For a non trivial program, 2 implementations of the same natural language spec will have thousands of observable differences. If they're not defined in the spec then these differences shouldn't matter, they're just implementation details. And if they do matter, then they should be included in the spec; a natural language spec that doesn't specify some things that should be specified is not a good spec.

There are a limitless number of implementation details you don't think you care about until they are constantly changing.

I doubt there exists a single piece of nontrivial software today where you could randomly alter 5% of the implementation details while keeping to the spec, without resulting in a flood of support tickets.

Re: Verification debt: the hidden cost of AI-generated code

#50

Post by pjm331 »

fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

My bet is that the last item is what we’ll end up leaning heavily on - feels like the path of least resistance

Throw in some simulated user interactions in a staging environment with a bunch of agents acting like customers a la StrongDM so you can catch the bugs earlier

Post reply on HN