Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verification debt: the hidden cost of AI-generated code

fazy.medium.com

Post reply on HN
Print view View original

51–60 of 103 posts

Re: Verification debt: the hidden cost of AI-generated code

#51

Post by regus »

hirvi74 wrote: post #36
johngossman wrote: post #5

This verification problem is general. As an experiment, I had Claude Cowork write a history book. I chose as subject a biography of Paolo Sarpi, a Venetian thinker most active in the early 17th century. I chose the subject because I know something about him, but am far from expert, because many of the sources in Italian, in which I am a beginner, and because many of the sources are behind paywalls, which does not mea…

I believe that, on a fundamental level, the principle of 'trust, but verify' can be followed to its logical endpoint, as covered in Ken Thompson's lecture, 'Reflections on Trusting Trust' [1]. At some point, one simply has to trust that something is correct, unless they have the capability to verify every step of a long chain of indirection. So, in regard to your book: Claude may or may not have hallucinated the info…

This is great, your final line summarizes my thoughts as well. When it comes to matters of faith your average Redditor and Hacker News commenter will heap scorn and derision on religious people for accepting things blindly without any proof, yet they will blindly accept what other people tell them is true, or now what an LLM says is true.

Re: Verification debt: the hidden cost of AI-generated code

#52

Post by Herring »

sarchertech wrote: post #34
fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

Translating from a natural language spec to code involves a truly massive amount of decision making. For a non trivial program, 2 implementations of the same natural language spec will have thousands of observable differences. Where we are today, that is agents require guardrails to keep from spinning out, there is no way to let agents work on code autonomously that won’t end up with all of those observable differenc…

Agreed, but with one exception: are tests supposed to cover all observable behavior? Usually people are happy with just eliminating large/easy classes of bad (unintended) behavior, otherwise they go for formal verification which is an entirely different ballgame.

Re: Verification debt: the hidden cost of AI-generated code

#53

Post by abetusk »

Both empirically and theoretically, verification is often much more tractable than discovery.

Software development is a highly complex task and verification becomes not just validation of the output but also verification that the work is solving the problem desired, not just the problem specified.

I'm empathetic to that scenario, but this was a problem with software development to begin with. I would much rather be in a situation of reducing friction to verification than reducing friction to discovery.

Cognitive load might be the same but now we get a potential boost in productivity for the same cost.

Re: Verification debt: the hidden cost of AI-generated code

#54

Post by dwb »

fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

What is this obsession with specifications? For a start it’s certainly not fair to assume an LLM has translated it into correct code, even if there is one reasonable way to do so, and there probably isn’t. I like a good, well-targeted spec as much as anyone, but come on. A spec detailed enough to describe a program is more-or-less the program but written in a non-executable language. I want to review the code, not a spec.

Re: Verification debt: the hidden cost of AI-generated code

#56

Post by sarchertech »

Herring wrote: post #52
sarchertech wrote: post #34

Earlier quoted context omitted.

Translating from a natural language spec to code involves a truly massive amount of decision making. For a non trivial program, 2 implementations of the same natural language spec will have thousands of observable differences. Where we are today, that is agents require guardrails to keep from spinning out, there is no way to let agents work on code autonomously that won’t end up with all of those observable differenc…

Agreed, but with one exception: are tests supposed to cover all observable behavior? Usually people are happy with just eliminating large/easy classes of bad (unintended) behavior, otherwise they go for formal verification which is an entirely different ballgame.

No they aren’t because they can’t (at least not without becoming so complicated that there’s no longer a point).

But humans are much better at reasoning about whether a change is going to impact observable behavior than current LLMs are as evidenced by the fact that LLMs require a test suite or something similar to build a working app longer than a few thousand lines.

Re: Verification debt: the hidden cost of AI-generated code

#57

Post by gopalv »

fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

> We'll need to figure out the techniques and strategies that let us merge AI code sight unseen

Every strategy which worked with an off-shore team in India works well for AI.

Sometime in mid 2017, I found myself running out of hours in the day stopping code from being merged.

On one hand, I needed to stamp the PRs because I was an ASF PMC member and not a lot of the folks who were opening JIRAs were & this wasn't a tech debt friendly culture, because someone from LinkedIn or Netflix or EMR could say "Your PR is shit, why did you merge it?" and "Well, we had a release due in 6 days" is not an answer.

Claude has been a drop-in replacement for the same problem, where I have to exercise the exact same muscles, though a lot easier because I can tell the AI that "This is completely wrong, throw it away and start over" without involving Claude's manager in the conversation.

The manager conversations were warranted and I learned to be nicer two years into that experience [1], but it's a soft skill which I no longer use with AI.

Every single method which worked with a remote team in a different timezone works with AI for me & perhaps better, because they're all clones of the best available - specs, pre-commit verifiers, mandatory reviews by someone uncommitted on the deadline, ease of reproducing bugs outside production and less clever code over all.

[1] - https://notmysock.org/blog/2018/Nov/17/

Re: Verification debt: the hidden cost of AI-generated code

#58

Post by bigstrat2003 »

fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

> Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO.

Or we could actually, you know, stop using a tool that doesn't work. People are so desperate to believe in the productivity boosts of AI that they are trying to contort the whole industry around a tool that is bad at its job, rather than going "yeah that tool sucks" and moving on like a sane person would.

Re: Verification debt: the hidden cost of AI-generated code

#59

Post by xantronix »

dwb wrote: post #54
fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

What is this obsession with specifications? For a start it’s certainly not fair to assume an LLM has translated it into correct code, even if there is one reasonable way to do so, and there probably isn’t. I like a good, well-targeted spec as much as anyone, but come on. A spec detailed enough to describe a program is more-or-less the program but written in a non-executable language. I want to review the code, not a…

I can't help but think that the logical conclusion of spec-first development is a return to Waterfall methodology. The amount of rigour required almost entirely negates the speed advantages of LLMs, even in the hands of seasoned developers. Unless the stakeholders are external, there will always be that necessary organisational bottleneck; of course, the C-suite could always decide to foist project management entirely on individual contributors, or take it on themselves, but I see that ending either in burnout or eventual neglect. All in the service of being on the forefront of adoption, and for what end?

Re: Verification debt: the hidden cost of AI-generated code

#60

Post by dwb »

xantronix wrote: post #59
dwb wrote: post #54

Earlier quoted context omitted.

What is this obsession with specifications? For a start it’s certainly not fair to assume an LLM has translated it into correct code, even if there is one reasonable way to do so, and there probably isn’t. I like a good, well-targeted spec as much as anyone, but come on. A spec detailed enough to describe a program is more-or-less the program but written in a non-executable language. I want to review the code, not a…

I can't help but think that the logical conclusion of spec-first development is a return to Waterfall methodology. The amount of rigour required almost entirely negates the speed advantages of LLMs, even in the hands of seasoned developers. Unless the stakeholders are external, there will always be that necessary organisational bottleneck; of course, the C-suite could always decide to foist project management entirel…

Yeah, agree. Either that or this idea of not reviewing the code at all takes hold, abdicating human engineering responsibility to the machines, until some big stupid disaster or when it’s Too Late.
Post reply on HN