Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verification debt: the hidden cost of AI-generated code

fazy.medium.com

Post reply on HN
Print view View original

81–90 of 103 posts

Re: Verification debt: the hidden cost of AI-generated code

#81

Post by threatofrain »

gjsman-1000 wrote: post #21
fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

Do you know what happens to every industry when they get too fast and slapdash? Regulation. It happened with plumbing. Electricians. Civil engineers. Bridge construction. Haircutting. Emergency response. Legal work. Tech is perhaps the least regulated industry in the world. Cutting someone’s hair requires a license, operating a commercial kitchen requires a license, holding the SSN of 100K people does not yet . If AI…

It's not just going to be software. We will absolutely be experiencing vibe law, vibe medicine, vibe legislation even. It'll be so much vibing that it's not worth saying the word anymore.

Re: Verification debt: the hidden cost of AI-generated code

#82

Post by whattheheckheck »

dwb wrote: post #60
xantronix wrote: post #59

Earlier quoted context omitted.

I can't help but think that the logical conclusion of spec-first development is a return to Waterfall methodology. The amount of rigour required almost entirely negates the speed advantages of LLMs, even in the hands of seasoned developers. Unless the stakeholders are external, there will always be that necessary organisational bottleneck; of course, the C-suite could always decide to foist project management entirel…

Yeah, agree. Either that or this idea of not reviewing the code at all takes hold, abdicating human engineering responsibility to the machines, until some big stupid disaster or when it’s Too Late.

Is the teleological fight. Do swe decide what the purpose of the system is or do non technical people?

Intention flows are important

Re: Verification debt: the hidden cost of AI-generated code

#83

Post by whattheheckheck »

bigstrat2003 wrote: post #58
fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

> Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. Or we could actually, you know, stop using a tool that doesn't work . People are so desperate to believe in the productivity boosts of AI that they are trying to contort the whole industry around a tool that is bad at its job, rather than going "yeah that tool sucks" and moving on like a sane person…

Neural nets sucked in 1960s and if they gave up then we wouldn't be here

Re: Verification debt: the hidden cost of AI-generated code

#84

Post by sarchertech »

wtallis wrote: post #71
sarchertech wrote: post #34

Earlier quoted context omitted.

Translating from a natural language spec to code involves a truly massive amount of decision making. For a non trivial program, 2 implementations of the same natural language spec will have thousands of observable differences. Where we are today, that is agents require guardrails to keep from spinning out, there is no way to let agents work on code autonomously that won’t end up with all of those observable differenc…

> Tests can’t prevent this because for a test suite to cover all observable behavior, it would need to be more complex than the code. In which case, it wouldn’t be any easier for machine or human to understand. I don't think "complex" is the right word here. A test suite would generally be more verbose than the implementation, but a lot of the time it can simply be a long list of input->output pairs that are individu…

At some point verbosity becomes complexity. If you’re talking all observable behavior the input and output pairs are likely to be quite verbose/complex.

Imagine testing a game where the inputs are the possible states of game, and the possible control inputs, and the outputs are the states that could result.

Of course very few human written programs require this level of testing, but if you are trying to prevent an a swarm of agents from changing observable behavior without human review, that’s what you’d need.

Even with simpler input output pairs, an AI tells you it added a feature and had to change 2,000 input/output pairs to do so. How do you verify that those were necessary to change, and how do you verify that you actually have enough cases to prevent the AI from doing something dumb?

Oops you didn’t have a test that said that items shouldn’t turn completely transparent when you drag them.

Re: Verification debt: the hidden cost of AI-generated code

#85

Post by skydhash »

wtallis wrote: post #71
sarchertech wrote: post #34

Earlier quoted context omitted.

Translating from a natural language spec to code involves a truly massive amount of decision making. For a non trivial program, 2 implementations of the same natural language spec will have thousands of observable differences. Where we are today, that is agents require guardrails to keep from spinning out, there is no way to let agents work on code autonomously that won’t end up with all of those observable differenc…

> Tests can’t prevent this because for a test suite to cover all observable behavior, it would need to be more complex than the code. In which case, it wouldn’t be any easier for machine or human to understand. I don't think "complex" is the right word here. A test suite would generally be more verbose than the implementation, but a lot of the time it can simply be a long list of input->output pairs that are individu…

Code is like f(x)=ax+b. You test would be a list of (x,y) tuple. You don’t verify the correctness of your points because they come from some source that you hold as true. What you want is the generic solution (the theory) proposed by the formula. And your test would be just a small set of points, mostly to ensure that no one has changed the a and b parameters. But if you have a finite number of points, The AI is more likely to give you a complicated spline formula than the simple formula above. Unless the tokens in the prompts push it to the right domain space. (Usually meaning that the problem is solved already)

Real code has more dimensionality than the above example. Experts have the right keywords, but even then that’s a whole of dice. And coming up with enough sample test cases is more arduous than writing the implementation.

Unless there’s no real solution (dimensionality is high), but we have a lot of tests data with a lower dimensionality than the problem. This used to be called machine learning and we have metrics like accuracy for it.

Re: Verification debt: the hidden cost of AI-generated code

#86

Post by habinero »

whattheheckheck wrote: post #83
bigstrat2003 wrote: post #58

Earlier quoted context omitted.

> Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. Or we could actually, you know, stop using a tool that doesn't work . People are so desperate to believe in the productivity boosts of AI that they are trying to contort the whole industry around a tool that is bad at its job, rather than going "yeah that tool sucks" and moving on like a sane person…

Neural nets sucked in 1960s and if they gave up then we wouldn't be here

And the Concorde did not replace normal jet travel.

Re: Verification debt: the hidden cost of AI-generated code

#87

Post by habinero »

hirvi74 wrote: post #36
johngossman wrote: post #5

This verification problem is general. As an experiment, I had Claude Cowork write a history book. I chose as subject a biography of Paolo Sarpi, a Venetian thinker most active in the early 17th century. I chose the subject because I know something about him, but am far from expert, because many of the sources in Italian, in which I am a beginner, and because many of the sources are behind paywalls, which does not mea…

I believe that, on a fundamental level, the principle of 'trust, but verify' can be followed to its logical endpoint, as covered in Ken Thompson's lecture, 'Reflections on Trusting Trust' [1]. At some point, one simply has to trust that something is correct, unless they have the capability to verify every step of a long chain of indirection. So, in regard to your book: Claude may or may not have hallucinated the info…

> But if the knowledge is in a book or comes from another other man-made source, it's some how infallible

Nobody who's ever done research believes that. Everything gets put along a spectrum of trust/accuracy.

You should be able to say what you believe, what you base that belief on, what it would take to disprove that belief, and how likely you think it is to be disproven.

That's why you do research from as many primary sources as you can, because yeah, otherwise you're reading someone else's interpretation. Sometimes you can't do that (you don't read the language, etc) and then you have to judge the quality of the interpretation.

It's an enormous amount of work to write a book, and making things up doesn't make that process a whole lot easier. So most people try to be accurate. Especially with editors and such doublechecking work. I still always judge the quality of the work as I'm reading it.

LLMs just flat out can't be trusted. They're endless fountains of words and aren't accurate by nature. They're fine if you already know the answer, and not fine if you don't.

Re: Verification debt: the hidden cost of AI-generated code

#88

Post by pcloadlett3r »

jldugger wrote: post #43

Verification debt has always been present, we just now feel an acute need for it, because we do it wrong. Clause and friends represent an increase in coders, without any corresponding increase in code reviewers. It's a break in the traditional model of reviewing as much code as you submit, and it all falls on human engineers, typically the most senior. Well, that model kinda sucked anyways. Humans are falliable and I…

Will the TLA+ spec Claude spits out do what the users actually desire? Will there be human oversight of the spec? If not, I don't see how it really helps if the future human machine interface is supposed to be loosey goosey natural language. The best thing I can conceive is some human observers of the system saying "Claude, the behavior as it stands now is perfect! Set it in stone with TLA+." But this whimsical idea has many problems.

Re: Verification debt: the hidden cost of AI-generated code

#89

Post by dwb »

whattheheckheck wrote: post #82
dwb wrote: post #60

Earlier quoted context omitted.

Yeah, agree. Either that or this idea of not reviewing the code at all takes hold, abdicating human engineering responsibility to the machines, until some big stupid disaster or when it’s Too Late.

Is the teleological fight. Do swe decide what the purpose of the system is or do non technical people? Intention flows are important

I’m not talking about which humans decide the purpose of the system, or even which humans engineer the system once designed at a higher level. I’m worried about leaving crucial decisions and understanding to LLMs, with humans just stepping back.

Re: Verification debt: the hidden cost of AI-generated code

#90

Post by tpoacher »

somewhereoutth wrote: post #13
lowsong wrote: post #8

Earlier quoted context omitted.

> AI is actually better getting those built as long as you clean it up afterwards I've never seen a quick PoC get cleaned up. Not once. I'm sure it happens sometimes, but it's very rare in the industry. The reality is that a PoC usually becomes "good enough" and gets moved into production with only the most perfunctory of cleanup.

There is nothing as permanent as a temporary solution!

Yes. Or, "Prototypes have a bad habit of becoming the product".
Post reply on HN