Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verification debt: the hidden cost of AI-generated code

fazy.medium.com

Post reply on HN
Print view View original

31–40 of 103 posts

Re: Verification debt: the hidden cost of AI-generated code

#31

Post by hackyhacky »

gjsman-1000 wrote: post #21
fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

Do you know what happens to every industry when they get too fast and slapdash? Regulation. It happened with plumbing. Electricians. Civil engineers. Bridge construction. Haircutting. Emergency response. Legal work. Tech is perhaps the least regulated industry in the world. Cutting someone’s hair requires a license, operating a commercial kitchen requires a license, holding the SSN of 100K people does not yet . If AI…

> Regulation will follow.

I would hope so, but it won't happen as long as the billionaire AI bros keep on paying politicians for favorable treatment.

Re: Verification debt: the hidden cost of AI-generated code

#32

Post by gowld »

hnthrow0287345 wrote: post #4

This still seems like technical debt to me. It's just debt with a much higher compounding interest rate and/or shorter due date. Credit cards vs. traditional loans or mortgages. >And six months later you discover you’ve built exactly what the spec said — and nothing the customer actually wanted. That's not a developer problem, it's a PM/business problem. Your PM or equivalent should be neck deep in finding out what t…

Bad code isn't Technical Debt, it's an unhedged Call Option

If you search for that quote, you can find #1 result is an AI slop paraphrase published last week, but the original article was 11 years ago, and republished 3 years ago.

https://higherorderlogic.com/programming/2023/10/06/bad-code...

Re: Verification debt: the hidden cost of AI-generated code

#33

Post by apical_dendrite »

gowld wrote: post #27
apical_dendrite wrote: post #7

My company recently hired a contractor. He submits multi-thousand line PRs every day, far faster than I can review them. This would maybe be OK if I could trust his output, but I can't. When I ask him really basic questions about the system, he either doesn't know or he gets it wrong. This week, I asked for some simple scripts that would let someone load data in a a local or staging environment, so that the system co…

Why are you paying someone who isn't doing the job you hired someone to do? Why are you acting like you work for the contractor, instead of the contractor workign for you? Why are you teaching a contractor anything? That's a violation of labor law. You are treating a contractor like an employee.

Excellent questions.

Re: Verification debt: the hidden cost of AI-generated code

#34

Post by sarchertech »

fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

Translating from a natural language spec to code involves a truly massive amount of decision making.

For a non trivial program, 2 implementations of the same natural language spec will have thousands of observable differences.

Where we are today, that is agents require guardrails to keep from spinning out, there is no way to let agents work on code autonomously that won’t end up with all of those observable differences constantly shifting, resulting in unusable software.

Tests can’t prevent this because for a test suite to cover all observable behavior, it would need to be more complex than the code. In which case, it wouldn’t be any easier for machine or human to understand.

The only solution to this problem is that LLMs get better. Personally I think at the point they can pull this off, they can do any white collar job, and there’s not point in planning for that future because it results in either Mad Mad or Star Trek.

Re: Verification debt: the hidden cost of AI-generated code

#35

Post by coffeefirst »

gjsman-1000 wrote: post #21
fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

Do you know what happens to every industry when they get too fast and slapdash? Regulation. It happened with plumbing. Electricians. Civil engineers. Bridge construction. Haircutting. Emergency response. Legal work. Tech is perhaps the least regulated industry in the world. Cutting someone’s hair requires a license, operating a commercial kitchen requires a license, holding the SSN of 100K people does not yet . If AI…

He’s right. Exhibit A is age-gating social media. If the industry keeps being this careless that’s going to be the tip of the iceberg.

Re: Verification debt: the hidden cost of AI-generated code

#36

Post by hirvi74 »

johngossman wrote: post #5

This verification problem is general. As an experiment, I had Claude Cowork write a history book. I chose as subject a biography of Paolo Sarpi, a Venetian thinker most active in the early 17th century. I chose the subject because I know something about him, but am far from expert, because many of the sources in Italian, in which I am a beginner, and because many of the sources are behind paywalls, which does not mea…

I believe that, on a fundamental level, the principle of 'trust, but verify' can be followed to its logical endpoint, as covered in Ken Thompson's lecture, 'Reflections on Trusting Trust' [1]. At some point, one simply has to trust that something is correct, unless they have the capability to verify every step of a long chain of indirection.

So, in regard to your book: Claude may or may not have hallucinated the information from its cited sources. Gemini, as well. However, say you had access to the cited information behind a paywall. How would you go about verifying the information cited in those sources was correct?

Since the release of LLMs over the past four years or so, I have noticed a trend where people are (rightfully) hesitant to trust the output of LLMs. But if the knowledge is in a book or comes from another other man-made source, it's some how infallible? Such thinking reminds me of my primary schooling days. Teachers would not let us use Wikipedia as a source because, "Anyone can edit anything." Though, it's as one cannot write anything they want in a book -- be it true or false?

How many scientific researchers have p-hacked their research, falsified data, or used other methods of deceit? I do not believe it's a truly an issue on a grand scale nor does it make vast amounts of science illegitimate. When caught, the punishments are usually handled in a serious manner, but no telling how much falsified research was never caught.

I do believe any and all information provided by LLMs should be verified and not blindly trusted, however, I extend that same policy to works from my fellow humans. Of course, no one has the time to verify every single detail of every bit of information one comes across. Hence, at some point, we all must settle on trusting in trust. Knowledge that we cannot verify is not knowledge. It is faith.

[1] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

Re: Verification debt: the hidden cost of AI-generated code

#37

Post by leptons »

hackyhacky wrote: post #31
gjsman-1000 wrote: post #21

Earlier quoted context omitted.

Do you know what happens to every industry when they get too fast and slapdash? Regulation. It happened with plumbing. Electricians. Civil engineers. Bridge construction. Haircutting. Emergency response. Legal work. Tech is perhaps the least regulated industry in the world. Cutting someone’s hair requires a license, operating a commercial kitchen requires a license, holding the SSN of 100K people does not yet . If AI…

> Regulation will follow. I would hope so, but it won't happen as long as the billionaire AI bros keep on paying politicians for favorable treatment.

The word is "bribing", and the current (bribable) administration won't be around forever (hopefully).

Re: Verification debt: the hidden cost of AI-generated code

#38

Post by gregoryl »

lowsong wrote: post #8
hnthrow0287345 wrote: post #4

This still seems like technical debt to me. It's just debt with a much higher compounding interest rate and/or shorter due date. Credit cards vs. traditional loans or mortgages. >And six months later you discover you’ve built exactly what the spec said — and nothing the customer actually wanted. That's not a developer problem, it's a PM/business problem. Your PM or equivalent should be neck deep in finding out what t…

> AI is actually better getting those built as long as you clean it up afterwards I've never seen a quick PoC get cleaned up. Not once. I'm sure it happens sometimes, but it's very rare in the industry. The reality is that a PoC usually becomes "good enough" and gets moved into production with only the most perfunctory of cleanup.

The key to every quick POC having a short life, is a reliance on manual work outside of the engineering team.

Re: Verification debt: the hidden cost of AI-generated code

#39

Post by user3939382 »

fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

I made a distributed operating system that manages all of this. Not just for agents per se but in general allows many devs to work simultaneously without tons of central review and allows them to keep standards high while working independently.

Re: Verification debt: the hidden cost of AI-generated code

#40

Post by Copyrightest »

fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

[dead]
Post reply on HN