Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Verification debt: the hidden cost of AI-generated code

fazy.medium.com

Post reply on HN
Print view View original

11–20 of 103 posts

Re: Verification debt: the hidden cost of AI-generated code

#11

Post by apical_dendrite »

johngossman wrote: post #5

This verification problem is general. As an experiment, I had Claude Cowork write a history book. I chose as subject a biography of Paolo Sarpi, a Venetian thinker most active in the early 17th century. I chose the subject because I know something about him, but am far from expert, because many of the sources in Italian, in which I am a beginner, and because many of the sources are behind paywalls, which does not mea…

I used gemini to look up a relative with a connection to a famous event. The relative himself is obscure, but I have some of his writings and I've heard his story from other relatives. Gemini fabricated a completely false narrative about my relative that was much more exciting than what actually happened. I spent a bunch of time looking at the sources that Gemini supplied trying to verify things and although the sources were real, the story Gemini came up with was completely made up.

Re: Verification debt: the hidden cost of AI-generated code

#12

Post by johngossman »

apical_dendrite wrote: post #11
johngossman wrote: post #5

This verification problem is general. As an experiment, I had Claude Cowork write a history book. I chose as subject a biography of Paolo Sarpi, a Venetian thinker most active in the early 17th century. I chose the subject because I know something about him, but am far from expert, because many of the sources in Italian, in which I am a beginner, and because many of the sources are behind paywalls, which does not mea…

I used gemini to look up a relative with a connection to a famous event. The relative himself is obscure, but I have some of his writings and I've heard his story from other relatives. Gemini fabricated a completely false narrative about my relative that was much more exciting than what actually happened. I spent a bunch of time looking at the sources that Gemini supplied trying to verify things and although the sour…

Yup. I've had Gemini create fake citations to papers. I've also had it hallucinate the contents of paywalled papers, so I know I can't trust anything it writes, though I am getting better at using it recursively to verify things.

Re: Verification debt: the hidden cost of AI-generated code

#13

Post by somewhereoutth »

lowsong wrote: post #8
hnthrow0287345 wrote: post #4

This still seems like technical debt to me. It's just debt with a much higher compounding interest rate and/or shorter due date. Credit cards vs. traditional loans or mortgages. >And six months later you discover you’ve built exactly what the spec said — and nothing the customer actually wanted. That's not a developer problem, it's a PM/business problem. Your PM or equivalent should be neck deep in finding out what t…

> AI is actually better getting those built as long as you clean it up afterwards I've never seen a quick PoC get cleaned up. Not once. I'm sure it happens sometimes, but it's very rare in the industry. The reality is that a PoC usually becomes "good enough" and gets moved into production with only the most perfunctory of cleanup.

There is nothing as permanent as a temporary solution!

Re: Verification debt: the hidden cost of AI-generated code

#14

Post by lpnam0201 »

apical_dendrite wrote: post #7

My company recently hired a contractor. He submits multi-thousand line PRs every day, far faster than I can review them. This would maybe be OK if I could trust his output, but I can't. When I ask him really basic questions about the system, he either doesn't know or he gets it wrong. This week, I asked for some simple scripts that would let someone load data in a a local or staging environment, so that the system co…

Do you think he used AI to generate that much code without ever understanding or having a look at the code ? Why was he hired ?

Re: Verification debt: the hidden cost of AI-generated code

#15

Post by somewhereoutth »

VanTodi wrote: post #6

I've come to the point where I think generated code is nothing better than a random package I install. Did I read it all and just accepted what was promised? Yes Can it bite me in the butt somewhere down the road? Probably, but I currently at least have more doubt about the generated code than a random package I picked up somewhere on git which readme I just partly skipped over.

However a random [but well established] package will have been used many many times, thus will have been verified in the wild, and likely will have a bug tracker, updates, and perhaps even a community of people who care about that particular code. No comparison really.

Re: Verification debt: the hidden cost of AI-generated code

#16

Post by apical_dendrite »

lpnam0201 wrote: post #14
apical_dendrite wrote: post #7

My company recently hired a contractor. He submits multi-thousand line PRs every day, far faster than I can review them. This would maybe be OK if I could trust his output, but I can't. When I ask him really basic questions about the system, he either doesn't know or he gets it wrong. This week, I asked for some simple scripts that would let someone load data in a a local or staging environment, so that the system co…

Do you think he used AI to generate that much code without ever understanding or having a look at the code ? Why was he hired ?

Yes, because he can't answer basic questions about the code.

He was hired because we needed a contractor quickly and he and his company represented to us that he was a lot more experienced than he actually is.

Re: Verification debt: the hidden cost of AI-generated code

#17

Post by fishtoaster »

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO.

We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around:

- Include the spec for the change in your PR and only bother reviewing that, on the assumption that the AI faithfully executed it

- Lean harder on your deterministic verification: unit tests, full stack tests, linters, formatters, static analysis

- Get better ai-based review: greptile and bugbot and half a dozen others

- Lean into your observability tooling so that AIs can fix your production bugs so fast they don't even matter.

None of these seem fully sufficient right now, but it's such a new problem that I suspect we'll be figuring this out for the next few years at least. Maybe one of these becomes the silver bullet or maybe it's just a bunch of lead bullets.

But anyone who's able to ship AI code without human review (and without their codebase collapsing) will run circles around the rest.

Re: Verification debt: the hidden cost of AI-generated code

#19

Post by chromaton »

Historically, the cycle has been requirements -> code -> test, but with coding becoming much faster, the bottlenecks have changed. That's one of the reasons I've been working on Spark Runner to help automate testing for web apps: https://https://github.com/simonarthur/spark-runner

Re: Verification debt: the hidden cost of AI-generated code

#20

Post by orsorna »

fishtoaster wrote: post #17

Figuring out how to trust AI-written code faster is the project of software engineering for the next few years, IMO. We'll need to figure out the techniques and strategies that let us merge AI code sight unseen. Some ideas that have already started floating around: - Include the spec for the change in your PR and only bother reviewing that , on the assumption that the AI faithfully executed it - Lean harder on your d…

>Lean harder on your deterministic verification: unit tests, full stack tests, linters, formatters, static analysis

It's wild that the gamut of PRs being zipped around don't even do these. You would run such validations as a human...

Post reply on HN