The biggest issue right now is most AI tools aren't hooked up appropriately to an environment they can test in (Chrome typically). Replit works extremely well because it has an integrated browser and testing strategy. AI works very well when it has the ability to check its own work.
When AI writes the software, who verifies it?
61–70 of 309 posts
Re: When AI writes the software, who verifies it?
#62Earlier quoted context omitted.
> You do I really want to say: "You are absolutely right" But here is a problem I am facing personally (numbers are hypothetical). I get a review request 10-15/day by 4 teammates, who are generating code by prompting, and I am doing same, so you can guess we might have ~20 PRs/day to review. now each PR is roughly updating 5-6 files and 10-15 lines in each. So you can estimate that, I am looking at around 50-60 files…
If reviewing has become the bottleneck, the obvious - albeit slightly boring - solution is to slow down spitting out new code, and spend relatively more time reviewing. Just going ahead and piling up PRs or skipping the review process is of course not recommended.
you are not wrong, but solution you are proposing is just throttling the system because of the bottleneck, and it doesn't solve the bottleneck problem.
Re: When AI writes the software, who verifies it?
#63I think the issue goes even deeper than verification. Verification is technically possible. You could, in theory, build a C compiler or a browser and use existing tests to confirm it works. The harder problem is discovery: how do you build something entirely new, something that has no existing test suite to validate against? Verification works because someone has already defined what "correct" looks like. There is po…
Well that's a problem the software industry has been building for itself for decades. Software has, since at least the adoption of "agile" created an industry culture of not just refusing to build to specs but insisting that specs are impossible to get from a customer.
Agile is a pretty badly defined beast at the best of times but even the most twisted interpretation doesnt mean that. It's mainly just a rejection of BDUF.
Re: When AI writes the software, who verifies it?
#64Earlier quoted context omitted.
> You do I really want to say: "You are absolutely right" But here is a problem I am facing personally (numbers are hypothetical). I get a review request 10-15/day by 4 teammates, who are generating code by prompting, and I am doing same, so you can guess we might have ~20 PRs/day to review. now each PR is roughly updating 5-6 files and 10-15 lines in each. So you can estimate that, I am looking at around 50-60 files…
Tests. All changes must have tests. If they're generating the code, they can generate the tests too.
who reviews the tests? again me? -> that's exactly why I am saying review is a bottleneck, especially with current tooling, which doesn't show second order impacts of the changes and its not easy to reason about when method gets called by 10 other methods with 4 level of parent hierarchy
Re: When AI writes the software, who verifies it?
#65Because of the scale of generated code, often it is the AI verifying the AI's work.
I of course cannot say what the future holds, but current frontier models are - in my experience - nowhere near good enough for such autonomy.
Even with other agents reviewing the code, good test coverage, etc., both smaller - and every now and then larger - mistakes make their way through, and the existence of such mistakes in the codebase tend to accellerate even more of them.
It for sure depends on many factors, but I have seen enough to feel confident that we are not there yet.
Re: When AI writes the software, who verifies it?
#66Earlier quoted context omitted.
Well that's a problem the software industry has been building for itself for decades. Software has, since at least the adoption of "agile" created an industry culture of not just refusing to build to specs but insisting that specs are impossible to get from a customer.
Agile hasn't been insisting that specs are impossible to get from a customer. They have been insisting that getting specs from a customer is best performed as a dynamic process. In my opinion, that's one of agile's most significant contributions. It lines up with a learning process that doesn't assume the programmer or the customer knows the best course ahead of time.
And good luck when getting misaligned specs (communication issues customer side, docs that are not aligned with the product,...). Drafting specs and investigating failure will require both a diplomat hat and a detective hat. Maybe with the developer hat, we will get DDD being meaningful again.
Re: When AI writes the software, who verifies it?
#67This is the biggest problem going forward. I wrote about the problem many times on my blog, in talks, and as premises in my sci-fi novels Sitting in your cubical with your perfect set of test suites, code verification rules, SOP's and code reviews you wont want to hear this, but other companies will be gunning for your market; writing almost identical software to yours in the future from a series of prompts that gene…
How do you get an extensive test suite?
Re: When AI writes the software, who verifies it?
#68LLM generated code combined with formal verification just feels like we're entering the most ridiculous timeline. We know formal verification doesn't work at scale, hence we don't use it. We might get fully vibe coded systems but we sure as hell won't be able to verify them.
The collapse of civilisation is real.
Re: When AI writes the software, who verifies it?
#69I encourage everyone to RTFA and not just respond to the headline. This really is a glimpse into where the future is going. I've been saying "the last job to be automated will be QA" and it feels more true every day. It's one thing to be a product engineer in this era. It's another to be working at the level the author is, where code needs to be verifiable. However, once people stop vibing apps and start vibing kerne…
I am as enthusiastic about formal methods as the next guy, but I very much doubt any LLM-based technique will make it economical to write a substantial fraction of application software in Lean. The LLM can play a powerful heuristic role in searching for proof-bearing code in areas where there is good training data. Unfortunately those areas are few and far between.
Moreover, humans will still need to read even rigorously proved code if only to suss out performance issues. And training people to read Lean will continue to be costly.
Though, as the OP says, this is a very exciting time for developing provably correct systems programming.
Re: When AI writes the software, who verifies it?
#70This is the biggest problem going forward. I wrote about the problem many times on my blog, in talks, and as premises in my sci-fi novels Sitting in your cubical with your perfect set of test suites, code verification rules, SOP's and code reviews you wont want to hear this, but other companies will be gunning for your market; writing almost identical software to yours in the future from a series of prompts that gene…
If you need to interact with some things in platform.openai.com, you know it is not months away, it is there already now. I had to go through forms and flows there, so buggy and untested, simply broken. They really eat their own dog food. Interacting with the support, resulted in literally weeks of ping pong between me and AI smoothed replies via email to fix their bugs. Terrible.