Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When AI writes the software, who verifies it?

leodemoura.github.io

Post reply on HN
Print view View original

51–60 of 309 posts

Re: When AI writes the software, who verifies it?

#51

Post by radlad »

throwaw12 wrote: post #50
oakpond wrote: post #6

You do. Even the latest models still frequently write really weird code. The problem is some developers now just submit code for review that they didn't bother to read. You can tell. Code review is more important than ever imho.

> You do I really want to say: "You are absolutely right" But here is a problem I am facing personally (numbers are hypothetical). I get a review request 10-15/day by 4 teammates, who are generating code by prompting, and I am doing same, so you can guess we might have ~20 PRs/day to review. now each PR is roughly updating 5-6 files and 10-15 lines in each. So you can estimate that, I am looking at around 50-60 files…

I don't quite follow - are you describing an issue with the way your team has structured PRs? IMO, a PR should contain just enough code to clearly and completely solve "a thing" without solving too much at once. But what this means in practice depends on the team, product, velocity, etc. It sounds like your PRs might be broken up into too small of chunks if you can't understand why the code is being added.

Re: When AI writes the software, who verifies it?

#52

Post by radlad »

redhed wrote: post #18
sausagefeet wrote: post #13

Earlier quoted context omitted.

I agree with you. But I have to say, it is an uphill battle and all the incentives are against you. 1. AI is meant to make us go faster, reviews are slow, the AI is smart, let it go. 2. There are plenty of AI maximizers who only think we should be writing design docs and letting the AI go to town on it. Maybe, this might be a great time to start a company. Maximize the benefits of AI while you can without someone who…

I actually agree with good time to start a company. Lot of available software engineers that can actually understand code, AI at a level that can actually speed up development, and so many startups focusing on AI wrapper slop that you can actually make a useful product and separate yourself from the herd. Or you can be a grifter and make some AI wrapper yourself and cash out with some VC investment. So good time for…

It's gonna be like that HBO Silicon Valley bit again, where everyone and their doctor is telling you about their app.

Re: When AI writes the software, who verifies it?

#53

Post by charlieflowers »

madrox wrote: post #42

I encourage everyone to RTFA and not just respond to the headline. This really is a glimpse into where the future is going. I've been saying "the last job to be automated will be QA" and it feels more true every day. It's one thing to be a product engineer in this era. It's another to be working at the level the author is, where code needs to be verifiable. However, once people stop vibing apps and start vibing kerne…

> "any sufficiently advanced agent is indistinguishable from a DSL."

I don't quite follow but I'd love to hear more about that.

Re: When AI writes the software, who verifies it?

#54

Post by throwaw12 »

radlad wrote: post #51
throwaw12 wrote: post #50

Earlier quoted context omitted.

> You do I really want to say: "You are absolutely right" But here is a problem I am facing personally (numbers are hypothetical). I get a review request 10-15/day by 4 teammates, who are generating code by prompting, and I am doing same, so you can guess we might have ~20 PRs/day to review. now each PR is roughly updating 5-6 files and 10-15 lines in each. So you can estimate that, I am looking at around 50-60 files…

I don't quite follow - are you describing an issue with the way your team has structured PRs? IMO, a PR should contain just enough code to clearly and completely solve "a thing" without solving too much at once. But what this means in practice depends on the team, product, velocity, etc. It sounds like your PRs might be broken up into too small of chunks if you can't understand why the code is being added.

I am saying PRs I get are around 60-70 lines of change, which is small enough to be considered as single unit (add to this unit tests which must pass with new change, so we are talking about 30 line change + 30 line unit test)

But when looking at the PR changes, you don't always see whole picture because review subjects (code lines) are scattered across files and methods, and GitHub also shows methods and files partially making it even more difficult to quickly spot the context around those updated lines.

Its difficult problem, because even if GitHub shows whole body of the updated method or a file, you still don't see grand picture.

For example: A (calls) -> B -> C -> D

And you made changes in D, how do you know the side effect on B, what if it broke A?

Re: When AI writes the software, who verifies it?

#55

Post by righthand »

galbar wrote: post #10
righthand wrote: post #3

No one really. Code is for humans to read and for machines to compile and execute. Llms are enabling people to just write the code and not have anyone read it. It’s solving a problem that didn’t really exist (we already had code generators before llms). It’s such an intoxicating copyright-abuse slot machine that a buddy who is building an ocaml+htmx tree editor told me “I always get stuck and end up going to the llm…

This is something I've been wondering about... If boilerplate was such a big issue, we should have worked on improving code generation. In fact, many tools and frameworks exist that did this already: - rails has fantastic code generation for CRUD use cases - intelliJ IDEs have been able to do many types of refactors and class generation that included some of the boilerplate I haven't reached a conclusion on this trai…

Pre-llm corpos my thoughts were that we should be training juniors on code generators. Instead we’re somewhere between rtfm or dont.

Re: When AI writes the software, who verifies it?

#56

Post by jra_samba »

throwaw12 wrote: post #50
oakpond wrote: post #6

You do. Even the latest models still frequently write really weird code. The problem is some developers now just submit code for review that they didn't bother to read. You can tell. Code review is more important than ever imho.

> You do I really want to say: "You are absolutely right" But here is a problem I am facing personally (numbers are hypothetical). I get a review request 10-15/day by 4 teammates, who are generating code by prompting, and I am doing same, so you can guess we might have ~20 PRs/day to review. now each PR is roughly updating 5-6 files and 10-15 lines in each. So you can estimate that, I am looking at around 50-60 files…

Tests. All changes must have tests. If they're generating the code, they can generate the tests too.

Re: When AI writes the software, who verifies it?

#58

Post by ptnpzwqd »

throwaw12 wrote: post #50
oakpond wrote: post #6

You do. Even the latest models still frequently write really weird code. The problem is some developers now just submit code for review that they didn't bother to read. You can tell. Code review is more important than ever imho.

> You do I really want to say: "You are absolutely right" But here is a problem I am facing personally (numbers are hypothetical). I get a review request 10-15/day by 4 teammates, who are generating code by prompting, and I am doing same, so you can guess we might have ~20 PRs/day to review. now each PR is roughly updating 5-6 files and 10-15 lines in each. So you can estimate that, I am looking at around 50-60 files…

If reviewing has become the bottleneck, the obvious - albeit slightly boring - solution is to slow down spitting out new code, and spend relatively more time reviewing.

Just going ahead and piling up PRs or skipping the review process is of course not recommended.

Re: When AI writes the software, who verifies it?

#60

Post by esafak »

charlieflowers wrote: post #53
madrox wrote: post #42

I encourage everyone to RTFA and not just respond to the headline. This really is a glimpse into where the future is going. I've been saying "the last job to be automated will be QA" and it feels more true every day. It's one thing to be a product engineer in this era. It's another to be working at the level the author is, where code needs to be verifiable. However, once people stop vibing apps and start vibing kerne…

> "any sufficiently advanced agent is indistinguishable from a DSL." I don't quite follow but I'd love to hear more about that.

https://en.wikipedia.org/wiki/Clarke's_three_laws
Post reply on HN