Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your job is to deliver code you have proven to work

simonwillison.net

Post reply on HN
Print view View original

341–350 of 699 posts

Re: Your job is to deliver code you have proven to work

#341

Post by newsoftheday »

jennyholzer2 wrote: post #124
troyvit wrote: post #78

Earlier quoted context omitted.

I used to do that in simpler days. I'd add a link to where I copied it from so we could reference it if there were problems. This was for relatively small projects with just a few people.

> I'd add a link to where I copied it from LLMs can't do this. Your code is unambiguously better than any LLM code if you can comment a link to the stackoverflow post you copied it from.

Agreed on the first part for sure since an LLM is the computer/software version of a blender.

So, I'm agreed on the second part too then.

Re: Your job is to deliver code you have proven to work

#342

Post by WhyOhWhyQ »

ncruces wrote: post #340
WhyOhWhyQ wrote: post #284

Earlier quoted context omitted.

I think it would be interesting if professional engineering becomes more like producing formally correct documents for the AI to implement.

We have these tools that we use to write formally correct documents. They're called programing languages, and a deterministic algorithm translates them to machine code. Are we sure English and a probabilistic algorithm is any better at this?

I actually hate AI in my core, to the point that if it gets too much more advanced I'll likely be in existential crisis, so don't attack me on those grounds. Given it exists, I'm going to find what's good about it though. I do think the problem of AI existing has to be confronted. Maybe one solution is what the human does is produce specs like the HTML 5 one, and what the AI does is implement it in software.

Re: Your job is to deliver code you have proven to work

#343

Post by newsoftheday »

dekhn wrote: post #308

Prove is a strong word. There are few cases in real-world programming where you can prove anything. I prefer to make this probabilistic: use testing to reduce the probability that your code isn't correct, for the situations in which it is expected to be deployed. In this sense, coding and testing is much like doing experimental physics: we never really prove a theory or disprove it, we just invalidate clearly wrong o…

Testing must cover all cases else a 10 LOC LLM created PR is inherently more dangerous than a human 100 LOC PR because the LLM will likely also have written the test cases and it will try to make it all balance out with all passing; instead of making sure the test cases actually cover everything with the type of logic a human would apply.

Re: Your job is to deliver code you have proven to work

#344

Post by colinb »

jopsen wrote: post #274
tyrust wrote: post #243

Earlier quoted context omitted.

Code review is rarely done live. It's usually asynchronous, giving the reviewer plenty of time to read, digest, and give considered feedback on the changes. Perhaps a spicy patch would involve some kind of meeting. Or maybe in a mentor/mentee situation where you'd want high-bandwidth communication.

Doing only IRL code reviews would certainly improve quality in some projects :) It's probably also fairly expensive to do.

Fagan inspection has entered the room

Re: Your job is to deliver code you have proven to work

#345

Post by sowbug »

Since they’re robots, automated tests and manual tests are effectively the same thing.

I'd buttress this statement with a nuance. Automated tests typically run in their entirety, usually by a well-known command like cargo test or at least by the CI tools. Manual tests are often skipped because the test seems to be far away from the code being changed.

My all-time favorite team had a rule that your code didn't exist if it didn't have automated tests to "defend" it. If it didn't, it was OK, or at least not surprising, for someone else to break or refactor it out of existence (not maliciously, of course).

Re: Your job is to deliver code you have proven to work

#346

Post by zhyder »

"Almost anyone can prompt an LLM to generate a thousand-line patch and submit it for code review. That’s no longer valuable. What’s valuable is contributing code that is proven to work."

I'd go further: what's valuable is code review. So review the AI agent's code yourself first, ensuring not only that it's proven to work, but also that it's good quality (across various dimensions but most importantly in maintainability in future). If you're already overwhelmed by that thousand-line patch, try to create a hundred-line patch that accomplishes the same task.

I expect code review tools to also rapidly change, as lines of code written per person dramatically increase. Any good new tools already?

Re: Your job is to deliver code you have proven to work

#347

Post by jghn »

jopsen wrote: post #274
tyrust wrote: post #243

Earlier quoted context omitted.

Code review is rarely done live. It's usually asynchronous, giving the reviewer plenty of time to read, digest, and give considered feedback on the changes. Perhaps a spicy patch would involve some kind of meeting. Or maybe in a mentor/mentee situation where you'd want high-bandwidth communication.

Doing only IRL code reviews would certainly improve quality in some projects :) It's probably also fairly expensive to do.

Am old enough that this was status quo for part of my career, and have also been in some groups that did this as a rejection of modern code review techniques.

There are pros & cons to both sides. As you point out it's quite expensive in terms of time to do the in person style. Getting several people together is a big hassle. I've found that the code reviews themselves, and what people get out of them, are wildly different though. In person code reviews have been much more holistic in my experience, sometimes bordering on bigger picture planning. And much better as a learning tool for other people involved. Whereas the diff style online code review tends to be more focused on the immediate concerns.

There's not a right or wrong answer between those tradeoffs, but people need to realize they're not the same thing.

Re: Your job is to deliver code you have proven to work

#348

Post by nunez »

hnthrow0287345 wrote: post #153
endorphine wrote: post #104

> there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. It's even worse than that: non -junior devs are doing it as well.

>It's even worse than that: non-junior devs are doing it as well. This might be unpopular, but that is seeming more like an opportunity if we want to continue allowing AI to generate code. One of the annoying things engineers have to deal with is stopping whatever they're doing and doing a review. Obviously this gets worse if more total code is being produced. We could eliminate that interruption by having someone do…

This sounds like what unit tests after every commit and e2e tests before every PR are supposed to solve.

Re: Your job is to deliver code you have proven to work

#349

Post by sodapopcan »

rootusrootus wrote: post #253
sodapopcan wrote: post #229

Earlier quoted context omitted.

Complaining about including a ticket number in the commit is a new one for me. Good grief.

It could be a death-by-a-thousand-cuts situation and we don't have enough context. My company has spent the last few years really going 1000% on the capitalization of software expenses, and now we have to include a whole slew of unrelated attributes in every last Jira ticket. Then the "engineering team" (there is only one of these, somehow, in a 5K employee company) decrees all sorts of requirements about how we test…

Ah ya, death-by-a-thousand-cuts is certainly a charitable take!

Re: Your job is to deliver code you have proven to work

#350

Post by sharkjacobs »

trevor-e wrote: post #159

> Your job is to deliver code you have proven to work. Strong disagree here, your job is to deliver solutions that help the business solve a problem. In _most_ cases that means delivering code that you should be able to confidently prove satisfies the requirements like the OP mentioned, but I think this is an important nitpick distinction I didn't understand until later on in my career.

Maybe I'm not late enough in my career to understand what you're saying, but what kind of problems are you helping the business solve with code that hasn't been proven to work?
Post reply on HN