Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your job is to deliver code you have proven to work

simonwillison.net

Post reply on HN
Print view View original

551–560 of 699 posts

Re: Your job is to deliver code you have proven to work

#551

Post by tunesmith »

endorphine wrote: post #104

> there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. It's even worse than that: non -junior devs are doing it as well.

As always, this requires nuance. Just yesterday and today, I did exactly that to my direct reports (I'm director-level). We had gotten a bug report, and the team had collectively looked into it and believed it was not our problem, but that of an external vendor. Reported it to the vendor, who looked into it, tested it, and then pushed back and said it was our problem. My team is still more LLM-averse than me, so I had Codex look at it, and it believed it found the problem and prepared the PR. I did not review or test the PR myself, but instead assigned it to the team to validate, partly for learnings. They looked it over and agreed it was a valid fix for a problem on our side. I believe that process was better than me just fully validating it myself, and part of the process toward encouraging them to use LLM as a tool for their work.

Re: Your job is to deliver code you have proven to work

#552

Post by Yodel0914 »

dfxm12 wrote: post #6

there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. Is anyone else seeing this in their orgs? I'm not...

Not so much the huge PRs, but definitely the LLM generated code that the “developer” doesn’t understand.

Re: Your job is to deliver code you have proven to work

#554

Post by xyzzy_plugh »

tunesmith wrote: post #551
endorphine wrote: post #104

> there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. It's even worse than that: non -junior devs are doing it as well.

As always, this requires nuance. Just yesterday and today, I did exactly that to my direct reports (I'm director-level). We had gotten a bug report, and the team had collectively looked into it and believed it was not our problem, but that of an external vendor. Reported it to the vendor, who looked into it, tested it, and then pushed back and said it was our problem. My team is still more LLM-averse than me, so I ha…

> I believe that process was better than me just fully validating it myself

Why?

> and part of the process toward encouraging them to use LLM as a tool for their work.

Did you look at it from their perspective? You set the exact opposite example and serve as a perfect example for TFA: you did not deliver code you have proven to work. I imagine some would find this demoralizing.

I've worked with a lot of director-level software folk and many would just do the work. If they're not going to do the work, then they should probably assign someone to do it.

What if it didn't work? What if you just wasted a bunch of engineering time reviewing slop? I don't comprehend this mindset. If you're supposedly a leader, then lead.

Re: Your job is to deliver code you have proven to work

#555

Post by layer8 »

anthonypasq wrote: post #366
layer8 wrote: post #138

I’d go further and say while testing is necessary, it is not sufficient. You have to understand the code and convince yourself that it is logically correct under all relevant circumstances, by reasoning over the code. Testing only “proves” correctness for the specific state, environment, configuration, and inputs the code was tested with. In practice that only tests a tiny portion of possible circumstances, and omits…

if your tests cover the acceptance criteria as defined in the ticket, why is all htat other stuff necessary?

If your acceptance criteria state something like “produces output f(x) for any inout x, where f(x) is defined as follows: […]”, then you can’t possibly test that, because you can’t test all possible values of x. And if the criteria don’t state that, then they don’t cover the full specification of how the software is expected to behave, hence you have to go beyond those criteria to ensure that the software always behaves as expected.

You can’t prove that something is correct by example. Examples can only disprove correctness. And tests are always only examples.

Re: Your job is to deliver code you have proven to work

#556

Post by Swannie »

robgibbons wrote: post #26

For what it's worth, writing good PRs applies in more cases than just AI generated contributions. In my PR descriptions, I usually start by describing how things currently work, then a summary of what needs to change, and why. Then I go on to describe what exactly is changing with the PR. This high level summary serves to educate the reviewer, and acts as a historical record in the git log for the benefit of those wh…

If only there were community standards for this...

Oh, there are, for years :D This has really stood the test of time:

https://rfc.zeromq.org/spec/42/#24-development-process

And its rationale is well explained too:

https://hintjens.gitbooks.io/social-architecture/content/cha...

Saddened by realizing that Pieter would have had amazing things to say about AI.

Re: Your job is to deliver code you have proven to work

#557

Post by layer8 »

9rx wrote: post #453
layer8 wrote: post #138

I’d go further and say while testing is necessary, it is not sufficient. You have to understand the code and convince yourself that it is logically correct under all relevant circumstances, by reasoning over the code. Testing only “proves” correctness for the specific state, environment, configuration, and inputs the code was tested with. In practice that only tests a tiny portion of possible circumstances, and omits…

Testing is not perfect, but what else is there? Even formal proofs are just another expression of testing. With greater mathematical guarantees than other expressions, granted, but still testing all the same; prone to all the very same human problems testing is burdened with.

The difference with proofs (whether formal or informal) is that they quantify over all possible cases, whereas testing is always limited to specific cases.

Re: Your job is to deliver code you have proven to work

#558

Post by mullingitover »

distances wrote: post #516
mullingitover wrote: post #233

Earlier quoted context omitted.

> expects the “code review” process to handle the rest. The LLMs/agents have actually been doing a stellar job with code reviews. Frankly that’s one area that humans rush through, to the point it’s a running joke that the best way to get a PR granted a “lgtm” is to make it huge. I’ve almost never seen Copilot wave a PR through on the first attempt, but I usually see humans doing that.

That smells of bad team practices. Put a practical limit on PRs sizes as the first step, around 500 lines max is a good rule of thumb in my experience. Larger than that, and the expectation then is a number of small PRs to a feature branch. I rarely see a PR that should pass without comments. Your team is being sloppy.

> Your team is being sloppy.

I'm talking about a running joke in the industry, not my team.

Re: Your job is to deliver code you have proven to work

#559

Post by rcxdude »

rootusrootus wrote: post #242
endorphine wrote: post #104

> there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. It's even worse than that: non -junior devs are doing it as well.

One thing I've pushed developers on my team to do since way before AI slop became a thing was to review their own PR. Go through the PR diff and leave comments in places where it feels like a little explanation of your thought process could be helpful in the review. It's a bit like rubber duck debugging, I've seen plenty of things get caught that way. As an upside, it helps with AI slop too. Because as I see it, what…

Yeah, I always think it's kinda rude to throw something to someone else to review without reviewing it yourself, even if you were the one to write it. Looking at it twice yourself can help with catching things even faster than someone else getting up to speed with what you were doing and querying it. Now it seems like with LLMs people are putting code up for review that hasn't even been looked at once.

Re: Your job is to deliver code you have proven to work

#560

Post by Swannie »

Posted down thread, but worth posting as a comment too.

I know Simon follows this "Issue First" style of work in his projects, with a strong requirement for passing tests to be included.

It's been a best practice for a long time. I really enjoyed this when I read it ~10 years ago, and it still stands the test of time:

https://rfc.zeromq.org/spec/42/#24-development-process

The rationale was articulated clearly in:

https://hintjens.gitbooks.io/social-architecture/content/cha...

If you have time, do yourself a favour and read the whole lot. And then liberally copy parts of C4 into your own process. I have advocated for many components of it, in many contexts, at $employer, and will continue to do so.

Post reply on HN