Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your job is to deliver code you have proven to work

simonwillison.net

Post reply on HN
Print view View original

11–20 of 699 posts

Re: Your job is to deliver code you have proven to work

#11

Post by simonw »

andy99 wrote: post #5

I think the problem is in what “proven” means. People that don’t know any better will just do that all with LLMs and still deliver the giant untested PRs but with some LLM written tests attached. I vibe code a lot of stuff for myself, mostly for viewing data, when I don’t really need to care how it works. I’m coming around to the idea that outside of some specific circumstances where everyone has agreed they don’t ne…

That's why I emphasized the manual testing component as well. Attaching a screenshot or video of a feature working to your PR is a great way to prove that you've actually seen it work correctly - at least once, which is still a huge improvement over it not actually working at all.

Re: Your job is to deliver code you have proven to work

#12

Post by simonw »

allcentury wrote: post #4

Manual testing as the first step… not very productive imo. Outside in testing is great but I typically do automated outside in testing and only manual at the end. The loop process of testing needs to be repeatable and fast, manual is too slow

Yeah that's fair, the manual testing doesn't have to sequentially go first - but it does have to get done.

I've lost count of the number of times I've skipped it because the automated test passed and then found there was some dumb but obvious bug that I missed, instantly exposed when I actually exercised the feature myself.

Re: Your job is to deliver code you have proven to work

#13

Post by jairuhme »

webdev1234568 wrote: post #7

Whole article seems very much all llm generated Edit: I'm an idiot ignore me.

Did you read the article and come to that conclusion or just blindly count the number of em-dashes and assume that? Because I don't get the impression that it was LLM generated

Re: Your job is to deliver code you have proven to work

#14

Post by zkmon »

How about letting LLMs maintain a vast number of product versions all available at the same, which receive multiple versions of untested versions of the same patch, from LLMs, and then let the models elect a version of the software based on probabilistic or gradient methods? This elected version could change for different assessments. No human touches or looks at the code!

Just a wild thought, nothing serious.

Re: Your job is to deliver code you have proven to work

#16

Post by simonw »

webdev1234568 wrote: post #7

Whole article seems very much all llm generated Edit: I'm an idiot ignore me.

Not a single word of it was. I wrote this one entirely in Apple Notes, so there weren't even any VS Code completed sentences

It has emdashes because my blog turns " - " into an emdash here: https://github.com/simonw/simonwillisonblog/blob/06e931b397f...

Re: Your job is to deliver code you have proven to work

#17

Post by jennyholzer2 »

dfxm12 wrote: post #6

there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. Is anyone else seeing this in their orgs? I'm not...

[dead]

Re: Your job is to deliver code you have proven to work

#18

Post by briliantbrandon »

dfxm12 wrote: post #6

there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. Is anyone else seeing this in their orgs? I'm not...

I'm seeing a little bit of this. However, I will add that the primary culprits are engineers that were submitting low quality PRs before they had access to LLMs, they can just submit them faster now.

Re: Your job is to deliver code you have proven to work

#19

Post by Rperry2174 »

Im not fully convinced by "a computer can never be held accountable"

We already delegate accountability to non-humans all the time: - CI systems block merges - monitoring systems page people - test suites gate different things

In practice accountability is enforced by systems, not humans.. humans are defintiely "blamed" after the fact, but the day-to-day control loop is automated.

As agents get better at running code, inspecting ui state, correlating logs, screenshots, etc they're starting to operationally be "accountable" and preventing bad changes from shipping and producing evidence when something goes wrong .

At some point humans role shifts from "i personally verify this works" to "i trust this verification system and am accountable for configuring it correctly".

Thats still responsibility, but kind of different from whats described here. Taken to a logical extreme, the arguement here would suggest that CI shouldn't replace manual release checklists

Post reply on HN