Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your job is to deliver code you have proven to work

simonwillison.net

Post reply on HN
Print view View original

41–50 of 699 posts

Re: Your job is to deliver code you have proven to work

#41

Post by visarga »

I agree with the author overall. Manual testing is what I call "vibe testing" and I think by itself is insufficient, no matter if you or the agent wrote the code. If you build your tests well, using the coding agent becomes smooth and efficient, and the agent is safe to do longer stretches of work. If you don't do testing, the whole thing is just a bomb ticking in your face.

My approach to coding agents is to prepare a spec at the start, as complete as possible, and develop a beefy battery of tests as we make progress. Yesterday there was a story "I ported JustHTML from Python to JavaScript with Codex CLI and GPT-5.2 in hours". They had 9000+ tests. That was the secret juice.

So the future of AI coding as I see it ... it will be better than pre-2020, we will learn to spec and plan good tests, and the tests are actually our contract the code does what is supposed to do. You can throw away the code and keep the specs and tests and regenerate any time.

Re: Your job is to deliver code you have proven to work

#42

Post by robryk »

Rperry2174 wrote: post #19

Im not fully convinced by "a computer can never be held accountable" We already delegate accountability to non-humans all the time: - CI systems block merges - monitoring systems page people - test suites gate different things In practice accountability is enforced by systems, not humans.. humans are defintiely "blamed" after the fact, but the day-to-day control loop is automated. As agents get better at running code…

Why do you think that this other kind of accountability (which reminds me of the way captain's or commander's responsibility is often described) is incompatible with what the article describes? Due to the focus on necessity of manual testing?

Re: Your job is to deliver code you have proven to work

#43

Post by weatherlite »

> Almost anyone can prompt an LLM to generate a thousand-line patch and submit it for code review. That’s no longer valuable. What’s valuable is contributing code that is proven to work.

That's really not a great development for us. If our main point is now reduced to accountability over the result with barely any involvement in the implementation - that's very little moat and doesn't command a high salary. Either we provide real value or we don't ...and from that essay I think it's not totally clear what the value is - it seems like every QA, junior SWE or even product manager can now do the job of prompting and checking the output.

Re: Your job is to deliver code you have proven to work

#44

Post by simonw »

robgibbons wrote: post #26

For what it's worth, writing good PRs applies in more cases than just AI generated contributions. In my PR descriptions, I usually start by describing how things currently work, then a summary of what needs to change, and why. Then I go on to describe what exactly is changing with the PR. This high level summary serves to educate the reviewer, and acts as a historical record in the git log for the benefit of those wh…

100%. There's no difference at all in my mind between an AI-assisted PR and a regular PR: in both cases they should include proof that the change works and that the author has put the work in to test it.

Re: Your job is to deliver code you have proven to work

#45

Post by rmnclmnt »

> As software engineers we…

That’s the thing. People exposing such rude behavior usually are not, or haven’t been in a looong time…

As for the local testing part not being performed, this is a slippery slope I’m fighting everyday: more and more cloud based services and platforms are used to deploy software to run with specific shenanigans and running it locally requires some kind of deep craft and understanding. Vendor lock-in is coming back in style (e.g. Databricks)

Re: Your job is to deliver code you have proven to work

#46

Post by webdev1234568 »

simonw wrote: post #16
webdev1234568 wrote: post #7

Whole article seems very much all llm generated Edit: I'm an idiot ignore me.

Not a single word of it was. I wrote this one entirely in Apple Notes, so there weren't even any VS Code completed sentences It has emdashes because my blog turns " - " into an emdash here: https://github.com/simonw/simonwillisonblog/blob/06e931b397f...

My biggest appologies, a very bad move on my part. I'll pay more attention before any sort of accusation like this

Re: Your job is to deliver code you have proven to work

#47

Post by smokel »

visarga wrote: post #41

I agree with the author overall. Manual testing is what I call "vibe testing" and I think by itself is insufficient, no matter if you or the agent wrote the code. If you build your tests well, using the coding agent becomes smooth and efficient, and the agent is safe to do longer stretches of work. If you don't do testing, the whole thing is just a bomb ticking in your face. My approach to coding agents is to prepare…

This depends on the type of software you make. Testing the usability of a user interface for example, is something you can't automate (yet). So, ehm, it depends :)

Re: Your job is to deliver code you have proven to work

#48

Post by wrsh07 »

9rx wrote: post #3

> Your job is to deliver code you have proven to work. Your job is to solve customer problems. Their problems may only be solvable with code that is proven to work, but it is equally likely (I dare say even more likely) that their problem isn't best solved with code at all, or even solved with code that doesn't work properly but works well enough.

I would argue that the word "proof" in the title might be misleading you.

From the post and the example he links, the point is that if you don't at least look at the running code, you don't know that it works.

In my opinion the point is actually well illustrated by Chris's talk here:

https://v5.chriskrycho.com/elsewhere/seeing-like-a-programme...

(summary of the relevant section if you're not going to click)

>>>

In the talk "Seeing Like a Programmer," Chris Krycho quotes the conductor and composer Eímear Noone, who said:

> "The score is potential energy. It's the potential for music to happen, but it's not the music."

He uses this quote to illustrate the distinction between "software as artifact" (the code/score) and "software as system" (the running application/music). His point is that the code itself is just a static artifact—"potential energy"—and the actual "software" only really exists when that code is executed and running in the real world.

Re: Your job is to deliver code you have proven to work

#49

Post by zkmon »

throwuxiytayq wrote: post #23
zkmon wrote: post #14

How about letting LLMs maintain a vast number of product versions all available at the same, which receive multiple versions of untested versions of the same patch, from LLMs, and then let the models elect a version of the software based on probabilistic or gradient methods? This elected version could change for different assessments. No human touches or looks at the code! Just a wild thought, nothing serious.

Talk is cheap. Show me the proompt.

That's hard for me. Feed my comment to a model and ask for prompts.

Re: Your job is to deliver code you have proven to work

#50

Post by andy99 »

jennyholzer2 wrote: post #34
dfxm12 wrote: post #22

Earlier quoted context omitted.

What's the ratio of people who things the right way vs not? I mean, is it a matter of giving them feedback to remind them what a "quality PR" is? Does that help?

LLMs have dramatically empowered sociopath software developers. If you are sufficiently motivated to appear more "productive" than your coworkers, you can force them to review thousands of lines of incorrect AI slop code while you sit back and mess around with your chatbots. Your coworkers no longer have enough time to work on their in-progress PRs, so you can dominate the development team in terms of LOC shipped. Un…

Only if leadership lets them. Right now (anecdotally) a lot of “leaders” don’t understand the difference between AI generated and human generated work, and just look at loc as productivity so all incentives are on AI coding, but that will change.
Post reply on HN