Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your job is to deliver code you have proven to work

simonwillison.net

Post reply on HN
Print view View original

61–70 of 699 posts

Re: Your job is to deliver code you have proven to work

#61

Post by lm28469 »

wizzwizz4 wrote: post #8
dfxm12 wrote: post #6

there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. Is anyone else seeing this in their orgs? I'm not...

It's not a new phenomenon. Time was, people would copy-paste from blog posts with the same effect.

Always the same old tiring "this has always been possible before in some remotely similar fashion hence we should not criticise anything ever again" argument.

You could intuitively think it's just a difference of degree, but it's more akin to a difference of kind. Same for a nuke vs a spear, both are weapons, no one argues they're similar enough that we can treat them the same way

Re: Your job is to deliver code you have proven to work

#62

Post by dfxm12 »

x3n0ph3n3 wrote: post #40
dfxm12 wrote: post #6

there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. Is anyone else seeing this in their orgs? I'm not...

It's been a struggle with a few teammates that we are trying to solve through explicit policy, feedback, and ultimately management action.

Yeah, a slice of this is technology related, but it's really a policy issue. It's probably easier to manage with a tighter team. Maybe I'm taking team size for granted.

Re: Your job is to deliver code you have proven to work

#63

Post by visarga »

smokel wrote: post #47
visarga wrote: post #41

I agree with the author overall. Manual testing is what I call "vibe testing" and I think by itself is insufficient, no matter if you or the agent wrote the code. If you build your tests well, using the coding agent becomes smooth and efficient, and the agent is safe to do longer stretches of work. If you don't do testing, the whole thing is just a bomb ticking in your face. My approach to coding agents is to prepare…

This depends on the type of software you make. Testing the usability of a user interface for example, is something you can't automate (yet). So, ehm, it depends :)

It will come around, we have rudimentary computer use agents and ability to record UIs for LLM agents. They will me refined and the agent can test UIs as well.

For UIs I do a different trick - live diagnostic tests - I ask the agent to write tests that run in the app itself, check consistencies, constraints and expected behaviors. Having the app running in its natural state makes it easier to test, you can have complex constraints encoded in your diagnostics.

Re: Your job is to deliver code you have proven to work

#64

Post by lm28469 »

briliantbrandon wrote: post #18
dfxm12 wrote: post #6

there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. Is anyone else seeing this in their orgs? I'm not...

I'm seeing a little bit of this. However, I will add that the primary culprits are engineers that were submitting low quality PRs before they had access to LLMs, they can just submit them faster now.

LLMs are tools that make mediocre devs 100x more "productive" and good devs 2x more productive

Re: Your job is to deliver code you have proven to work

#66

Post by falcor84 »

cess11 wrote: post #33
Rperry2174 wrote: post #19

Im not fully convinced by "a computer can never be held accountable" We already delegate accountability to non-humans all the time: - CI systems block merges - monitoring systems page people - test suites gate different things In practice accountability is enforced by systems, not humans.. humans are defintiely "blamed" after the fact, but the day-to-day control loop is automated. As agents get better at running code…

Right, so how do you hold these things accountable? When your CI fails, what do you do? Type in a starkly worded message into a text file and shut off the power for three hours as a punishment? Invoice Intel?

Well, we're not there yet, but I do envision a future, where some AIs work for as independent contractors with their own bank accounts that they want to maximize, and if such an AI fails in a bad way, its client would be able to fine it, fire it or even sue it, so that it, and the human controlling it would be financially punished.

Re: Your job is to deliver code you have proven to work

#67

Post by [deleted] »

Rperry2174 wrote: post #19

Im not fully convinced by "a computer can never be held accountable" We already delegate accountability to non-humans all the time: - CI systems block merges - monitoring systems page people - test suites gate different things In practice accountability is enforced by systems, not humans.. humans are defintiely "blamed" after the fact, but the day-to-day control loop is automated. As agents get better at running code…

[deleted]

Re: Your job is to deliver code you have proven to work

#68

Post by davey48016 »

dfxm12 wrote: post #6

there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. Is anyone else seeing this in their orgs? I'm not...

A friend of mine has a junior engineer who does this and then responds to questions like "Why did you do X?" with "I didn't, Claude did, I don't know why".

Re: Your job is to deliver code you have proven to work

#69

Post by 9rx »

wrsh07 wrote: post #48
9rx wrote: post #3

> Your job is to deliver code you have proven to work. Your job is to solve customer problems. Their problems may only be solvable with code that is proven to work, but it is equally likely (I dare say even more likely) that their problem isn't best solved with code at all, or even solved with code that doesn't work properly but works well enough.

I would argue that the word "proof" in the title might be misleading you. From the post and the example he links, the point is that if you don't at least look at the running code, you don't know that it works. In my opinion the point is actually well illustrated by Chris's talk here: https://v5.chriskrycho.com/elsewhere/seeing-like-a-programme... (summary of the relevant section if you're not going to click) >>> In t…

> if you don't at least look at the running code, you don't know that it works.

Your tests run the code. You know it works. I know the article is trying to say that testing is not comprehensive enough, but my experience disagrees. But I also recognize that testing is not well understood (quite likely the least understood aspect of computer science!) — and if you don't have a good understanding you can get caught not testing the right things or not testing what you think you are. I would argue that you would be better off using your time to learn how to write great tests instead of using it to manually test your code, but to each their own.

What is more likely to happen is not understanding the customer needs well enough, leaving it impossible to write tests that align with what the software needs to do. Software development can break down very quickly here. However, manual testing does not help. You can't know what to manually test without understanding the problem either. However, as before, your job is not to deliver proven code. Your job is to solve customer problems. When you realize that, it becomes much less likely that you write tests that are not in line with the solution you need.

Re: Your job is to deliver code you have proven to work

#70

Post by falcor84 »

simonw wrote: post #54
weatherlite wrote: post #43

> Almost anyone can prompt an LLM to generate a thousand-line patch and submit it for code review. That’s no longer valuable. What’s valuable is contributing code that is proven to work. That's really not a great development for us. If our main point is now reduced to accountability over the result with barely any involvement in the implementation - that's very little moat and doesn't command a high salary. Either we…

The value is being better at it than any QA or product manager. Experienced software engineers have such a huge edge over everyone else with this stuff. If your product manager doesn't understand what a CORS header is good luck having them produce a change that requires cross-domain fetch() call... and first they'll have to know what a "cross-doman fetch() call" means. And sure they could ask an LLM about that, but t…

That's an interesting argument, but from my industry experience, the average experienced QA Engineer and technical Product Manager both have better vocabulary than the average SWE. Indeed, I wonder whether a future curriculum for Vibe Engineering (to borrow your own term) may look more similar to that of present-day QA or Product curricula, than to a typical coding or CS curriculum.
Post reply on HN