[dead]
Your job is to deliver code you have proven to work
281–290 of 699 posts
Re: Your job is to deliver code you have proven to work
#282> there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. It's even worse than that: non -junior devs are doing it as well.
Not at Meta - their job is to "Move fast and break things". I think people are just doing what they've been told.
“Move fast and break things” works well when you are a little player in a big world, because you can only perturb the system into so bad a state with you limited resources. Now, they got big, and everything is broken.
Re: Your job is to deliver code you have proven to work
#283I know this won't be popular, however, I think the idea of differentiating a "real developer" from one who relies mostly, or even solely on an LLM is coming to an end. Right now, I fully agree relying wholly upon an LLM and failing to test it is very irresponsible. LLMs do make mistakes. They do a sloppy job at times. But give it a year. Two years. five years. It seems unreasonable to assume they will hit a plateau t…
What's your point, though? Let's assume your hypothesis and 5 years from now everyone has access to an LLM that's as good as a typical staff engineer. Is it now acceptable for a junior engineer to submit LLM-generated PRs without having tested them?
> It was thought impossible for a computer to reach the point of being able to beat a grandmaster at chess.
This is oft-cited but it takes only some cursory research to show that it has never been close to a universally-held view.
Re: Your job is to deliver code you have proven to work
#284Isn't this in contradiction to your blog post from yesterday though? It's impossible to prove a complex project made in 4.5 hours works. It might have passed 9000 tests, but surely there are always going to be edge cases. I personally wouldn't be comfortable claiming I've proved it works and saying the job is done even, if the LLM did the whole thing and all existing tests passed, until I played with it for several m…
Consider that this isn't just a random AI slopped assortment of 9,000 tests, but instead is a robust suite of tests that cover 100% of the HTML5 spec. Does this guarantee that it functions completely with no errors whatsoever? Certainly not. You need formal verification for that. I don't think that contradicts what Simon was advocating for though in this post.
I think it would be interesting if professional engineering becomes more like producing formally correct documents for the AI to implement.
Re: Your job is to deliver code you have proven to work
#285I think the problem is in what “proven” means. People that don’t know any better will just do that all with LLMs and still deliver the giant untested PRs but with some LLM written tests attached. I vibe code a lot of stuff for myself, mostly for viewing data, when I don’t really need to care how it works. I’m coming around to the idea that outside of some specific circumstances where everyone has agreed they don’t ne…
That's why I emphasized the manual testing component as well. Attaching a screenshot or video of a feature working to your PR is a great way to prove that you've actually seen it work correctly - at least once, which is still a huge improvement over it not actually working at all.
Yes! This is something that I also value. Having demo gifs of before and after helps a lot. I have encountered situations where what I thought was a minor finishing clean up had an effect that I didn't anticipate. By including demos in the PR it becomes a kind of guardrail against those situations for me. I also think it is neat and generally helpful for everyone.
Re: Your job is to deliver code you have proven to work
#286> The first is manual testing. If you haven’t seen the code do the right thing yourself, that code doesn’t work. If it does turn out to work, that’s honestly just pure chance.
Depending on exactly what the author meant here, I disagree. Our first and default tool should be some form of lightweight automated testing. It's explicit (serves a form of spec and docs how to use the software), it's repeatable (manual testing is done once and it's result is invalidated moments later), and it's cost per minute of effort is more or less the same (most companies have the engineers do the tests, they are expensive).
Yes. There will be exceptions and exceptional cases. This author is not talking about exceptions and neither am I. They're not an interesting addition to this conversation.
Re: Your job is to deliver code you have proven to work
#287Earlier quoted context omitted.
Quality is not rewarded at most companies, it's not going to turn into more money, it might turn into less work later, but in all likelihood, the author won't be around to reap the benefits of less work later because they will have moved onto another company. On the contrary, since more effort doesn't yield more money, but less effort can yield the same money, the strategy is to contract the time spent on work to the…
Not everything is about money. Have you never wanted to be good at something because you enjoy it? Or do something for the love of the craft? Have you heard of altruism?
But why do that for the company instead of yourself?
Re: Your job is to deliver code you have proven to work
#288Im not fully convinced by "a computer can never be held accountable" We already delegate accountability to non-humans all the time: - CI systems block merges - monitoring systems page people - test suites gate different things In practice accountability is enforced by systems, not humans.. humans are defintiely "blamed" after the fact, but the day-to-day control loop is automated. As agents get better at running code…
Humans are only kind of held accountable. If you ship a bug do you go to jail? Even a bug so bad it puts your company out of business. Would there be any legal or physical or monetary consequences at all for you, besides you lose your job?
So the accountability situation for AI seems not that different. You can fire it. Exactly the same as for humans.
Re: Your job is to deliver code you have proven to work
#289> there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. It's even worse than that: non -junior devs are doing it as well.
In the company I’m at this is beginning to happen. PM’s want to “prototype” new features and expect the engineers to finish up the work. With the expectation that it ‘just needs some polishing’. What would be your recommendation on how to handle this constructively? Flat out rejecting LLM as a prototyping tool is not an option.
I would accept this because it'll increase demand for SWEs and prevent us from losing our jobs.
Re: Your job is to deliver code you have proven to work
#290there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. Is anyone else seeing this in their orgs? I'm not...
Voila: https://github.com/WireGuard/wireguard-android/pull/82 https://github.com/WireGuard/wireguard-android/pull/80 In that first one, the double pasted AI retort in the last comment is pretty wild. In both of these, look at the actual "files changed" tab for the wtf.
That's a good example of what we're seeing as leads, thanks.