Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your job is to deliver code you have proven to work

simonwillison.net

Post reply on HN
Print view View original

271–280 of 699 posts

Re: Your job is to deliver code you have proven to work

#271

Post by [deleted] »

oceanplexian wrote: post #135
simonw wrote: post #44

Earlier quoted context omitted.

100%. There's no difference at all in my mind between an AI-assisted PR and a regular PR: in both cases they should include proof that the change works and that the author has put the work in to test it.

At the last company I worked at (Large popular tech company) it took an act of the CTO to get engineers to simply attach a JIRA Ticket to the PR they were working on so we could track it for tax purposes. The Devs went in kicking and screaming. As an SRE it seemed like for SDEs, writing a description of the change, explaining the problem the code is solving, testing methodology, etc is harder than actually coding. Ir…

[deleted]

Re: Your job is to deliver code you have proven to work

#272

Post by roryirvine »

cynicalsecurity wrote: post #197
0xbadcafebee wrote: post #149

Actually it's more specific than that. A company pays you not just to "write code", not just to "write code that works", but to write code that works in the real world . Not on your laptop. Not in CI tests. Not on some staging environment. But in the real world . It may work fine in a theoretical environment, but deflate like a popped balloon in production. This code has no value to the business; they don't pay you t…

It gets interesting when a company assigns 2 story points to a task that requires 6 minimum. No time for writing tests, barely any time to perform code reviews and QA. Also, next year the company tells you since we have AI now, all tickets must be done 2 times quicker. Who popped this balloon? I know I need to change my employer, but it's not so easy. And I'm not sure another employer is going to be any better.

Are you not involved in doing the estimation?

Re: Your job is to deliver code you have proven to work

#273

Post by bee_rider »

sorokod wrote: post #178
hnthrow0287345 wrote: post #153

Earlier quoted context omitted.

>It's even worse than that: non-junior devs are doing it as well. This might be unpopular, but that is seeming more like an opportunity if we want to continue allowing AI to generate code. One of the annoying things engineers have to deal with is stopping whatever they're doing and doing a review. Obviously this gets worse if more total code is being produced. We could eliminate that interruption by having someone do…

> One of the annoying things engineers have to deal with is stopping whatever they're doing and doing a review. I would have thought that reviewing PRs and doing it well is in the job description. You latter mention "someone" a few times - who that someone might be?

Can we make an LLM do it?

“You are a cranky senior software engineer who loves to nitpick change requests. Here are your coding standards. You only sign off of a change after you are sure it works; if you run out of compute credits before you can prove it to yourself, reject the change as too complex.”

Balance things, pit the LLMs against each other.

Re: Your job is to deliver code you have proven to work

#274

Post by jopsen »

tyrust wrote: post #243
analog31 wrote: post #208

Earlier quoted context omitted.

Where are the junior devs while their code is being reviewed? I'm not a software developer, but I'd be loath to review someone's work unless they have enough skin in the game to be present for the review.

Code review is rarely done live. It's usually asynchronous, giving the reviewer plenty of time to read, digest, and give considered feedback on the changes. Perhaps a spicy patch would involve some kind of meeting. Or maybe in a mentor/mentee situation where you'd want high-bandwidth communication.

Doing only IRL code reviews would certainly improve quality in some projects :)

It's probably also fairly expensive to do.

Re: Your job is to deliver code you have proven to work

#275

Post by acedTrex »

endorphine wrote: post #104

> there’s one depressing anecdote that I keep on seeing: the junior engineer, empowered by some class of LLM tool, who deposits giant, untested PRs on their coworkers—or open source maintainers—and expects the “code review” process to handle the rest. It's even worse than that: non -junior devs are doing it as well.

Juniors aren't even the problem here, they can and should be taught better thats the point.

Its when your PEERS do it that its a huge problem.

Re: Your job is to deliver code you have proven to work

#276

Post by stuffn »

simianwords wrote: post #255
layer8 wrote: post #138

I’d go further and say while testing is necessary, it is not sufficient. You have to understand the code and convince yourself that it is logically correct under all relevant circumstances, by reasoning over the code. Testing only “proves” correctness for the specific state, environment, configuration, and inputs the code was tested with. In practice that only tests a tiny portion of possible circumstances, and omits…

I would like to challenge this claim. I think LLMs are maybe accurate enough that we don't need to check every line and remember everything. High level design is enough.

I have plenty of anecdata that counters your anecdata.

LLMs can generate code that works. That much is true. You can generate sufficiently complex projects that simply run on the first (or second try). You can even get the LLM to write tests for the code. You can prompt it for 100% test coverage and it will provide you exactly what you want.

But that doesn't mean OP isn't correct. First, you shouldnt be remembering everything. If you are finding yourself remembering everything your project is either small (I'd guess less than 1000 lines) or you are overburdened and need help. Reasoning, logically, through code you write can be done JIT as you're writing the code. LLMs even suffer from the same problem. Instead of calling it "having to remember to much" we refer to it as a quantity called "context window". The only problem is the LLM won't prompt you telling you that it's context window is so full it can't do it's job properly. A human will.

I think an engineer should always be reasoning about their code. They should be especially suspicious of LLM generated code. Maybe I'm alone but if I use an LLM to generate code I will review it and typically end up modifying it. I find even prompting with something like "the code you write should be maintainable by other engineers" doesn't produce good value.

Re: Your job is to deliver code you have proven to work

#277

Post by just_once »

simonw wrote: post #191
just_once wrote: post #162

I don't know if there's a word for this but this reads to me as like, software virtue signaling or software patronizing. It's bizarre to me to tell an engineer what their job is as a matter of fact and to claim a particular usage of a tool as mandated (a tool that no one really asked for, mind you), leveraging duty of all things. I guess to me, it's either the case that LLMs are just another tool, in which case the a…

LLMs are just another tool, but they're disruptive enough that existing best practices need to be either updated or re-explained. A lot of people using LLMs seem not to have understood that you can't expect them to write code that works without testing it first! If that wasn't clearly a problem I wouldn't have felt the need to write this.

Yep, it's a real problem. No dispute there.

My intention isn't to argue a point, just to share my perspective when I read it.

I read your response here to be saying something like "I noticed that people are misunderstood about X, so I wanted to inform them". In this case "X" isn't itself very obvious to me (For any given task, why can't you expect that a cutting edge LLM would be able to write it without requiring your testing that?) but most importantly, I don't think I would approach a pure misunderstanding (tantamount to a skills gap) with your particular framing. Again, to me it reads as patronizing.

Love the pelican on the bicycle, though. I think that's been a great addition to the zeitgeist.

Re: Your job is to deliver code you have proven to work

#278

Post by cm2012 »

bee_rider wrote: post #273
sorokod wrote: post #178

Earlier quoted context omitted.

> One of the annoying things engineers have to deal with is stopping whatever they're doing and doing a review. I would have thought that reviewing PRs and doing it well is in the job description. You latter mention "someone" a few times - who that someone might be?

Can we make an LLM do it? “You are a cranky senior software engineer who loves to nitpick change requests. Here are your coding standards. You only sign off of a change after you are sure it works; if you run out of compute credits before you can prove it to yourself, reject the change as too complex.” Balance things, pit the LLMs against each other.

This would probably catch a lot of errors

Re: Your job is to deliver code you have proven to work

#279

Post by BeefySwain »

WhyOhWhyQ wrote: post #263

Isn't this in contradiction to your blog post from yesterday though? It's impossible to prove a complex project made in 4.5 hours works. It might have passed 9000 tests, but surely there are always going to be edge cases. I personally wouldn't be comfortable claiming I've proved it works and saying the job is done even, if the LLM did the whole thing and all existing tests passed, until I played with it for several m…

Consider that this isn't just a random AI slopped assortment of 9,000 tests, but instead is a robust suite of tests that cover 100% of the HTML5 spec.

Does this guarantee that it functions completely with no errors whatsoever? Certainly not. You need formal verification for that. I don't think that contradicts what Simon was advocating for though in this post.

Re: Your job is to deliver code you have proven to work

#280

Post by just_once »

dkural wrote: post #170
just_once wrote: post #162

I don't know if there's a word for this but this reads to me as like, software virtue signaling or software patronizing. It's bizarre to me to tell an engineer what their job is as a matter of fact and to claim a particular usage of a tool as mandated (a tool that no one really asked for, mind you), leveraging duty of all things. I guess to me, it's either the case that LLMs are just another tool, in which case the a…

I've really seen both I suppose. A lot of devs don't take accountability / responsibility for their code, especially if they haven't done anything that actually got shipped and used, or in general haven't done much responsible adulting.

No doubt. Interesting to think about why that is without assuming it's a character flaw.
Post reply on HN