Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reviews have become expensive, rewrites have become cheap

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Reviews have become expensive, rewrites have become cheap

#61

Post by bob1029 »

LLM generated code does have some advantages at code review time. I find myself paying more attention to the diffs being in the right areas than their actual line-by-line content. It's been a long time since I've seen something like a conditional hallucinated incorrectly. The accuracy in the details is exceptional now. It's mostly a matter of ensuring we are dealing with the correct details.

The compiler does a lot of heavy lifting in the codebases I work with. Strongly typed languages seem mandatory if you want to use an LLM. Worrying about every symbol is too much. Let the CPU get hot for a few seconds on the paranoia and stay focused on the minimap.

Re: Reviews have become expensive, rewrites have become cheap

#63

Post by JodieBenitez »

usernametaken29 wrote: post #37

> The shortest path for the model is to implement it completely Have you worked with LLMs??????????? “I disabled the test so it’s not run so now all the tests pass” is not a hypothetical it’s pretty common. LLMs frequently do shortcut learning. The reason why reviews are expensive is because you still need to do all the steps in order to understand if a shortcut is justified.

> “I disabled the test so it’s not run so now all the tests pass” is not a hypothetical it’s pretty common.

I've never seen that though.

Re: Reviews have become expensive, rewrites have become cheap

#64

Post by elefanto »

ben_w wrote: post #55
SCdF wrote: post #47

> LLMs aren’t lazy. They don’t cut corners because a simpler solution feels good enough. If they know how to solve something thoroughly, they will. I don't know why they think this, but no? Perhaps it's badly expressed, but LLMs cut corners all the time. It's sort of their core fault really. Anyway, I disagree with the core premise[1]. Re-writes are not cheap, because 1) code can be so bad it's unclear how to rewrite…

> I don't know why they think this, but no? Perhaps it's badly expressed, but LLMs cut corners all the time. It's sort of their core fault really. I think this is a matter of perspective about what counts as "cutting corners". I think they look like you describe only because they have limited competence; this is on the basis that when I asked one to make a fusion reactor simulator (to see if it could) by using open s…

The diligence to the absurd that you describe is really just a consequence of cutting corners at planning stages. Consider an organization that relies heavily on formal proofs and specifications to one that uses what fits on the 3 bullets allowed in a PowerPoint slide. The first might do less work over all because the second is cutting corners in planning.

Re: Reviews have become expensive, rewrites have become cheap

#65

Post by ben_w »

elefanto wrote: post #64
ben_w wrote: post #55

Earlier quoted context omitted.

> I don't know why they think this, but no? Perhaps it's badly expressed, but LLMs cut corners all the time. It's sort of their core fault really. I think this is a matter of perspective about what counts as "cutting corners". I think they look like you describe only because they have limited competence; this is on the basis that when I asked one to make a fusion reactor simulator (to see if it could) by using open s…

The diligence to the absurd that you describe is really just a consequence of cutting corners at planning stages. Consider an organization that relies heavily on formal proofs and specifications to one that uses what fits on the 3 bullets allowed in a PowerPoint slide. The first might do less work over all because the second is cutting corners in planning.

Perhaps, though I find myself disagreeing as the planning stage where it itself told me about WarpX. It just then made stupid decisions when left to itself.

Can plan… just not well. Can code… just ok. Can do TDD… just most of the time.

Definitely none of these well enough to be a dark factory though.

Re: Reviews have become expensive, rewrites have become cheap

#66

Post by addingfriction »

rogerrogerr wrote: post #41
gobdovan wrote: post #27

Earlier quoted context omitted.

> Style is not neutral; it gives moral directions. > Nowadays every business in America says how warm it is and how much it cares — loan companies, supermarkets, hamburger chains. Guess which one is AI and which one is a quote from Martin Amis.

Without looking, I’m guessing one of these was cherry-picked from Amis and one was plucked nearly at random from LinkedIn. The writing style AI uses has its place, but not as _every sentence_. That’s what is exasperating. At the same time, I’m happy that I can still at least identify AI prose of more-than-trivial length.

both are Amis

Re: Reviews have become expensive, rewrites have become cheap

#67

Post by jkrejcha »

saidinesh5 wrote: post #33

We faced a lot of this this year. Eager new joinees sent 100s of lines of CLs to review to "improve" little things. Not only did the new changes did not fix what they thought it would fix but it broke other things in unexpected ways. I brought in two changes after that: * I'm not reviewing/reading anything that you yourself did not read / test in the target environments properly. If all it takes is an LLM prompt, I c…

For those curious what CL is (I had to look it up myself): it means change list[1]. Guessed from context that it was like "diff in PR/MR" or something but hadn't seen this acronym before

[1]: https://stackoverflow.com/a/27520705/2805120

Re: Reviews have become expensive, rewrites have become cheap

#68

Post by verall »

nomel wrote: post #40
usernametaken29 wrote: post #37

> The shortest path for the model is to implement it completely Have you worked with LLMs??????????? “I disabled the test so it’s not run so now all the tests pass” is not a hypothetical it’s pretty common. LLMs frequently do shortcut learning. The reason why reviews are expensive is because you still need to do all the steps in order to understand if a shortcut is justified.

> “I disabled the test so it’s not run so now all the tests pass” Also: "I implemented it this terrible way because of precedence in the codebase...that I just wrote" "I avoided implementing this correctly because of migration concern for existing installations of this code I'm writing right now" "I deferred this critical feature for the future, so we can deploy quicker" or, my favorite, "I hand rolled an buggy http…

> "I avoided implementing this correctly because of migration concern for existing installations of this code I'm writing right now"

This one grinds my gears so bad, probably 1/4 of my job at this point is telling an LLM (either in my own editor or in review comments for someone's MR) "how about we do this right _before_ merging it, eh?".

Or:

foo = abc

if foo != None:

    ...
In my experience giving it "rules" not to do this does nothing, but a separate pass (could be by a different model but really just fresh context is enough) does okay

Re: Reviews have become expensive, rewrites have become cheap

#69

Post by verall »

jamesknelson wrote: post #39
latentsea wrote: post #29

Earlier quoted context omitted.

That's not just you imagining things, that's the world around you changing. That's real and it matters.

So reading this, I know it’s probably a human posing as an LLM. But the problem with formats like this is that I don’t actually know. If a human said this to me in real life, and I laughed, it would probably help build a connection with that person, as it’s signaling that we have in common an unusually strong grasp of the patterns in LLM output (unusually strong at least in comparison to the general population). But…

In context it's funny, and that's what counts.

Re: Reviews have become expensive, rewrites have become cheap

#70

Post by cadamsdotcom »

This is very handy.

Say you discover in review that your agent wrote code before tests. A file that was added, no corresponding test file.

Tell the agent to read in the file, delete it (or roll back), then reimplement using red/green TDD.

Claude can test-drive an identical reimplementation but this time by writing red tests and making them green. Now you have coverage that you KNOW covers the code because you watched those tests fail before they passed.

Going further I added a note to my `/review` command so the agent looks for this mistake for me in its self-review. If it finds uncovered code it makes a plan which includes the delete-reimplement trick. Now all I do is approve that plan - no hunting needed, no explaining, no repeating myself.

You could even put this in a Ralph Wiggum loop and give yourself super high quality tests by going file by file.

Post reply on HN