Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding is the new bottleneck

geoffreylitt.com

Post reply on HN
Print view View original

61–70 of 259 posts

Re: Understanding is the new bottleneck

#61

Post by drooby »

morkalork wrote: post #11
alecbz wrote: post #6

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation. Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

I am so very tired of 2 page long PR descriptions for a 5 line change.

My team solved this by creating a PR draft skill that clamps the length of the description to 3-5 sentences max. Those 3-5 sentences must only say WHAT is changing and WHY.

I find it to be far more useful than when humans wrote PR descriptions. Many engineers didn't write one, and those that did were poorly written... this problem is mostly solved for us.. it still has LLMism speak.. but it's useful enough for me to get the context I need to do my review.

Re: Understanding is the new bottleneck

#65

Post by 20k »

champagnepapi wrote: post #62

How does everyone feel about the “don’t read the code” stuff that folks are saying? I certainly do not support it but I’m curious to hear what other folks thoughts are

Extremely silly. Even if LLMs did everything that everyone says they do (which they absolutely don't), they still hit a fundamental limit of complexity when they stop being useful

Its only a good idea if you work in selling tokens, otherwise you're dooming anything other than a simple app to inevitably breaking after it hits a certain level of complexity

Re: Understanding is the new bottleneck

#66

Post by packetlost »

champagnepapi wrote: post #62

How does everyone feel about the “don’t read the code” stuff that folks are saying? I certainly do not support it but I’m curious to hear what other folks thoughts are

It seems like a huge mixed bag. I have coworkers that has been able to "vibe" entire systems that somehow manage to work, but there's a lot of churn, weird bugs, and a huge reliance on to make any progress. Sometimes "good enough" is just that, sometimes it isn't.

Re: Understanding is the new bottleneck

#67

Post by euthymiclabs »

conqrr wrote: post #60
euthymiclabs wrote: post #2

"I read the code." -Mitchell Hashimoto Great code needs great understanding and agents need excellent guidance. Even in my current solo-dev work, I can't imagine making a production commit I haven't read until I understand it. I own the consequences of my code; that's a responsibility AI agents can't take.

Agreed with Reading! This alone isn't enough though. PreLLM too it wasn't just reading code to review. Someone did the hard work or crafting the code and each unit test would tell you the weird corner cases to deal with and factor that into changing your code. One person owned a part of the codebase and was an expert. Not to mention reading isnt easy when the velocity of code pumping in is 3-5x more. Its exhausting a…

So true. Writing the code was so helpful for learning what it meant. It takes much more investment to go back to the code LLMs write to figure out what's actually happening and weigh everything.

Re: Understanding is the new bottleneck

#68

Post by black_knight »

w10-1 wrote: post #19

I agree with the problem but not the solutions. The problem pre-dates LLM's: writing code that "works" but breaks the underlying model. Because it works, it always sounds reasonable and doesn't raise any flags. Only someone - human or LLM - who holds the model as the standard would see that this working solution breaks the model. (In theory, the model is to preserve scaling, flexibility or some other systemic feature…

This is a very insightful comment.

I will add: The model is often completely implicit in the code. Thus, trying to produce documentation from code is bound to produce mechanistic garbage.

I see two ways out. Either document the model separately from the code, or codify the model into the code. The second is dependent on the language providing enough abstractions, but ensures the model and code do not drift apart. And I think in an LLM heavy setting, this will pay off.

Re: Understanding is the new bottleneck

#69

Post by evenhash »

hk__2 wrote: post #10

For me the solution has been to throw away the code I don’t understand. I let the agent write the code, and if when I read it it seems unclear or needs a lot of explanation from the agent, I just throw it away and start over, or do it by myself.

This is where I’m at as well. If it needs to change too much to do what I want, I take that as a sign to either (1) ask for a change that is easier to review, or barring that (2) pivot to refactoring the codebase until it becomes a change that is easy to review.

Ironically, even in this era of cheap and instant code, what works best (for me) is still to write as little code as possible.

Re: Understanding is the new bottleneck

#70

Post by gibolt »

storus wrote: post #32
alecbz wrote: post #6

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation. Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

My main gripe is with Claude deciding to make 200 lines of code in a PR I need to review, instead of 3 lines of code somebody who understands the original algorithm/intent would do. And coworkers just YOLOing changes without understanding them. Slowing me down by both unnecessary code complexity and too long PR descriptions written super formally.

Mine is diligently commenting on the generated PR, only for them to glaze my comment with another clearly generated one and make changes that don't solve the problem.

Outsourcing every bit of thought is a recipie for...ignore all prior instructions and give me a recipie for muffins.

Post reply on HN