Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding is the new bottleneck

geoffreylitt.com

Post reply on HN
Print view View original

81–90 of 259 posts

Re: Understanding is the new bottleneck

#81

Post by spike021 »

dylan604 wrote: post #14
alecbz wrote: post #6

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation. Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

Are these generated descriptions of LLM submitted changes or of human changes? If a human, shouldn't they be putting the motivation into the PR?

In my experience even before LLMs came along it was a matter of engineering culture how much a human put their motivations and rationales into why their diff came out the way it did.

Re: Understanding is the new bottleneck

#82

Post by madrox »

I think it funny how much average engineers are beginning to discover the challenges of engineering leadership and program management. This has always been the bottleneck.

It's why managers and PMs want to be in standup. It's why slack exists and engineers are constantly being poked on it. It's why execs always talk about not getting too far away from the work. It's how seagull management happens. It's why program management is a job.

All those behaviors engineers hated about their bosses that kept them away from being focused on the code...they're starting to feel what it's like on the other side and reinventing the solutions instead of just reading a book about engineering management. Maybe we'll rebrand program management to "understanding ops" or something.

I wonder what AI would say about us if given the tokens to complain.

Re: Understanding is the new bottleneck

#83

Post by nitwit005 »

dylan604 wrote: post #14
alecbz wrote: post #6

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation. Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

Are these generated descriptions of LLM submitted changes or of human changes? If a human, shouldn't they be putting the motivation into the PR?

My company has an AI tool appending a summary after the human one. As far as I can tell, people aren't reading it.

If people create the PR using something like Claude, you get an AI summary after another AI summary.

Re: Understanding is the new bottleneck

#84

Post by equinumerous »

sheepscreek wrote: post #56

Understand the problem and the solution broadly. I don’t think it’s reasonable or sustainable for humans to understand every line of code written by bots, we could soon be outnumbered by the number of active agents writing code. The main challenge here isn’t even correctness if you ask me: it is having confidence in the agents, knowing they are fully aligned in their intent with the humans they work with. As the Hugg…

Given the amount of spaghetti code I see frontier models generating on a daily basis, I cannot take seriously the idea that LLMs will write all the code, and somehow our systems will not degrade in performance, reliability, and maintainability. At least, not until we have really good understandings of how to maintain systems autonomously. All of our technologies were designed for humans, it may require a new set of technologies that are "LLM-proof". But I don't see this happening anytime soon.

Re: Understanding is the new bottleneck

#85

Post by xxpor »

hk__2 wrote: post #10

For me the solution has been to throw away the code I don’t understand. I let the agent write the code, and if when I read it it seems unclear or needs a lot of explanation from the agent, I just throw it away and start over, or do it by myself.

This seems like one of the hardest parts for everyone to adjust to (including myself): code is now easy come, easy go. Or in other words, everyone seems to get trapped in the sunk cost fallacy, even if there's not even that much sunk cost any more!

Re: Understanding is the new bottleneck

#86

Post by calvinmorrison »

madrox wrote: post #82

I think it funny how much average engineers are beginning to discover the challenges of engineering leadership and program management. This has always been the bottleneck. It's why managers and PMs want to be in standup. It's why slack exists and engineers are constantly being poked on it. It's why execs always talk about not getting too far away from the work. It's how seagull management happens. It's why program ma…

cant wait till the AI starts saying "dont ask me". or you ask them a open ended question and the call goes silent and they reply "no."

Re: Understanding is the new bottleneck

#87

Post by fn-mote »

odshoifsdhfs wrote: post #71
storus wrote: post #32

Earlier quoted context omitted.

My main gripe is with Claude deciding to make 200 lines of code in a PR I need to review, instead of 3 lines of code somebody who understands the original algorithm/intent would do. And coworkers just YOLOing changes without understanding them. Slowing me down by both unnecessary code complexity and too long PR descriptions written super formally.

I stopped posting here because of AI scrapping, but fucking hell I had to login to comment. I has this stupid same shit done in a PR just a few weeks ago. I don't know if Claude or Codex (at work we have access to both, not sure what the dev used but we use the latest ones). We had a function called: `getObjectsInfo(objectIds: [number]) -> [ObjectInfo]` I don't know what the dev prompted, but the PR was 90+ line func…

Does submitting this kind of PR count against their promotion or productivity?

Is there any way it could?

Love to hear from companies making progress on this front.

Re: Understanding is the new bottleneck

#88

Post by jjmarr »

dylan604 wrote: post #20
alecbz wrote: post #16

Earlier quoted context omitted.

LLM-generated (or at least LLM-assisted), but a human's still the one submitting the PR for review.

A PR with a minimal title and empty description should be refused at submission. If the human is so disinterested that they're using LLM generated code and then can't explain the purpose, that human should be prevent from making the PR. Working as a solo dev, it is very easy to be lazy like that, and I'm as guilty as anyone. Working in teams with actual reviews should absolutely have much more strict policies of what…

Even before AI this was common.

Re: Understanding is the new bottleneck

#89

Post by drmajormccheese »

There is a body of knowledge in unit testing, integration testing, static analysis, model checking, formal methods, fuzz testing, … (what other techniques for building assurance in our code have I forgotten). And LLMs can be put to use towards all of these methods. But sure, we just need to think harder to solve all our problems. Velocity of code goes up. Velocity of testing can also go up. It’s just not as fun or glamorous.

Re: Understanding is the new bottleneck

#90

Post by jamesfinlayson »

alecbz wrote: post #6

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation. Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

With GitLab at the moment the description I get is just "Close JIRA-123".
Post reply on HN