Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding is the new bottleneck

geoffreylitt.com

Post reply on HN
Print view View original

41–50 of 259 posts

Re: Understanding is the new bottleneck

#41

Post by dylan604 »

alecbz wrote: post #26
dylan604 wrote: post #20

Earlier quoted context omitted.

A PR with a minimal title and empty description should be refused at submission. If the human is so disinterested that they're using LLM generated code and then can't explain the purpose, that human should be prevent from making the PR. Working as a solo dev, it is very easy to be lazy like that, and I'm as guilty as anyone. Working in teams with actual reviews should absolutely have much more strict policies of what…

I mean, the “meta” right now is trying to automate as much of the software engineering flow as possible, including writing PR descriptions. They’re shooting for LLMs being able to one-shot PRs or need minimal oversight. But yeah, in practice LLMs are not there IME.

This is the biggest issue I have with current state of affairs. It's not there yet. Because of that, extra work is needed to get them to work that otherwise would not need to be spent. Everyone is shouting from the roof tops about how great things are while suppressing these types of issues.

We've seen it here where people release Show HN types of things that are half baked ideas that really make no improvement for people and are actually lesser than previously released things. Yet they are expecting people to be amazed. Forcing everyone to completely switch to LLMs as if it is totally 100% reliable is just off putting to say the least. It takes discussing things with people honestly looking at the situation to have any semblance of thinking you're not the insane one for pushing back

Re: Understanding is the new bottleneck

#42

Post by nsingh2 »

jollyllama wrote: post #29
vjvjvjvjghv wrote: post #8

Earlier quoted context omitted.

I don’t know. From my experience I get pretty good descriptions of PRs if I ask the right questions and provide some context.

> provide some context That's pretty much what a PR description is.

Work doesn't start with a PR description though. I'm assuming most people that are using LLMs start with some sort of document (plan, spec, intent, etc) which captures intent.

I guess you could also use all the session rollouts saved to disk that were related to that task, and distill them somehow.

Re: Understanding is the new bottleneck

#43

Post by alecbz »

baq wrote: post #33
jollyllama wrote: post #29

Earlier quoted context omitted.

> provide some context That's pretty much what a PR description is.

The difference is an LLM can convert a stream of consciousness into well-formed prose for approximately free; I assume ‘provide some context’ means ‘brain dump’ in the OP

This has unfortunately not been my experience at all. Often LLMs miss or get wrong subtle details when I don't do the pre-work to organize my thoughts well ahead of time (at which point it's unclear how much value they're providing).

Re: Understanding is the new bottleneck

#44

Post by threethirtytwo »

This is a temporary bottleneck. AI is moving so fast that this will change. Wait six months and this article is no longer relevant.

About a year ago most people were still typing code. Having an agent do ALL code was crazy.

Within a year or two years at most, a lot of people will stop trying to understand code. The onus will shift to testing and QAing.

I know this is hard to hear but that’s the trendline. That’s where all of this is converging. Everyone’s to busy trying to lock themselves down as an expert of the new “paradigm” but it’s all moving so fast that the paradigm now won’t be the paradigm of tomorrow.

Re: Understanding is the new bottleneck

#45

Post by sajithdilshan »

Understanding has always been the bottleneck. Sometimes AI helps with it like explaining things pretty well with diagrams. However, in general I agree that more code is being generated per developer and it's difficult to keep up with the phase of new changes and understand it.

Re: Understanding is the new bottleneck

#46

Post by 3abiton »

alecbz wrote: post #6

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation. Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

I hate to be pedantic but you can finetune a skill to shape the PR message the way you like it. That being said, I did have exactly this issue you mentioned, but the defualt output can always be tuned.

Re: Understanding is the new bottleneck

#47

Post by skydhash »

dylan604 wrote: post #20
alecbz wrote: post #16

Earlier quoted context omitted.

LLM-generated (or at least LLM-assisted), but a human's still the one submitting the PR for review.

A PR with a minimal title and empty description should be refused at submission. If the human is so disinterested that they're using LLM generated code and then can't explain the purpose, that human should be prevent from making the PR. Working as a solo dev, it is very easy to be lazy like that, and I'm as guilty as anyone. Working in teams with actual reviews should absolutely have much more strict policies of what…

> A PR with a minimal title and empty description should be refused at submission

Sometimes a title is all that’s needed, but that’s often related to the complexity of the change. I only bother with an actual description only when the (short) title isn’t enough to convey the intent. But it’s very rare to go past one paragraph. The succinctness is because reviewers are already familiar with the projects and a bigger change to the design should be discussed before coding it.

Re: Understanding is the new bottleneck

#49

Post by csallen »

3abiton wrote: post #46
alecbz wrote: post #6

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation. Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

I hate to be pedantic but you can finetune a skill to shape the PR message the way you like it. That being said, I did have exactly this issue you mentioned, but the defualt output can always be tuned.

I don't know why you got downvoted, but I find myself wanting to say some version of what you just said over and over again. People write extremely lazy, straightforward prompts and expect the LLM's intelligence to take care of all of it. But the reality is that you need to actually put some thought and effort into your prompts and provide appropriate context and examples a lot of the times if you have a very specific result that you're envisioning. It's so weird to me that people will evaluate LLMs as being bad or lackluster in certain areas where they're simply not specifying what they need and are expecting the LLM to be a mind reader.

I'm not saying that the GP is necessarily doing this. But having repeatedly had plenty of success myself in getting LLMs to write things the way that I want, with a little bit of prompting, it seems likely

Re: Understanding is the new bottleneck

#50

Post by iainctduncan »

euroderf wrote: post #15
iainctduncan wrote: post #3

I am so dying to read more about the new/current/real bottleneck! Where is the bottleneck? WHERE?? Tell me! No evidence needed, just lay it on, man to man, thought-leader to thought-leader!

> thought-leader to thought-leader! This is my new chat-up line at networking events.

:-)
Post reply on HN