Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pitfalls of premature closure with LLM assisted coding

shayon.dev

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Pitfalls of premature closure with LLM assisted coding

#41

Post by daveguy »

stillpointlab wrote: post #37
0x457 wrote: post #35

Earlier quoted context omitted.

> If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. I thought so too, but sometimes I had better results with one sentence prompt (+README.md) where it delivered the exact thing I wanted. I also had a very detailed prompt with multiple subtasks, all were very detailed +README.md +AGENTS.md and results were very poor.

This is true in my experience but it doesn't go against my larger point. Choosing the "goldilocks" context is a bit of an art, not too big not too small. It reminds of a famous witty quote [1]: “I apologize for such a long letter - I didn't have time to write a short one.” If you send too much info at once it does seem to confuse the agent, just like if you ask it to do too much all at once. That is yet another prope…

Also, regardless of the prompt they only get ~80% accuracy on coding benchmarks. So even with the absolute perfect prompt incantation, you can expect it to fail 1 out of 5 times.

Re: Pitfalls of premature closure with LLM assisted coding

#42

Post by bluefirebrand »

deadbabe wrote: post #36
stillpointlab wrote: post #28

My experience is that AIs amplify what you put in them. If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. However, if you spend a good amount of time describing the problem, carefully construct a context that includes examples, documentation and relevant files and then review the code with care - you can get some very good cod…

The point of LLMs is to not spend a lot of effort. Zero-shot prompts is the ideal we have to work toward. There comes a point where you have to do so much work just to get a good output, that LLMs cease to be more productive than just writing something out yourself. If it cannot give you a good output with very little prompting, it’s a sign your problem probably isn’t something well known and it probably needs a huma…

> There comes a point where you have to do so much work just to get a good output, that LLMs cease to be more productive than just writing something out yourself.

I think this gets to the core of the problem with LLM workflows and why there are so many disagreements about effectiveness

Maybe I overestimate my skills or underestimate how long things would take, but I am constantly feeling like when I try to use AI it takes more time, not less

My suspicion is that if you could create a second version of me, give one copy of them an LLM and one copy solves the problem normally, this would be the case

But many people love these tools and feel more productive, so what gives? The problem it is impossible to really measure because we don't have convenient parallel universe clones to test against. It's all just vibes and made up numbers

Re: Pitfalls of premature closure with LLM assisted coding

#43

Post by DevDesmond »

bluefirebrand wrote: post #39
th0ma5 wrote: post #34

Earlier quoted context omitted.

This has been the problem with higher level natural language programming for years. I really wonder what people are doing if they don't see this core issue that precludes their use.

It makes me wonder if some people writing code just cannot think in terms of code? I imagine it is very slow if you always have to think in a human language and then translate each step into programming language When people describe being in flow state, I think what is happening is they are more or less thinking directly in the programming language they are writing. No translation step, just writing code LLM workflow…

CSS or Tailwind has always been a tough one for me. I have banks of flashcards to help me remember stuff, (align-items, justify-content, grid-template-columns, etc.). Even with all that effort and many projects of practice, though, I've never had things click.

LLM assisted programming, however? – instant flow state. Instead of thinking in code I can think in product, and I can go straight from a pencil sketch to describing it as a set of constraints, and then say, "make sure it's ARIA compliant and responsive", and 95% of the work is done.

I feel similarly about configuration heavy files like Nginx or something. I really don't care to spend my time reading documentation, I'd rather copy paste the entire docs into the context window and then describe what I want in English.

Also good for SQL. And library code for a one off tool or API. And Bash scripting.

Re: Pitfalls of premature closure with LLM assisted coding

#44

Post by daveguy »

lubujackson wrote: post #17

My experience with LLMs currently is that they can handle any level of abstraction and focus, but you have discern the "layer" to isolate and resolve. The next improvement may be something like "abstraction isolation" but for now I can vibe code a new feature which will produce something mediocre. Then I ask "is that the cleanest approach?" and it will improve it. Then I might ask "is this performant?" Or "does this…

Doesn't being able to only handle a single layer of abstraction as long as it is well defined and isolated essentially mean it can't handle abstraction at all?

Aren't the layers the point of abstraction?

Re: Pitfalls of premature closure with LLM assisted coding

#45

Post by 0x457 »

stillpointlab wrote: post #37
0x457 wrote: post #35

Earlier quoted context omitted.

> If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. I thought so too, but sometimes I had better results with one sentence prompt (+README.md) where it delivered the exact thing I wanted. I also had a very detailed prompt with multiple subtasks, all were very detailed +README.md +AGENTS.md and results were very poor.

This is true in my experience but it doesn't go against my larger point. Choosing the "goldilocks" context is a bit of an art, not too big not too small. It reminds of a famous witty quote [1]: “I apologize for such a long letter - I didn't have time to write a short one.” If you send too much info at once it does seem to confuse the agent, just like if you ask it to do too much all at once. That is yet another prope…

That's why I make initial context (e.g. AGENTS.md) is about how to bootstrap context for a current task/project. Now, my prompts only need to be good enough to hint how to read the graph correctly.

Re: Pitfalls of premature closure with LLM assisted coding

#46

Post by bluefirebrand »

DevDesmond wrote: post #43
bluefirebrand wrote: post #39

Earlier quoted context omitted.

It makes me wonder if some people writing code just cannot think in terms of code? I imagine it is very slow if you always have to think in a human language and then translate each step into programming language When people describe being in flow state, I think what is happening is they are more or less thinking directly in the programming language they are writing. No translation step, just writing code LLM workflow…

CSS or Tailwind has always been a tough one for me. I have banks of flashcards to help me remember stuff, (align-items, justify-content, grid-template-columns, etc.). Even with all that effort and many projects of practice, though, I've never had things click. LLM assisted programming, however? – instant flow state. Instead of thinking in code I can think in product, and I can go straight from a pencil sketch to desc…

> Instead of thinking in code I can think in product

I think you are talking about something very different than I am when you say flow state

Re: Pitfalls of premature closure with LLM assisted coding

#47

Post by lazy_moderator1 »

danielbln wrote: post #26
lazy_moderator1 wrote: post #23

Earlier quoted context omitted.

did it detect n+1 in the first one, race condition in the second one and memory leak in the third one?

It did, yeah.

Could it be that it just found this article and suggested issues based on that? Because the issues are somewhat arbitrary. One could make a case for a several different issues for any of the snippets and yet the model chose the exact ones mentioned by the article.

Re: Pitfalls of premature closure with LLM assisted coding

#48

Post by marxism »

danielbln wrote: post #2

I put the examples he gave into Claude 4(Sonnet) purely asking to eval the code, it pointed out every single issue about the code snippets (N+1 Query, race condition, memory leak). The article doesn;t mention which model was used, or how exactly it was used, or in which environment/IDE it was used. The rest of the advice in there is sound, but without more specifics I don't know how actionable the section "The spectr…

As you pointed out the examples in the blog post are not an LLM failure. The real failure is asking too little.

Engineers think "the LLM can handle the simple code change, but if I ask for too much it'll fall over." Wrong. Modern LLMs can easily handle a 50-line function plus 50 lines of detailed comments explaining assumptions, performance implications, and what changes would invalidate this approach.

But most engineers are either asking for solutions without enough context or failing to ask the LLM to document its assumptions.

Then they're shocked when they have to reverse engineer out that the code assumes 100 users when they have 100k, or why it's doing individual API calls when they needed batch processing.

Most engineers have never seen good comments, so they don't know they can ask LLMs to write them.

The default LLM comment is just English pseudo-code: "this function takes a user ID and sends them a notification." Completely useless. But that's because most engineers have never experienced comments that explain trade-offs, performance implications, or future system evolution.

Writing clear technical explanations is genuinely difficult. Almost no one does it well. So when you ask an LLM for "comments," you get the same terrible pattern you've seen everywhere else.

But you can literally ask for explanations of assumptions, performance characteristics, and scenarios where this approach would break. The LLM handles it perfectly. You just have to know that's even possible. Makes the code review so much easier.

Most engineers don't, because they've never seen it done.

[1] https://peoplesgrocers.com/en/writing/asking-llms-the-right-...

Re: Pitfalls of premature closure with LLM assisted coding

#49

Post by owebmaster »

marxism wrote: post #48
danielbln wrote: post #2

I put the examples he gave into Claude 4(Sonnet) purely asking to eval the code, it pointed out every single issue about the code snippets (N+1 Query, race condition, memory leak). The article doesn;t mention which model was used, or how exactly it was used, or in which environment/IDE it was used. The rest of the advice in there is sound, but without more specifics I don't know how actionable the section "The spectr…

As you pointed out the examples in the blog post are not an LLM failure. The real failure is asking too little. Engineers think "the LLM can handle the simple code change, but if I ask for too much it'll fall over." Wrong. Modern LLMs can easily handle a 50-line function plus 50 lines of detailed comments explaining assumptions, performance implications, and what changes would invalidate this approach. But most engin…

This seems to be the default for Gemini 2.5 Pro now

Re: Pitfalls of premature closure with LLM assisted coding

#50

Post by gaeb69 »

0x457 wrote: post #35
stillpointlab wrote: post #28

My experience is that AIs amplify what you put in them. If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. However, if you spend a good amount of time describing the problem, carefully construct a context that includes examples, documentation and relevant files and then review the code with care - you can get some very good cod…

> If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. I thought so too, but sometimes I had better results with one sentence prompt (+README.md) where it delivered the exact thing I wanted. I also had a very detailed prompt with multiple subtasks, all were very detailed +README.md +AGENTS.md and results were very poor.

Reasoning model prompting is relevant here. OpenAI in their docs state that giving detailed, step-by-step instructions often hinders reasoning models. It's better to clearly define the outcome along with what you're working with. Then let the mode interpolate.
Post reply on HN