Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pitfalls of premature closure with LLM assisted coding

shayon.dev

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Pitfalls of premature closure with LLM assisted coding

#31

Post by drewnick »

soulofmischief wrote: post #14
shayonj wrote: post #4

Earlier quoted context omitted.

My experience hasn't changed between models, given the core issue mentioned in the article. Primarily I have used Gemini and Claude 3.x and 4. Some GPT 4.1 here and there. All via Cursor, some internal tools and Tines Workbench

My experience changes just throughout the day on the same model, it seems pretty clear that during peak hours (lately most of the daytime) Anthropic is degrading their models in order to meet demand. Claude becomes a confident idiot and the difference is quite noticeable.

I too have noticed variability and it's impossible to know for sure but late one Friday or Saturday night (PST) it seemed to be brilliant, several iterations in a row. Some of my best output has been in very short windows.

Re: Pitfalls of premature closure with LLM assisted coding

#32

Post by bluefirebrand »

nonethewiser wrote: post #25
cryptonym wrote: post #21

Earlier quoted context omitted.

Sounds like coding with extra steps.

What is the extra step? You have to do the upfront legwork either way.

In my experience when you have a problem that is small enough for an LLM to solve, you could just write the code directly. You don't have to produce a detailed spec first

If the LLM needs a detailed spec to solve the same problem then you're doing unnecessary work to produce the spec for the LLM first

Re: Pitfalls of premature closure with LLM assisted coding

#33

Post by soulofmischief »

genewitch wrote: post #24
soulofmischief wrote: post #14

Earlier quoted context omitted.

My experience changes just throughout the day on the same model, it seems pretty clear that during peak hours (lately most of the daytime) Anthropic is degrading their models in order to meet demand. Claude becomes a confident idiot and the difference is quite noticeable.

this is on paid plans?

This is through providers such as Cursor, but the consistency of this experience has put me off from directly subscribing to Anthropic since I'm already subscribed up to my eyeballs in various AI services.

Last I'd checked, Anthropic would not admit that they were degrading models for obvious scummy business reasons, but they are probably quantizing them, reducing beam search, lowering precision/sampling), etc. because the model goes from being superpowered to completely unusable, constantly dropping code and mangling files, getting caught in loops, doing the weirdest detours, and sometimes completely ignoring my instructions from just one message prior.

t first I wondered if Cursor was mishandling the context, and while they indeed aren't doing the best with context stuffing, the rest of the issues are not context-related.

Re: Pitfalls of premature closure with LLM assisted coding

#34

Post by th0ma5 »

bluefirebrand wrote: post #32
nonethewiser wrote: post #25

Earlier quoted context omitted.

What is the extra step? You have to do the upfront legwork either way.

In my experience when you have a problem that is small enough for an LLM to solve, you could just write the code directly. You don't have to produce a detailed spec first If the LLM needs a detailed spec to solve the same problem then you're doing unnecessary work to produce the spec for the LLM first

This has been the problem with higher level natural language programming for years. I really wonder what people are doing if they don't see this core issue that precludes their use.

Re: Pitfalls of premature closure with LLM assisted coding

#35

Post by 0x457 »

stillpointlab wrote: post #28

My experience is that AIs amplify what you put in them. If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. However, if you spend a good amount of time describing the problem, carefully construct a context that includes examples, documentation and relevant files and then review the code with care - you can get some very good cod…

> If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster.

I thought so too, but sometimes I had better results with one sentence prompt (+README.md) where it delivered the exact thing I wanted. I also had a very detailed prompt with multiple subtasks, all were very detailed +README.md +AGENTS.md and results were very poor.

Re: Pitfalls of premature closure with LLM assisted coding

#36

Post by deadbabe »

stillpointlab wrote: post #28

My experience is that AIs amplify what you put in them. If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. However, if you spend a good amount of time describing the problem, carefully construct a context that includes examples, documentation and relevant files and then review the code with care - you can get some very good cod…

The point of LLMs is to not spend a lot of effort. Zero-shot prompts is the ideal we have to work toward. There comes a point where you have to do so much work just to get a good output, that LLMs cease to be more productive than just writing something out yourself.

If it cannot give you a good output with very little prompting, it’s a sign your problem probably isn’t something well known and it probably needs a human touch.

Re: Pitfalls of premature closure with LLM assisted coding

#37

Post by stillpointlab »

0x457 wrote: post #35
stillpointlab wrote: post #28

My experience is that AIs amplify what you put in them. If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. However, if you spend a good amount of time describing the problem, carefully construct a context that includes examples, documentation and relevant files and then review the code with care - you can get some very good cod…

> If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. I thought so too, but sometimes I had better results with one sentence prompt (+README.md) where it delivered the exact thing I wanted. I also had a very detailed prompt with multiple subtasks, all were very detailed +README.md +AGENTS.md and results were very poor.

This is true in my experience but it doesn't go against my larger point. Choosing the "goldilocks" context is a bit of an art, not too big not too small. It reminds of a famous witty quote [1]: “I apologize for such a long letter - I didn't have time to write a short one.”

If you send too much info at once it does seem to confuse the agent, just like if you ask it to do too much all at once. That is yet another property it shares with a junior engineer. It is easy to overwhelm a new contributor to a project with too much information, especially if it isn't strictly relevant.

Re: Pitfalls of premature closure with LLM assisted coding

#38

Post by stillpointlab »

deadbabe wrote: post #36
stillpointlab wrote: post #28

My experience is that AIs amplify what you put in them. If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. However, if you spend a good amount of time describing the problem, carefully construct a context that includes examples, documentation and relevant files and then review the code with care - you can get some very good cod…

The point of LLMs is to not spend a lot of effort. Zero-shot prompts is the ideal we have to work toward. There comes a point where you have to do so much work just to get a good output, that LLMs cease to be more productive than just writing something out yourself. If it cannot give you a good output with very little prompting, it’s a sign your problem probably isn’t something well known and it probably needs a huma…

I could not disagree more. I don't think you are wrong, I just choose a different approach.

There seem to be multiple approaches to working with LLMs. My own personal experience has been that carefully explaining my request, providing specific and highly relevant context (and avoiding irrelevant and distracting context) has lead to significant productivity on my side. That is, it may take me 15 minutes to prepare a really good prompt but the output can save me hours of work. Conversely, if I fire of a bunch of low-effort prompts I get poor results and I end up spending a lot of time in back-and-forth with the LLM and a lot of time fixing up its output.

Re: Pitfalls of premature closure with LLM assisted coding

#39

Post by bluefirebrand »

th0ma5 wrote: post #34
bluefirebrand wrote: post #32

Earlier quoted context omitted.

In my experience when you have a problem that is small enough for an LLM to solve, you could just write the code directly. You don't have to produce a detailed spec first If the LLM needs a detailed spec to solve the same problem then you're doing unnecessary work to produce the spec for the LLM first

This has been the problem with higher level natural language programming for years. I really wonder what people are doing if they don't see this core issue that precludes their use.

It makes me wonder if some people writing code just cannot think in terms of code?

I imagine it is very slow if you always have to think in a human language and then translate each step into programming language

When people describe being in flow state, I think what is happening is they are more or less thinking directly in the programming language they are writing. No translation step, just writing code

LLM workflows completely remove the ability to achieve that imo

Re: Pitfalls of premature closure with LLM assisted coding

#40

Post by deadbabe »

stillpointlab wrote: post #38
deadbabe wrote: post #36

Earlier quoted context omitted.

The point of LLMs is to not spend a lot of effort. Zero-shot prompts is the ideal we have to work toward. There comes a point where you have to do so much work just to get a good output, that LLMs cease to be more productive than just writing something out yourself. If it cannot give you a good output with very little prompting, it’s a sign your problem probably isn’t something well known and it probably needs a huma…

I could not disagree more. I don't think you are wrong, I just choose a different approach. There seem to be multiple approaches to working with LLMs. My own personal experience has been that carefully explaining my request, providing specific and highly relevant context (and avoiding irrelevant and distracting context) has lead to significant productivity on my side. That is, it may take me 15 minutes to prepare a r…

If I think carefully about a problem for 15 minutes I generally can also save hours of work.
Post reply on HN