they estimate as if human will build it
yeah this is my suspicion too
Why does it look like LLMs consistently overestimate implementation time?
11–18 of 18 posts
Re: Why does it look like LLMs consistently overestimate implementation time?
#12[flagged]
Re: Why does it look like LLMs consistently overestimate implementation time?
#13maybe they're rewarded for being under their estimated implementation time in their training. they could learn a similar behavior (to safely underestimate) in other contexts and that could've spilled over.
you can blame everything on wired quirks in the training (claude overusing the words "true" and "genuine" when their not needed, AIs using em-dashes because the pretrain has a ton of them)
Re: Why does it look like LLMs consistently overestimate implementation time?
#14[flagged]
Re: Why does it look like LLMs consistently overestimate implementation time?
#15If what you say is reality, I would keep that.
We have a tendency to give overly optimistic estimates, best case scenario, no other tasks, no roadblocks...
Whenever asked for an estimate, think how long it would take you to make it and multiply by 5.
Re: Why does it look like LLMs consistently overestimate implementation time?
#16As a workarond, I have noticed that a better estimate of time is number of sessions required to fully implement, review, and test a given feature. You can ask the LLM to help you estimate how complex a feature will be during the planning phase. The number of tokens can easilly be translated to sessions, asumming that you normally aim to keep a session below 60% of the context limit. As a rule of thumb, a simple feature can be fully implemented, tested and reviewed in a single session. Whereas a complex feature might take multiple sessions. Then depending on your workload and attention span, you can have a decent idea of how many full context sessions you can actually plan and review in a single day. Of course, some features can happen in parallel but in general the bottleneck is your ability to understand the context that goes into each session.
Re: Why does it look like LLMs consistently overestimate implementation time?
#17[deleted]
Re: Why does it look like LLMs consistently overestimate implementation time?
#18I think they overestimate less than it looks.
The actual coding part compressed dramatically.
The unpredictable part is still everything around the code.