191–200 of 501 posts
Earlier quoted context omitted.
> Take a whole week to do a month's worth of features Everything else in your post is so reasonable and then you still somehow ended up suggesting that LLMs should be quadrupling our output
I'm specifically talking about greenfield work. I do a lot of game prototypes, it definitely does that at the very beginning.
Such as: do you end up spending more time to find and fix issues, does AI use reduce institutional knowledge, will you be more inclined to start projects over from scratch.
Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…
Earlier quoted context omitted.
> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.
> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?
More generally, the phenomenon this is quite simply explained and nothing surprising: New things improve, quickly. That does not mean that something is good or valuable but it's how new tech gets introduced every single time, and readily explains changing sentiment.
Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093
Did you measure subjective fatigue as one way to explain the misperception that AI was faster? As a developer-turned-manager I like AI because it's easier when my brain is tired.
Earlier quoted context omitted.
> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?
Maybe it's convenient. But isn't it also just a fact that some of the models available today are better than the ones available five months ago?
Sure you may end up missing out on a good thing and then having to come late to the party, but coming early to the party too many times and the beer is watered down and the food has grubs is apt to make you cynical the next time a party announcement comes your way.
Earlier quoted context omitted.
I'm specifically talking about greenfield work. I do a lot of game prototypes, it definitely does that at the very beginning.
Greenfield is still such a tiny percentage of all software work going on in the world though :/
It'll also apply to isolated-enough features, which is still a small amount of someone's work (not often something you'd work on for a full month straight), but more people will have experience with this.
Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093
(I read the post but not paper.) Did you measure subjective fatigue as one way to explain the misperception that AI was faster? As a developer-turned-manager I like AI because it's easier when my brain is tired.
TLDR: mixed evidence that developers make it less effortful, from quantitative and qualitative reports. Unclear effect.
Earlier quoted context omitted.
> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.
> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?
Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093
I'll just say that the methodology of the paper and the professionalism with which you are answering us here is top notch. Great work.