261–270 of 501 posts
AI by design can only repeat and recombine past material. Therefore actual invention is out.
Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
Earlier quoted context omitted.
> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?
Convenient for whom and what...? There is nothing tangible to gain from you believing or not believing that someone else does (or does not) get a productivity boost from AI. This is not a religion and it's not crypto. The AI users' net worth is not tied to another ones use of or stance on AI (if anything, it's the opposite). More generally, the phenomenon this is quite simply explained and nothing surprising: New thi…
We're in a hype cycle, and it means we should be extra critical when evaluating the tech so we don't get taken in by exaggerated claims.
Earlier quoted context omitted.
You can see this analysis in the factor analysis of "Below-average use of AI tools" (C.2.7) in the paper [1], which we mark as an unclear effect. TLDR: over the first 8 issues, developers do not appear to get majorly less slowed down. [1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
Thanks, that's great! But: if all developers did 136 AI-assisted issues, why only analyze excluding the 1st 8, rather than, say, the first 68 (half)?
Earlier quoted context omitted.
Or, one person in 16 has a particular personality, inclined to LLM dependence.
Invoking personality is to the behavioral science as invoking God is to the natural sciences. One can explain anything by appealing to personality, and as such it explains nothing. Psychologists have been trying to make sense of personality for over a century without much success (the best efforts so far have been a five factor model [Big 5] which has ultimately pretty minor predictive value), which is why most behav…
Earlier quoted context omitted.
Honestly the hype cycle feels very like crypto, and just like crypto prominent vcs have a lot of money riding on the outcome.
Of course, lot's of hype, but my point is that the reason why is very different and it matters: As an early bc adopter making your believe in bc is super important to my net worth (and you not believing in bc makes me look like an idiot and lose a lot of money). In contrast, what do I care if you believe in code generation AI? If you do, you are probably driving up pricing. I mean, I am sure that there are people tha…
Earlier quoted context omitted.
Sorry, that’s not my take. I didn’t think these tools were useful until the latest set of models, that is, they crossed the threshold of usefulness to me. Even then though, “technology gets better over time” shouldn’t be surprising, as it’s pretty common.
Do you really see a massive jump? For context, I've been using AI, a mix of OpenAi + Claude, mainly for bashing out quick React stuff. For over a year now. Anything else it's generally rubbish and slower than working without. Though I still use it to rubber duck, so I'm still seeing the level of quality for backend. I'd say they're only marginally better today than they were even 2 years ago. Every time a new model c…
An LLM that can test the code it is writing and then iterate to fix the bugs turns out to be a huge step forward from LLMs that just write code without trying to then exercise it.
Earlier quoted context omitted.
> That's not the argument being made though, which is that it does "work" now and implying that actually it didn't quite work before Right. > except that that is the same thing the same people say for every model release, I did not say that, no. I am sure you can find someone who is in a Groundhog Day about this, but it’s just simpler than that: as tools improve, more people find them useful than before. You’re not t…
> You’re not talking to the same people, you are talking to new people each time who now have had their threshold crossed. no, it's the same names, again and again
That sounds like a claim you could back up with a little bit of time spent using Hacker News search or similar.
(I might try to get a tool like o3 to run those searches for me.)
Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
The short version is that devs want to give instructions instead of ask for what outcome they want. When it doesn’t follow the instructions, they double down by being more precise, the worst thing you can do. When non devs don’t get what they want, they add more detail to the description of the desired outcome.
Once you get past the control problem, then you have a second set of issues for devs where the things that should be easy or hard don’t necessarily map to their mental model of what is easy or hard, so they get frustrated with the LLM when it can’t do something “easy.”
Lastly, devs keep a shit load of context in their head - the project, what they are working on, application state, etc. and they need to do that for LLMs too, but you have to repeat themselves often and “be” the external memory for the LLM. Most devs I have taught hate that, they actually would rather have it the other way around where they get help with context and state but want to instruct the computer on their own.
Interestingly, the best AI assisted devs have often moved to management/solution architecture, and they find the AI code tools brought back some of the love of coding. I have a hypothesis they’re wired a bit differently and their role with AI tools is actually closer to management than it is development in a number of ways.