81–90 of 175 posts
I am pretty convinced that for most types of day to day work, any perceived improvements from the latest Claude models for example were total placebo. In blind tests and with normal tasks, people would probably have no idea if they're using Opus 4.5 or 4.6.
Haiku 4.5 is already so good it's ok for 80% (95%?) of dev tasks.
As they become more capable peoples commits will also become more ambitious. So I’d say fairly flat commit acceptance numbers make sense even in the context of improving LLMs
I think what happened with static image generation is happening with LLMs. Basically the tools around are becoming better, but all the AI improvements stall, the error rate stay the same (but external tools curate the results so it won't be noticeable if you don't run your own model), the accuracy is still slightly improving, but slower and slower, and never reach the 'perfect' point. Basically stablediffusion early…
Image quality has improved a lot in recent months thanks to better models. The ability of people to notice these improvements is plateauing because they are not trained to spot artifacts, which are becoming more obscure.
Earlier quoted context omitted.
I agree with your sentiment, but I think we've yet to see the full application of the current technology. (Even if LLMs themselves don't improve, there's significant opportunity for people to use it in ways not currently being done)
The issue with llm’s is trust. I don’t see that ever going away. Humans have learned to trust other humans over a large time scale with rules in place to control behaviour.
We've compared now more than a hundred replies to that of GPT Pro, and the quality is roughly the same. Sometimes a little worse, sometimes a little better. Always more detailed. Never unacceptable.
But how to convince our customers that we have the right technology and know how to use it appropriately? We're trying, but it's not easy.
Part of that's accountability. In the event of the LLM producing rubbish, as rare as it may be, who is accountable? There is not a person and her reputation attached to it.
Controversial opinion from a casual user, but state-of-art LLMs now feel to me more intelligent then the average person on the steet. Also explains why training on more average-quality data (if there's any left) is not making improvements. But LLMs are hamstrung by their harnesses. They are doing the equivalent of providing technical support via phone call: little to no context, and limited to a bidirectional stream…
Steet? Do you mean street? They're smarter in the same way a search engine is smarter.
And search engines are narrow tools that can only output copies of its dataset. An LLM is capable of surprisingly novel output, even if the exact level of creativity is heavily debated.
Assuming we get no better than opus 4.6, they're very capable. Even if they make up nonsense 5% of the time!
Earlier quoted context omitted.
Image quality has improved a lot in recent months thanks to better models. The ability of people to notice these improvements is plateauing because they are not trained to spot artifacts, which are becoming more obscure.
Yes, slight increase in that kind of accuracy. And newer models still generate absurd stuff. Ask for an historical picture, like 'a London market in the 18th century', and it is still as historically wrong as it was 2 years ago. It is useful for fantasy/sci-fi though, I use them a lot. But I don't see the point of newer models since late 2024.
I agree completely. I haven't noticed much improvement in coding ability in the last year. I'm using frontier models. What's been the game changer are tools like Claude Code. Automatic agentic tool loops purpose built for coding. This is what I have seen as the impetus for mainstream adoption rather than noticeable improvements in ability.
My anecdotal experience is rather different. I write a lot of C++ and QML code. Codex 5.3, only released in Feb, is the the first model I've used that would regularly generate code that passes my 25 years expert smell test and has turned generative coding from a timesap/nuisance into a tool I can somewhat rely on not to set me back. Claude still wasn't quite there at the time, but I haven't tried 4.6 yet. QML is a de…
Data is missing on this chart. It's my experience that opus 4, and then, particularly, 4.5, in Claude code, are head and shoulders above the competition. I wrote an agentic coder years ago and it yielded trash. (Tried to make it do then what kiro does today). The models are better. Now, caveat - I don't use anything but opus for coding - Sonnet doesn't do the trick. My experience with Codex and Gemini is that their t…
Although I feel like for chasing bugs and big systems codex is even better
On the other hand, LLMs tend to go for an average by their nature (if you squint enough). What's more common in their training data, it's more common in the output, so getting them better without fundamental changes, requires one to improve the training data on average too which is hard.
What did improve a lot is the tooling around them. That's gotten way better.