Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The last six months in LLMs in five minutes

simonwillison.net

Post reply on HN
Print view View original

21–30 of 631 posts

Re: The last six months in LLMs in five minutes

#21

Post by bluegatty »

Insanity wrote: post #10

I wonder how much the 'inflection point' is a thing vs marketing. I'm sure the models got somewhat better, but even now when I'm trying to 'vibe code' a game with the latest models (combination of Codex w/ gpt5.5 and gpt5.3-codex), they really do struggle. They definitely get something barebones up and running, but it's far from a fully fledged application.

Paradox - you can get multiple inflection points even as systems start to have dimishing marginal returns in core capability, I think this is due to 'threshold crossing' where something 'becomes good enough for a specific purpose' - it just unlocks capabilities.

'Nail Guns' used to be heavy, required heavy power cords, they were extremely expensive. When they got lighter, cheaper, battery pack ... at some point, they blend seamlessly into the roofers process, and multiply dramatically the work that can be done. Marginal improvements beyond that may not yield the same 'unlocks' because the threshold has been crossed.

Re: The last six months in LLMs in five minutes

#22

Post by _puk »

shepherdjerred wrote: post #7

> and there’s zero chance any AI lab would train a model for such a ridiculous task. I'm not sure that's true anymore considering how popular Simon's blog is

> So maybe the AI labs have been paying attention after all!

> I think this mainly demonstrates that the pelican on the bicycle has firmly exceeded its limits as a useful benchmark.

As acknowledged in the article.

Re: The last six months in LLMs in five minutes

#23

Post by halflife »

Insanity wrote: post #10

I wonder how much the 'inflection point' is a thing vs marketing. I'm sure the models got somewhat better, but even now when I'm trying to 'vibe code' a game with the latest models (combination of Codex w/ gpt5.5 and gpt5.3-codex), they really do struggle. They definitely get something barebones up and running, but it's far from a fully fledged application.

I feel the change. It went from an autocomplete tool, to an agent running 5 tasks in parallel while I just supervise. The improvement is enormous.

Re: The last six months in LLMs in five minutes

#24

Post by nl »

rTX5CMRXIfFG wrote: post #12

Am I crazy, or are these differences between the best models so marginal that you’d get roughly the same performance if you use the same high-quality harness (ie preloaded instructions from md files, including custom skills)?

The difference is very noticeable as your codebase gets bigger and you give higher and higher level tasks.

I've certainly had things that Opus fixed using some kind of work around that GPT-5.5 actually solved.

And the difference between the Sonnet/Gemini/DeepSeek tier to the Opus/GPT-5.5 tier is immediately obvious.

Re: The last six months in LLMs in five minutes

#25

Post by raincole »

rTX5CMRXIfFG wrote: post #12

Am I crazy, or are these differences between the best models so marginal that you’d get roughly the same performance if you use the same high-quality harness (ie preloaded instructions from md files, including custom skills)?

By definition the differences between "best models" are small. It's tautology. If a model is significantly dumber than the others then it's not one of the best models.

Re: The last six months in LLMs in five minutes

#26

Post by _puk »

throwaway2027 wrote: post #8

December 2025 was the breakthrough for me. January Claude was euphoric, ChatGPT was up there. February Gemini cooked for a second there. March amazing. April the big bad nerf. May GPT 5.5 is just pure bliss altough 2x limits temporarily, not sure about Claude it's sort of okay still not as good as it felt before, slowly increasing limits with more compute and rebuilding good will.

I think Opus 4.6 at its peak was the "how can anyone not get that this is good" for me.

Then the nerf, and the massive uplift in tokens for 4.7, a model which I find lazy and prone to hallucinate.

It's probably time to try GPT5.5. Like many I'm pretty heavily invested in the anthropic ecosystem at this point, which I suppose gives another strong reason to make the switch.

Re: The last six months in LLMs in five minutes

#27

Post by nl »

bluegatty wrote: post #17

'Producing Images' or even 'Some Code that is Valid and Compiles' is in some ways one of the most misleading ways we assess quality of the AI. It is getting very good at producing code that compiles - at the algorithmic level. This is definitely noteworthy - and the AI is crossing a critical 'productivity threshold'. But 'Drawing of a Proper Duck' is almost arbitrary because it may have nothing to do with the ' Speci…

> But 'Drawing of a Proper Duck' is almost arbitrary because it may have nothing to do with the 'Specific Duck You Wanted'.

That might be the case, but Simon's case "Generate an SVG of a pelican riding a bicycle" is very different.

The model actually has to understand what parts of a pelican and bicycle come together in something like an anatomically plausible way. That's a higher level of abstraction than something like passing the same prompt to Stable Diffusion etc

(The new Nano Banana/GPT Image 2.0 models are different though - they have significant world knowledge baked in)

Re: The last six months in LLMs in five minutes

#29

Post by bluegatty »

nl wrote: post #27
bluegatty wrote: post #17

'Producing Images' or even 'Some Code that is Valid and Compiles' is in some ways one of the most misleading ways we assess quality of the AI. It is getting very good at producing code that compiles - at the algorithmic level. This is definitely noteworthy - and the AI is crossing a critical 'productivity threshold'. But 'Drawing of a Proper Duck' is almost arbitrary because it may have nothing to do with the ' Speci…

> But 'Drawing of a Proper Duck' is almost arbitrary because it may have nothing to do with the 'Specific Duck You Wanted'. That might be the case, but Simon's case "Generate an SVG of a pelican riding a bicycle" is very different. The model actually has to understand what parts of a pelican and bicycle come together in something like an anatomically plausible way. That's a higher level of abstraction than something…

"That's a higher level of abstraction"

No, it's not because it's seen 'anatomy' for Pelicans, Animals - even how it's represented in Animals.

If you try to get the AI to actually decompose it and start to 'draw pelicans' in very obscure ways, it will immediately fail.

Try to get the AI to draw the pelican form a very odd angle - like underneath, to the right, one wing extended, one wing not ... 0% chance.

Precisely because it does not understand those things.

FYI it's a slightly unfair case because it does not have 'world model' yet, which will actually solve that problem, but even then not through very much abstracting.

We're a long way away - but in the meantime, there's lots to unpack.

Re: The last six months in LLMs in five minutes

#30

Post by kvakkefly »

Insanity wrote: post #10

I wonder how much the 'inflection point' is a thing vs marketing. I'm sure the models got somewhat better, but even now when I'm trying to 'vibe code' a game with the latest models (combination of Codex w/ gpt5.5 and gpt5.3-codex), they really do struggle. They definitely get something barebones up and running, but it's far from a fully fledged application.

I remember this very clearly myself. Before opus 4.5, I was doing a lot of hand holding and was coding a lot myself, but I have not written code since that day more or less.

I did write some stuff myself just to learn how the enigma encryption machine worked, so wrote myself to learn. But professionally, I stopped coding in November.

Post reply on HN