My only take home is they are all terrible and I should hire a professional.
My only take home is that a spanner can work as a hammer, but you probably should just get a hammer
The last six months in LLMs, illustrated by pelicans on bicycles
71–80 of 244 posts
Re: The last six months in LLMs, illustrated by pelicans on bicycles
#72If you would give a human the SVG documentation and ask to write an SVG, I think the results would be quite similar.
Re: The last six months in LLMs, illustrated by pelicans on bicycles
#73If you would give a human the SVG documentation and ask to write an SVG, I think the results would be quite similar.
Lets give it a try, if you're willing to be the experiment subject :)
The prompt is "Generate an SVG of a pelican riding a bicycle" and you're supposed to write it by hand, so no graphical editor. The specification is here: https://www.w3.org/TR/SVG2/
I'm fairly certain I'd lose interest in getting it right before I got something better than most of those.
Re: The last six months in LLMs, illustrated by pelicans on bicycles
#74Earlier quoted context omitted.
It might not be 100% clear from the writing but this benchmark is mainly intended as a joke - I built a talk around it because it's a great way to make the last six months of model releases a lot more entertaining. I've been considering an expanded version of this where each model outputs ten images, then a vision model helps pick the "best" of those to represent that model in a further competition with other models.…
I'd say definitely do not do that. That would make the benchmark look more serious while still being problematic for knowledge cutoff reasons. Your prompt has become popular even outside your blog, so the odds of some SVG pelicans on bicycles making it into the training data have been going up and up. Karpathy used it as an example in a recent interview: https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3...
Yeah, this is the problem with benchmarks where the questions/problems are public. They're valuable for some months, until it bleeds into the training set. I'm certain a lot of the "improvements" we're seeing are just benchmarks leaking into the training set.
Re: The last six months in LLMs, illustrated by pelicans on bicycles
#75Re: The last six months in LLMs, illustrated by pelicans on bicycles
#76I don’t know what secret sauce Anthropic has, but in real world use, Sonnet is somehow still the best model around. Better than Opus and Gemini Pro
Statements like these are useless without sharing exactly all the models you've tried. Sonnet beats O1 Pro Mode for example? Not in my experience, but I haven't tried the latest Sonnet versions, only the one before, so wouldn't claim O1 Pro Mode beats everything out there.
Besides, it's so heavily context-dependent that you really need your own private benchmarks to make head or tails out of this whole thing.
Re: The last six months in LLMs, illustrated by pelicans on bicycles
#77My biggest gripe is that he's comparing probabilistic models (LLMs) by a single sample. You wouldn't compare different random number generators by taking one sample from each and then concluding that generator 5 generates the highest numbers... Would be nicer to run the comparison with 10 images (or more) for each LLM and then average.
I think you mean non-deterministic, instead of probabilistic. And there is no reason that these models need to be non-deterministic.
A deterministic algorithm can still be unpredictable in a sense. In the extreme case, a procedural generator (like in Minecraft) is deterministic given a seed, but you will still have trouble predicting what you get if you change the seed, because internally it uses a (pseudo-)random number generator.
So there’s still the question of how controllable the LLM really is. If you change a prompt slightly, how unpredictable is the change? That can’t be tested with one prompt.
Re: The last six months in LLMs, illustrated by pelicans on bicycles
#78Earlier quoted context omitted.
Even if it is a joke, having a consistent methodology is useful. I did it for about a year with my own private benchmark of reasoning type questions that I always applied to each new open model that came out. Run it once and you get a random sample of performance. Got unlucky, or got lucky? So what. That's the experimental protocol. Running things a bunch of times and cherry picking the best ones adds human bias, and…
It wasn't until I put these slides together that I realized quite how well my joke benchmark correlates with actual model performance - the "better" models genuinely do appear to draw better pelicans and I don't really understand why!
I just don't get the fuss from the pro-LLM people who don't want anyone to shame their LLMs...
people expect LLMs to say "correct" stuff on the first attempt, not 10000 attempts.
Yet, these people are perfectly OK with cherry-picked success stories on youtube + advertisements, while being extremely vehement about this simple experiment...
...well maybe these people rode the LLM hype-train too early, and are desperate to defend LLMs lest their investment go poof?
obligatory hype-graph classic: https://upload.wikimedia.org/wikipedia/commons/thumb/9/94/Ga...
Re: The last six months in LLMs, illustrated by pelicans on bicycles
#79Earlier quoted context omitted.
Even if it is a joke, having a consistent methodology is useful. I did it for about a year with my own private benchmark of reasoning type questions that I always applied to each new open model that came out. Run it once and you get a random sample of performance. Got unlucky, or got lucky? So what. That's the experimental protocol. Running things a bunch of times and cherry picking the best ones adds human bias, and…
It wasn't until I put these slides together that I realized quite how well my joke benchmark correlates with actual model performance - the "better" models genuinely do appear to draw better pelicans and I don't really understand why!
until they start targeting this benchmark
Re: The last six months in LLMs, illustrated by pelicans on bicycles
#80My biggest gripe is that he's comparing probabilistic models (LLMs) by a single sample. You wouldn't compare different random number generators by taking one sample from each and then concluding that generator 5 generates the highest numbers... Would be nicer to run the comparison with 10 images (or more) for each LLM and then average.
I think you mean non-deterministic, instead of probabilistic. And there is no reason that these models need to be non-deterministic.
> I think you mean non-deterministic, instead of probabilistic.
My thoughts too. It's more accurate to label LLMs as non-deterministic instead of "probablistic".