Earlier quoted context omitted.
The fact that pelicans can't ride bicycles is pretty much the point of the benchmark! Asking an LLM to draw something that's physically impossible means it can't just "get it right" - seeing how different models (especially at different sizes) handle the problem is surprisingly interesting. Honestly though, the benchmark was originally meant to be a stupid joke. I only started taking it slightly more seriously about…
> If a model draws a really good picture of a pelican riding a bicycle there's a solid chance it will be great at all sorts of other things. Why? If I hired a worker that was really good at drawing pelicans riding a bike, it wouldn't tell me anything about his/her other qualities?!
Mistral releases Devstral2 and Mistral Vibe CLI
301–310 of 363 posts
Re: Mistral releases Devstral2 and Mistral Vibe CLI
#302Earlier quoted context omitted.
I don't know exactly, but I guess in a more declarative manner rather than anything. Maybe we set goals/milestones/concrete objectives, or similar, rather than imperatively steer it, give it space to experiment yet make it very easy to understand exactly what important tradeoffs everything is doing. It's all very fluffy and theoretical of course.
I think the problem is that models are just not that good yet. At least for my usage at work, the CLI tools are the fastest way to get something useful, but if you can't describe basically exactly what you want, you get garbage.
Re: Mistral releases Devstral2 and Mistral Vibe CLI
#303Earlier quoted context omitted.
Condescending and disrespectful to whom? Everybody wholsale? This doesnt seem reasonable? Please elaborate.
Not sure if I'd use the same descriptions so pointedly , but I can see what they mean. It's perfectly fine to link for convenience, but it does feel a little disrespectful/SEO-y to not 'continue the conversation' . A summary in the very least, how exactly it pertains. Sell us. In a sense, link-dropping [alone] is saying: "go read this and establish my rhetorical/social position, I'm done here" Imagine meeting an auth…
However, there are always people who are “native” to a platform and field. Pieter Levels is native to Twitter and the nomad community. Swyx is native to Twitter/HN and the devtools community. And simonw is native to at least HN and the LLM-interest community. And various streamers and onlyfans creators do the same with theirs.
Through some degree of releasing things that whatever that community values they build a relationship that allows them greater freedom in participating there. It does create a positive feedback cycle for them (and hopefully the community) that most of them will try to parlay into something else: Levels and the OnlyFans creators are probably best at this monetization of reputation but each of them is doing this. One success step for simonw would be “Creator of Pelican LLM benchmark”.
Once you’ve breached some stable point in the community the norms are somewhat relaxed. But it’s not easy to do that. You have to produce some extraordinary volume of things that people value.
I think, tbh, tptacek here could most effectively monetize if he decided to. But he doesn’t appear to want to so he’s just a participant not an influencer so to speak. Whereas someone like Levels or simonw is both.
It’s just creator economy stuff. Meta discussions like this always pop up. But ultimately simonw is past the threshold of trust. There are people who say “wtf? Why is levels making $50k/mo on a stupid vibe-coded flying game?”
It ain’t the game. It’s the following before the game. The resource is the audience.
Re: Mistral releases Devstral2 and Mistral Vibe CLI
#304Earlier quoted context omitted.
We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. It would be a great way to ensure an initial thumbs up from a prominent reviewer. It's a good benchmark but it seems like it would be a good idea to include an additional random or unannounced similar test to catch any benchmaxxing.
I wrote about that possibility here: https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
Re: Mistral releases Devstral2 and Mistral Vibe CLI
#305Re: Mistral releases Devstral2 and Mistral Vibe CLI
#306Next step, just need a shitload of vram. ;)
Maybe those Intel Battlematrix 48GB cards might be useful after all... :)
https://www.storagereview.com/review/intel-arc-pro-b60-battl...
Re: Mistral releases Devstral2 and Mistral Vibe CLI
#307Earlier quoted context omitted.
Not sure if I'd use the same descriptions so pointedly , but I can see what they mean. It's perfectly fine to link for convenience, but it does feel a little disrespectful/SEO-y to not 'continue the conversation' . A summary in the very least, how exactly it pertains. Sell us. In a sense, link-dropping [alone] is saying: "go read this and establish my rhetorical/social position, I'm done here" Imagine meeting an auth…
It is SEO-y and I’m sure no small impulse is to drive traffic to his website since he’s primarily an AI influencer. However, there are always people who are “native” to a platform and field. Pieter Levels is native to Twitter and the nomad community. Swyx is native to Twitter/HN and the devtools community. And simonw is native to at least HN and the LLM-interest community. And various streamers and onlyfans creators…
The best guy spinning the sign puts some effort in, or more crass, the best strippers make you believe.
Re: Mistral releases Devstral2 and Mistral Vibe CLI
#308Earlier quoted context omitted.
> send me to a discussion I might have expressed interest in No, no, remember? Points to the blog you were already reading! Working diligently to build a brand: podcast, paid newsletter, the works.
I wasn't speaking to this interaction, and my point is genuine. Simonw has done fantastic work in the LLM space
This interaction is, effectively, a link dropped with an announcement of an announcement. For what has already occurred. Over-fitting, training? You don't say.
If I wanted to be more of an ass, I'd look to argue about hype generation. But I don't, I appreciate any honest effort, which I believe for Simon.
Re: Mistral releases Devstral2 and Mistral Vibe CLI
#309Earlier quoted context omitted.
> It's not nessessarily the best benchmark, it's a popular one, probably because it's funny. > Yes it's like the wine glass thing. No, it's not! That's part of my point; the wine glass scenario is a _realistic_ scenario. The pelican riding a bike is not. It's a _huge_ difference. Why should we measure intelligence (...) in regards to something that is realistic and something that is unrealistic? I just don't get it.
> the wine glass scenario is a _realistic_ scenario It is unrealistic because if you go to a restaurant, you don't get served a glass like that. It is frowned upon (alcohol is a drug, after all) and impractical (wine stains are annoying) to fill a glass of wine as such. A pelican riding a bike, on the other hand, is realistic in a scenario because of TV for children. Example from 1950's animation/comic involving a pe…
Since people look at a glass of wine and judge how much "value" they got based partly on how much wine it looks like, many bars and restaurants choose bad wine-glasses (for the purpose of enjoying wine) that are smalle and thus can be fulled more.
Re: Mistral releases Devstral2 and Mistral Vibe CLI
#310Somehow it writes bad React code and misses to check linting prompts half the time. But surprisingly, the Python coding was great!