Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral releases Devstral2 and Mistral Vibe CLI

mistral.ai

Post reply on HN
Print view View original

231–240 of 363 posts

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#231

Post by thatwasunusual »

Jimmc414 wrote: post #109
simonw wrote: post #33

llm install llm-mistral llm mistral refresh llm -m mistral/devstral-2512 "Generate an SVG of a pelican riding a bicycle" https://tools.simonwillison.net/svg-render#%3Csvg%20xmlns%3D... Pretty good for a 123B model! (That said I'm not 100% certain I guessed the correct model ID, I asked Mistral here: https://x.com/simonw/status/1998435424847675429 )

We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. It would be a great way to ensure an initial thumbs up from a prominent reviewer. It's a good benchmark but it seems like it would be a good idea to include an additional random or unannounced similar test to catch any benchmaxxing.

> We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data.

I may be stupid, but _why_ is this prompt used as a benchmark? I mean, pelicans _can't_ ride a bicycle, so why is it important for "AI" to show that they can (at least visually)?

The "wine glass problem"[0] - and probably others - seems to me to be a lot more relevant...?

[0] https://medium.com/@joe.richardson.iii/the-curious-case-of-t...

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#232

Post by simonw »

thatwasunusual wrote: post #231
Jimmc414 wrote: post #109

Earlier quoted context omitted.

We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. It would be a great way to ensure an initial thumbs up from a prominent reviewer. It's a good benchmark but it seems like it would be a good idea to include an additional random or unannounced similar test to catch any benchmaxxing.

> We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. I may be stupid, but _why_ is this prompt used as a benchmark? I mean, pelicans _can't_ ride a bicycle, so why is it important for "AI" to show that they can (at least visually)? The "wine glass problem"[0] - and probably others - seems to me to be a lot more r…

The fact that pelicans can't ride bicycles is pretty much the point of the benchmark! Asking an LLM to draw something that's physically impossible means it can't just "get it right" - seeing how different models (especially at different sizes) handle the problem is surprisingly interesting.

Honestly though, the benchmark was originally meant to be a stupid joke.

I only started taking it slightly more seriously about six months ago, when I noticed that the quality of the pelican drawings really did correspond quite closely to how generally good the underlying models were.

If a model draws a really good picture of a pelican riding a bicycle there's a solid chance it will be great at all sorts of other things. I wish I could explain why that was!

If you start here and scroll through and look at the progression of pelican on bicycle images it's honestly spooky how well they match the vibes of the models they represent: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-...

So ever since then I've continue to get models to draw pelicans. I certainly wouldn't suggest anyone take serious decisions on model usage based on my stupid benchmark, but it's a fun first-day initial impression thing and it appears to be a useful signal for which models are worth diving into in more detail.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#233

Post by wisty »

thatwasunusual wrote: post #231
Jimmc414 wrote: post #109

Earlier quoted context omitted.

We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. It would be a great way to ensure an initial thumbs up from a prominent reviewer. It's a good benchmark but it seems like it would be a good idea to include an additional random or unannounced similar test to catch any benchmaxxing.

> We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. I may be stupid, but _why_ is this prompt used as a benchmark? I mean, pelicans _can't_ ride a bicycle, so why is it important for "AI" to show that they can (at least visually)? The "wine glass problem"[0] - and probably others - seems to me to be a lot more r…

It's not nessessarily the best benchmark, it's a popular one, probably because it's funny.

Yes it's like the wine glass thing.

Also it's kind of got depth. Does it draw the pelican and the bicycle? Can the penguin reach the peddles? How?

I can imagine a really good AI finding a funny or creative or realistic way for the penguin to reach the peddles.

An slightly worse AI will do an OK job, maybe just making the bike small or the legs too long.

An OK AI will draw a penguin on top of a bicycle and just call it a day.

It's not as binary as the wine glass example.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#234

Post by zmmmmm »

embedding-shape wrote: post #224
zmmmmm wrote: post #222

Earlier quoted context omitted.

the question is, how do you want to provide instructions for what the AI is to do? You might not like calling it "chat" but somehow you need to communicate that, right? With aider you can write a comment for a function and then instruct it to finish the function inline (see other comments). But unless you just want pure autocomplete based on it guessing things, you need to provide guidance to it somehow.

I don't know exactly, but I guess in a more declarative manner rather than anything. Maybe we set goals/milestones/concrete objectives, or similar, rather than imperatively steer it, give it space to experiment yet make it very easy to understand exactly what important tradeoffs everything is doing. It's all very fluffy and theoretical of course.

I find a good compromise on that front is not to use the chat primarily, but to create files like 'ARCHITECTURE.md', 'REQUIREMENTS.md' and put information in there describing how the application works. Then you add those to the chat as context docs.From the chat interface then you are just referring to those not just describing features willy nilly. So the nice thing is you are building documentation for the application in a formal sense as part of instructing the LLM.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#235

Post by 0cf8612b2e1e »

Jimmc414 wrote: post #109
simonw wrote: post #33

llm install llm-mistral llm mistral refresh llm -m mistral/devstral-2512 "Generate an SVG of a pelican riding a bicycle" https://tools.simonwillison.net/svg-render#%3Csvg%20xmlns%3D... Pretty good for a 123B model! (That said I'm not 100% certain I guessed the correct model ID, I asked Mistral here: https://x.com/simonw/status/1998435424847675429 )

We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. It would be a great way to ensure an initial thumbs up from a prominent reviewer. It's a good benchmark but it seems like it would be a good idea to include an additional random or unannounced similar test to catch any benchmaxxing.

I assume all of the models also have variations on, “how many ‘r’s in strawberry”.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#236

Post by Workaccount2 »

Jimmc414 wrote: post #109
simonw wrote: post #33

llm install llm-mistral llm mistral refresh llm -m mistral/devstral-2512 "Generate an SVG of a pelican riding a bicycle" https://tools.simonwillison.net/svg-render#%3Csvg%20xmlns%3D... Pretty good for a 123B model! (That said I'm not 100% certain I guessed the correct model ID, I asked Mistral here: https://x.com/simonw/status/1998435424847675429 )

We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. It would be a great way to ensure an initial thumbs up from a prominent reviewer. It's a good benchmark but it seems like it would be a good idea to include an additional random or unannounced similar test to catch any benchmaxxing.

It would be easy to out models that train on the bike pelican, because they would probably suck at the kayaking bumblebee.

So far though, the models good at bike pelican are also good at kayak bumblebee, or whatever other strange combo you can come up with.

So if they are trying to benchmaxx by making SVG generation stronger, that's not really a miss, is it?

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#237

Post by embedding-shape »

zmmmmm wrote: post #234
embedding-shape wrote: post #224

Earlier quoted context omitted.

I don't know exactly, but I guess in a more declarative manner rather than anything. Maybe we set goals/milestones/concrete objectives, or similar, rather than imperatively steer it, give it space to experiment yet make it very easy to understand exactly what important tradeoffs everything is doing. It's all very fluffy and theoretical of course.

I find a good compromise on that front is not to use the chat primarily, but to create files like 'ARCHITECTURE.md', 'REQUIREMENTS.md' and put information in there describing how the application works. Then you add those to the chat as context docs.From the chat interface then you are just referring to those not just describing features willy nilly. So the nice thing is you are building documentation for the applicat…

But that is the typical agentic LLM coder style program I was initially referring to, saying we maybe should explore other alternatives to. It's too basic and primitive, with some imagination.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#238

Post by mhast »

embedding-shape wrote: post #224
zmmmmm wrote: post #222

Earlier quoted context omitted.

the question is, how do you want to provide instructions for what the AI is to do? You might not like calling it "chat" but somehow you need to communicate that, right? With aider you can write a comment for a function and then instruct it to finish the function inline (see other comments). But unless you just want pure autocomplete based on it guessing things, you need to provide guidance to it somehow.

I don't know exactly, but I guess in a more declarative manner rather than anything. Maybe we set goals/milestones/concrete objectives, or similar, rather than imperatively steer it, give it space to experiment yet make it very easy to understand exactly what important tradeoffs everything is doing. It's all very fluffy and theoretical of course.

The typical "best practice" for these tools tend to be to ask it something like

"I want you to do feature X. Analyse the code for me and make suggestions how to implement this feature."

Then it will go off and work for a while and typically come back after a bit with some suggestions. Then iterate on those if needed and end with.

"Ok. Now take these decided upon ideas and create a plan for how to implement. And create new tests where appropriate."

Then it will go off and come back with a plan for what to do. And then you send it off with.

"Ok, start implementing."

So sure. You probably can work on this to make it easier to use than with a CLI chat. It would likely be less like an IDE and more like a planning tool you'd use with human colleagues though.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#239

Post by Barbing »

bravetraveler wrote: post #213
simonw wrote: post #209

Earlier quoted context omitted.

I've genuinely been answering the question "what if the labs are training on your pelican benchmark" 3-4 times a week for several months at this point. I wrote that piece precisely so I didn't have to copy and paste the same arguments into dozens of different conversations.

Oh, no . Does this policing job pay well? /s Seriously: less is more, trust the process, any number of platitudes work here. Who are you defending against? Readers, right? You wrote your thing, defended it with more of the thing. It'll permeate. Or it won't. Does it matter? You could be done, nothing is making you defend this (sorry) asinine benchmark across the internet. Not trying to (m|y)uck your yum, or whatever.…

Surprised to see snark re: what I thought was a standard practice (linking FAQs, essentially).

I hadn’t seen the post. It was relevant. I just read it. Lucky Ten Thousand can read it next time even though I won’t.

Simon has never seemed annoying so unlike other comments that might worry me (even “Opus made this” even though it’s cool but I’m concerned someone astroturfed), that comment would’ve never raised my eyebrows. He’s also dedicated and I love he devotes his time to a new field like this where it’s great to have attempts at benchmarks, folks cutting through chaff, etc.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#240

Post by JoshTriplett »

kouteiheika wrote: post #96
badsectoracula wrote: post #60

Earlier quoted context omitted.

These are not permissively licensed though, the terms "permissive license" has connotations that pretty much everyone who is into FLOSS understands (same with "open source"). I do not mind having a license like that, my gripe is with using the terms "permissive" and "open source" like that because such use dilutes them. I cannot think of any reason to do that aside from trying to dilute the term (especially when some…

> I do not mind having a license like that, my gripe is with using the terms "permissive" and "open source" like that because such use dilutes them. I cannot think of any reason to do that aside from trying to dilute the term (especially when some laws, like the EU AI Act, are less restrictive when it comes to open source AIs specifically). Good. In this case, let it be diluted! These extra "restrictions" don't affec…

> I couldn't care less that the term is "diluted" and that makes it harder

It also makes life harder for individuals and small companies, because this is not Open Source. It's incompatible with Open Source, it can't be reused in other Open Source projects.

Terms have meanings. This is not Open Source, and it will never be Open Source.

Post reply on HN