Mistral releases Devstral2 and Mistral Vibe CLI • Hacker News Board

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#331

Post by 0cf8612b2e1e » Wed, Dec 10, 2025, 4:38 PM UTC

theshrike79 wrote: post #298

0cf8612b2e1e wrote: post #235

Earlier quoted context omitted.

I assume all of the models also have variations on, “how many ‘r’s in strawberry”.

The easiest way to fix these is give the model an environment to run code. Any model can easily one-shot a python script that can count the occurrence of any letter anywhere and return the result. It's just a tooling issue. You really can't "train" an LLM to do it because tokenisation and ... stuff.

I am not convinced they are executing code. Otherwise I would expect LLMs to not frequently guess the result of math questions.

Of course you could train it. Some quick scripting to find all words with repeat letters, build up sample sentences (aardvark has three a,) and you have hard coded the answer to simple questions that make your LLM look stupid.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#332

Post by samgutentag » Wed, Dec 10, 2025, 5:44 PM UTC

simonw wrote: post #33

llm install llm-mistral llm mistral refresh llm -m mistral/devstral-2512 "Generate an SVG of a pelican riding a bicycle" https://tools.simonwillison.net/svg-render#%3Csvg%20xmlns%3D... Pretty good for a 123B model! (That said I'm not 100% certain I guessed the correct model ID, I asked Mistral here: https://x.com/simonw/status/1998435424847675429 )

"Generate an SVG of a pelican riding a bicycle" is the new "but can it run Crysis"

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#333

Post by lgrapenthin » Wed, Dec 10, 2025, 6:01 PM UTC

I tried this on a small Clojure codebase and asked it to write some tests. It couldn't get its parentheses balanced. After 10 attempts or so it tried to write a smaller test file first, but again failed. Regardless of the parentheses, the test code it came up with was quite basic and arbitrary. It didn't try to come up with interesting edge cases or anything.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#334

Post by dugidugout » Wed, Dec 10, 2025, 6:02 PM UTC

renewiltord wrote: post #303

bravetraveler wrote: post #178

Earlier quoted context omitted.

Not sure if I'd use the same descriptions so pointedly , but I can see what they mean. It's perfectly fine to link for convenience, but it does feel a little disrespectful/SEO-y to not 'continue the conversation' . A summary in the very least, how exactly it pertains. Sell us. In a sense, link-dropping [alone] is saying: "go read this and establish my rhetorical/social position, I'm done here" Imagine meeting an auth…

It is SEO-y and I’m sure no small impulse is to drive traffic to his website since he’s primarily an AI influencer. However, there are always people who are “native” to a platform and field. Pieter Levels is native to Twitter and the nomad community. Swyx is native to Twitter/HN and the devtools community. And simonw is native to at least HN and the LLM-interest community. And various streamers and onlyfans creators…

Well put and thank you for adding much depth here!

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#335

Post by lacoolj » Wed, Dec 10, 2025, 6:12 PM UTC

Wonder why Gemini 3 Pro and Sonnet 4.5 are on this comparison but Opus 4.5 is not?

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#336

Post by lacoolj » Wed, Dec 10, 2025, 6:13 PM UTC

simonw wrote: post #33

llm install llm-mistral llm mistral refresh llm -m mistral/devstral-2512 "Generate an SVG of a pelican riding a bicycle" https://tools.simonwillison.net/svg-render#%3Csvg%20xmlns%3D... Pretty good for a 123B model! (That said I'm not 100% certain I guessed the correct model ID, I asked Mistral here: https://x.com/simonw/status/1998435424847675429 )

How did you run a 123B model locally? Or did you do this on a GPU host somewhere? If so, what spec was it?

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#337

Post by fastball » Wed, Dec 10, 2025, 6:19 PM UTC

JimDabell wrote: post #270

fastball wrote: post #168

Earlier quoted context omitted.

yeahhhhhhh, that's not how this works. Unless this authority has some ownership over the term and can prevent its misuse (e.g. with lawsuits or similar), it is not actually the authority of the term, and people will continue to use it how they see fit. Indeed, I am not part of a movement (nor would I want to be) which focuses more on what words are used rather than what actions are taken.

> people will continue to use it how they see fit. And whenever they do so, this pointless argument will happen. Again, and again, and again. Because that’s not what the word means and your desired redefinition has been consistently and continuously rejected over and over again for decades. What do you gain from misusing this term? The only thing it does is make you look dishonest and start arguments.

Prescriptivists about language always lose in the end. That is the only point I am making. Words mean what people use them for, not what you want them to mean.

I am not misusing the term, but people are, according to your standards. And it is easy for them to do so, because "open source" was poorly named to begin with.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#338

Post by ido » Wed, Dec 10, 2025, 6:29 PM UTC

vidarh wrote: post #84

embedding-shape wrote: post #62

Earlier quoted context omitted.

Claude Code has absolutely zero features that help me review code or do anything else than vibe-coding and accept changes as they come in. We need diff-comparisons between different executions, tailored TUI for that kind of work and more. Claude Code is basically a MVP of that. Still, I do use Claude Code and Codex daily as there is nothing better out there currently. But they still feel tailored towards vibe-coding…

I really do not want those things in Claude COde - I much prefer choosing my own diff tools etc. and running them in a separate terminal. If they start stuffing too much into the TUI they'd ruin it - if you want all that stuff built in, they have the VS Code integration.

using claude code via the VS Code plugin gives you side by side diffs as it works.

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#339

Post by jgalt212 » Wed, Dec 10, 2025, 6:59 PM UTC

100721 wrote: post #326

jgalt212 wrote: post #313

Earlier quoted context omitted.

Aiden is perhaps misinformed. From a Bing search performed just now. > Yes, I am familiar with the "pelican riding a bicycle" SVG generation test. It is a benchmark for evaluating the ability of AI models, particularly large language models (LLMs) and multi-modal systems, to generate original, high-quality SVG vector graphics based on a deliberately unusual and complex prompt. The benchmark was popularized by Simon W…

Web search-based RAG is very different from having something embedded in a model's training data, though.

ChatGPT website gives a similar answer. Are they running RAG, or the model?

> Yes — I’m familiar with the “pelican riding a bicycle” SVG generation test.

> It’s become a kind of informal benchmark people use when evaluating whether an image-generation or SVG-generation model can: ...

Re: Mistral releases Devstral2 and Mistral Vibe CLI

#340

Post by tgtweak » Wed, Dec 10, 2025, 7:25 PM UTC

lostmsu wrote: post #225

tgtweak wrote: post #207

Earlier quoted context omitted.

dual 3090's (24GB each) on 8x+8x pcie has been a really reliable setup for me (with nvlink bridge... even though it's relatively low bandwidth compared to tesla nvlink, it's better than going over pcie!) 48GB of vram and lots of cuda cores, hard to beat this value atm. If you want to go even further, you can get an 8x V100 32GB server complete with 512GB ram and nvlink switching for $7000 USD from unixsurplus (ebay.c…

V100 is outdated (no bf16, dropped in CUDA 13) and power hungry (8 cards 3 years continuous use are about $12k of electricity).

Depends where you are plugging them in - but yes they are older gen - despite this, 8xV100 will outperform most of what you can buy for that price simply by way of memory and nvlink bandwidth. If you want to practically run a local model that takes 200GB of memory (Devstral-2-123B-Instruct-2512 for example or GPT-OSS-120B with long context window) without resorting to aggressive ggufs or memory swapping, you don't have many cheaper options. You can also parallelize several models on one node to get some additional throughput for bulk jobs.