Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral 3 family of models released

mistral.ai

Post reply on HN
Print view View original

141–150 of 243 posts

Re: Mistral 3 family of models released

#141

Post by giancarlostoro »

mrtksn wrote: post #69
barrell wrote: post #38

I use large language models in http://phrasing.app to format data I can retrieve in a consistent skimmable manner. I switched to mistral-3-medium-0525 a few months back after struggling to get gpt-5 to stop producing gibberish. It's been insanely fast, cheap, reliable, and follows formatting instructions to the letter. I was (and still am) super super impressed. Even if it does not hold up in benchmarks, it still out…

Some time ago I canceled all my paid subscriptions to chatbots because they are interchangeable so I just rotate between Grok, ChatGPT, Gemini, Deepseek and Mistral. On the API side of things my experience is that the model behaving as expected is the greatest feature. There I also switched to Openrouter instead of paying directly so I can use whatever model fits best. The recent buzz about ad-based chatbot services…

Maybe give Perplexity a shot? It has Grok, ChatGPT, Gemini, Kimi K2, I dont think it has Mistral unfortunately.

Re: Mistral 3 family of models released

#142

Post by airstrike »

pants2 wrote: post #132
mbowcut2 wrote: post #125

Earlier quoted context omitted.

It makes me wonder about the gaps in evaluating LLMs by benchmarks. There almost certainly is overfitting happening which could degrade other use cases. "In practice" evaluation is what inspired the Chatbot Arena right? But then people realized that Chatbot arena over-prioritizes formatting, and maybe sycophancy(?). Makes you wonder what the best evaluation would be. We probably need lots more task-specific models. T…

The best benchmark is one that you build for your use-case. I finally did that for a project and I was not expecting the results. Frontier models are generally "good enough" for most use-cases but if you have something specific you're optimizing for there's probably a more obscure model that just does a better job.

If you and others have any insights to share on structuring that benchmark, I'm all ears.

There a new model seemingly every week so finding a way to evaluate them repeatedly would be nice.

The answer may be that it's so bespoke you have to handroll every time, but my gut says there's a set of best practiced that are generally applicable.

Re: Mistral 3 family of models released

#143

Post by moffkalast »

nullbio wrote: post #85

Anyone else find that despite Gemini performing best on benches, it's actually still far worse than ChatGPT and Claude? It seems to hallucinate nonsense far more frequently than any of the others. Feels like Google just bench maxes all day every day. As for Mistral, hopefully OSS can eat all of their lunch soon enough.

Yes, and likewise with Kimi K2. Despite being on the top of open source benches it makes up more batshit nonsense than even Llama 3.

Trust no one, test your use case yourself is pretty much the only approach, because people either don't run benchmarks correctly or have the incentive not to.

Re: Mistral 3 family of models released

#144

Post by barrell »

acuozzo wrote: post #136
barrell wrote: post #38

I use large language models in http://phrasing.app to format data I can retrieve in a consistent skimmable manner. I switched to mistral-3-medium-0525 a few months back after struggling to get gpt-5 to stop producing gibberish. It's been insanely fast, cheap, reliable, and follows formatting instructions to the letter. I was (and still am) super super impressed. Even if it does not hold up in benchmarks, it still out…

I have a need to remove loose "signature" lines from the last 10% of a tremendous e-mail dataset. Based on your experience, how do you think mistral-3-medium-0525 would do?

What's your acceptable error rate? Honestly ministral would probably be sufficient if you can tolerate a small failure rate. I feel like medium would be overkill.

But I'm no expert. I can't say I've used mistral much outside of my own domain.

Re: Mistral 3 family of models released

#145

Post by MadDemon »

denysvitali wrote: post #102
ot wrote: post #20

Earlier quoted context omitted.

Is it so hard for people to understand that Europe is a continent, EU is a federation of European countries, and the two are not the same?

I honestly think it is. The amount of people who thinks Europe and EU are the same thing is really concerning. And no, it's not only americans. I keep hearing this thing from people living in Europe as well (or better, in the EU). I also very often hear phrases like "Switzerland is not in Europe" to indicate that the country is not part of the European Union.

Switzerland has such close ties to the EU that I would consider them half in.

Re: Mistral 3 family of models released

#146

Post by barrell »

data-ottawa wrote: post #107
barrell wrote: post #73

Earlier quoted context omitted.

Yes. I spent about 3 days trying to optimize the prompt to get gpt-5 to not produce gibberish, to no avail. Completions took several minutes, had an above 50% timeout rate (with a 6 minute timeout mind you), and after retrying they still would return gibberish about 15% of the time (12% on one task, 20% on another task). I then tried multiple models, and they all failed in spectacular ways. Only Grok and Mistral had…

With gpt5 did you try adjusting the reasoning level to "minimal"? I tried using it for a very small and quick summarization task that needed low latency and any level above that took several seconds to get a response. Using minimal brought that down significantly. Weirdly gpt5's reasoning levels don't map to the OpenAI api level reasoning effort levels.

Reasoning was set to minimal and low (and I think I tried medium at some point). I do not believe the timeouts were due to the reasoning taking to long, although I never streamed the results. I think the model just fails often. It stops producing tokens and eventually the request times out.

Re: Mistral 3 family of models released

#147

Post by tootie »

nullbio wrote: post #85

Anyone else find that despite Gemini performing best on benches, it's actually still far worse than ChatGPT and Claude? It seems to hallucinate nonsense far more frequently than any of the others. Feels like Google just bench maxes all day every day. As for Mistral, hopefully OSS can eat all of their lunch soon enough.

No? My recent experience with Gemini was terrific. The last big test I gave of Claude it spun an immaculate web of lies before I forced it to confess.

Re: Mistral 3 family of models released

#149

Post by mesebrec »

Havoc wrote: post #36
yoavm wrote: post #19

Earlier quoted context omitted.

How is this different from Llama 3.2 "vision capabilities"? https://www.llama.com/docs/how-to-guides/vision-capabilities...

Guessing GP commenter considers Apache more "open" than Meta's license. Which to be fair isn't terrible but also not quite as clean as straight apache

Llama's license explicitly disallows its usage in the EU.

If that doesn't even meet the threshold for "terrible", then what does?

Re: Mistral 3 family of models released

#150

Post by mrtksn »

acuozzo wrote: post #140
mrtksn wrote: post #69

Earlier quoted context omitted.

Some time ago I canceled all my paid subscriptions to chatbots because they are interchangeable so I just rotate between Grok, ChatGPT, Gemini, Deepseek and Mistral. On the API side of things my experience is that the model behaving as expected is the greatest feature. There I also switched to Openrouter instead of paying directly so I can use whatever model fits best. The recent buzz about ad-based chatbot services…

> because they are interchangeable What is your use-case? Mine is: I use "Pro"/"Max"/"DeepThink" models to iterate on novel cross-domain applications of existing mathematics. My interaction is: I craft a detailed prompt in my editor, hand it off, come back 20-30 minutes later, review the reply, and then repeat if necessary. My experience is that they're all very, very different from one another.

my use case is Google replacement, things that I can do by myself so I can verify and things that are not important so I don’t have to verify.

Sure, they produce different output so sometimes I will run the same thing on a few different models when Im not sure or happy but I’d don’t delegate the thinking part actually, I always give a direction in my prompts. I don’t see myself running 30min queries because I will never trust the output and will have to do all the work myself. Instead I like to go step by step together.

Post reply on HN