161–170 of 243 posts
Earlier quoted context omitted.
How is there such a gap between Gemini 3 vs GPT 5.1/Opus 4.5? What is Gemini 3 crushing the others on?
Could be optimized for benchmarks, but Gemini 3 has been stellar for my tasks so far. Maybe an architectural leap?
Earlier quoted context omitted.
I think people from the US often aren't aware how many companies from the EU simply won't risk losing their data to the providers you have in mind, OpenAI, Anthropic and Google. They simply are no option at all. The company I work for for example, a mid-sized tech business, currently investigates their local hosting options for LLMs. So Mistral certainly will be an option, among the Qwen familiy and Deepseek. Mistral…
Mistral is founded by multiple Meta engineers, no? Funded mostly by US VCs? Hosted primarily on Azure? Do you really have to go out of your way to start calling their competition "data leeches" for out-executing them?
The 3B vision model runs in the browser (after a 3GB model download). There's a very cool demo of that here: https://huggingface.co/spaces/mistralai/Ministral_3B_WebGPU Pelicans are OK but not earth-shattering: https://simonwillison.net/2025/Dec/2/introducing-mistral-3/
Earlier quoted context omitted.
And implicit in this is that it compares very poorly to SOTA models. Do you disagree with that? Do you think these Models are beating SOTA and they did not include the benchmarks, because they forgot?
> Do you disagree with that? I think that Qwen3 8B and 4B are SOTA for their size. The GPQA Diamond accuracy chart is weird: Both Qwen3 8B and 4B have higher scores, so they used this weid chart where "x" axis shows the number of output tokens. I missed the point of this.
Earlier quoted context omitted.
Mistral is founded by multiple Meta engineers, no? Funded mostly by US VCs? Hosted primarily on Azure? Do you really have to go out of your way to start calling their competition "data leeches" for out-executing them?
Mistral are mostly focusing on b2b, and for customers that want to self-host (banks and stuff). So their founders being from Meta, or where their cloud platform are hosted, are entirely irrelevant to the story.
Pan-nationalism is a hell of a drug: a company that does not know you exist puts out an objectively awful release, and people take frank discussion of it as a personal slight.
The new large model uses DeepseekV2 architecture. 0 mention on the page lol. It's a good thing that open source models use the best arch available. K2 does the same but at least mentions "Kimi K2 was designed to further scale up Moonlight, which employs an architecture similar to DeepSeek-V3". --- vllm/model_executor/models/mistral_large_3.py ``` from vllm.model_executor.models.deepseek_v2 import DeepseekV3ForCausalL…
Earlier quoted context omitted.
What's your acceptable error rate? Honestly ministral would probably be sufficient if you can tolerate a small failure rate. I feel like medium would be overkill. But I'm no expert. I can't say I've used mistral much outside of my own domain.
I'd prefer for the error rate to be as close to 0% as possible under the strict requirement of having to use a local model. I have access to nodes with 8xH200, but I'd prefer to not tie those up with this task. I'd, instead, prefer to use a model I can run on an M2 Ultra.
But in that case the larger the better. If mistral medium can run on your M2 Ultra then it should be up to the task. Should eek out ministral and be just shy of the biggest frontier models.
But I wouldn’t even trust GPT-5 or Claude Opus or Gemini 3 Pro to get close to a zero percent success rate, and for a task such as this I would not expect mistral medium to outperform the big boys
Earlier quoted context omitted.
I think people from the US often aren't aware how many companies from the EU simply won't risk losing their data to the providers you have in mind, OpenAI, Anthropic and Google. They simply are no option at all. The company I work for for example, a mid-sized tech business, currently investigates their local hosting options for LLMs. So Mistral certainly will be an option, among the Qwen familiy and Deepseek. Mistral…
Mistral is founded by multiple Meta engineers, no? Funded mostly by US VCs? Hosted primarily on Azure? Do you really have to go out of your way to start calling their competition "data leeches" for out-executing them?
I mean why do you think those guys left Meta? It reminds me of a time ten years ago I was sitting on a flight with a guy who works for the natural gas industry. I was (cough still am) a pretty naive environmentalist, so I asked him what he thought of solar, wind, etc. and why should we be investing in natural gas when there are all these other options. His response was simple. Natural gas can serve as a bridge from hydrocarbons to true green energy sources. Leverage that dense energy to springboard the other sources in the mix and you build a path forward to carbon free energy.
I see Mistral's use of US VCs the same way. Those VCs are hedging their bets and maybe hoping to make a few bucks. A few of them are probably involved because they're buddies with the former Meta guys "back in the day." If Mistral executes on their plan of being a transparent b2b option with solid data protections then they used those VCs the way they deserve to be used and the VCs make a few bucks. If Europe ever catches up to the US in terms of data centers, would Mistral move off of Azure? I'd bet $5 that they would.
Earlier quoted context omitted.
The best benchmark is one that you build for your use-case. I finally did that for a project and I was not expecting the results. Frontier models are generally "good enough" for most use-cases but if you have something specific you're optimizing for there's probably a more obscure model that just does a better job.
If you and others have any insights to share on structuring that benchmark, I'm all ears. There a new model seemingly every week so finding a way to evaluate them repeatedly would be nice. The answer may be that it's so bespoke you have to handroll every time, but my gut says there's a set of best practiced that are generally applicable.
1. Sample a set of prompts / answers from historical usage.
2. Run that through various frontier models again and if they don't agree on some answers, hand-pick what you're looking for.
3. Test different models using OpenRouter and score each along cost / speed / accuracy dimensions against your test set.
4. Analyze the results and pick the best, then prompt-optimize to make it even better. Repeat as needed.