Mistral is playing the long game here ngl, lower sized models, lower costs, overall good enough performance!
Mistral Medium 3.5
221–230 of 248 posts
Re: Mistral Medium 3.5
#222Earlier quoted context omitted.
>This model? You can run it at Q4 with 70GB of VRAM. >This beats the latest Sonnet while running locally Not sure it will beat Sonet at Q4. >This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM for ~3500 USD). For $3500 I can get 7-8 years of GLM using coding plans, have a faster model and much better code quality.
> Not sure it will beat Sonet at Q4. Very valid. Importance-weighted quantization and TurboQuant on model weights can reduce loss a lot compared to "traditional" Q4 so one can be hopeful. > For $3500 I can get 7-8 years of GLM using coding plans, have a faster model and much better code quality But you will own no computer, and that's also assuming prices stay what they are. Anyway my point was not whether or not it…
@simjnd, I hate this idea but you remember how radio had been regulated to death? And how fast one will be triangulated if one decides to run a "self hosted" radio station today? My bet is in 5 years not only owning AI-inference-capable computer but using AI itself will be regulated. Essentially, we will have to scan biometrics to just ask any SOTA model to "summarise this".
Why? Because capable and free models at the dawn of AI almost made people think again and - oh oh - ask questions!
Re: Mistral Medium 3.5
#223Earlier quoted context omitted.
> For $3500 I can get 7-8 years of GLM mind sharing where's the go to place to pay for open models?
I recommend using OpenRouter (openrouter.ai). Basically a broker between inference providers and you which allows you to pick, try, and switch models from a massive catalog, extremely transparent about usage and pricing.
+5% to every API call.
Re: Mistral Medium 3.5
#224Earlier quoted context omitted.
I don't think any models are natively INT4? I wouldn't see the point to nerf the model out-of-the-box.
It's not nerfed, it's natively trained at that quantization a.k.a. Quantization Aware Training.
QAT typically uses BF16/FP32 during the training process to simulate lower precision.
Re: Mistral Medium 3.5
#225Earlier quoted context omitted.
DeepMind, which is headquartered in London, probably had a significant role in the development of the Gemini and Gemma models. Yes, it might be a problem that the UK allows companies like this to be bought up by foreign countries.
Without Google’s funding its not obvious i DeepMind would have went anywhere. Unless the moved to US for funding while keeping a back office in the UK. It’s strange to expect anything significant to come out from Europe when VCs there are either very risk averse and/or don’t have enough cash to begin with. It’s not like government or EU funding can replace that since its almost always wasted or missdirected
I have no idea why @wasfgwp is downvoted - it's very true. +1 on that.
Re: Mistral Medium 3.5
#226I'm using mistral-medium-2508 for some text transformation operations. It's giving me better results than mistral-large for my use cases. Looking forward to testing this new model, although I'm not sure if it's really meant at replacing the previous medium model since it's a lot more expensive and presented more as a coding / agentic model (mistral-medium-2508 was priced $0.4/$2 per 1M tokens, mistral-medium-3.5 is $…
I actually use Mistral Large to go through some large text chunks (in production). It gives about the same level of results as Sonnet, while being 90% cheaper. Definitely wouldn't use it for coding, but for this text-analyzing task it has been great. Much better than all the latest Chinese models, for example. So I was waiting for this release and it's... 5x more expensive than the latest Mistral Large. So now I'm wo…
Yeah I use Mistral Large for a lot of formatting work. For this one use case of mine, it outperforms frontier models by a significant margin. I've found tons of use cases for mistral small as well.
I'd love to use Mistral for more tasks, but Mistral Large doesn't quite cut it for all tasks. So on the one hand, I'm excited there is another model, and presumably more performant based on the price? But the fact it's a "Medium" and 5x the price of the Large definitely concerns me.
The entire release is also about Vibe Coding, and so I'm not even sure if this model is applicable outside of coding, or even worth testing.
Re: Mistral Medium 3.5
#227I'm not sure what people are on in the comments. It doesn't beat the other models, but it sure competes despite its size. GLM 5.1 is an excellent model, but even at Q4 you're looking at ~400GB. Kimi K2.5 is really good too, and at Q4 quantization you're looking at almost ~600GB. This model? You can run it at Q4 with 70GB of VRAM. This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM…
> It doesn't beat the other models, but it sure competes despite its size. But what is the rationale for running a dumb model? Because it can ocasionally produce something passable? I don't get where is the value apart from mild entertainment, as in "I am somewhat of Anthropic myself".
Are you dumb because you're not Einstein? Intelligence is a spectrum. Just because you're not #1 doesn't mean you're dumb. A lot of small models are not frontier but are still very competent and are very useful coding agent. It may take better prompting and more guiding, but that can be a reasonable tradeoff for some people.
Re: Mistral Medium 3.5
#228Earlier quoted context omitted.
> There is no "original model", the model is generated using Quantization Aware Training (QAT). The original model is the model used for the benchmarks People will say "You can run it locally!" then show the benchmarks of the original model, but what they really mean is that you can run a heavily quantized adaptation of the model which has difference performance characteristics.
That remark was specific to newer models like Kimi 2.x and DeepSeek V4 series, and this is clearly stated in my comment. As for other models, we quantize them because we are generally constrained by the model's total footprint in bytes, and running a larger model that's been quantized to fit in the same footprint as a smaller one improves performance compared to a smaller original, generally up to Q4 or so, with even…
When you say DeepSeek v4... you do realise it is a 1.6T param model right?
What kind of consumer hardware can run it reasonably in your mind?
Re: Mistral Medium 3.5
#229Earlier quoted context omitted.
This is the bar for Europe, huh?
Where are the competitive models from Singapore, Japan, Taiwan, Korea, Russia, Canada, India, the UK? From anywhere that isn't China or the US? There are none. Mistral Small 4 is pareto-competitive in its pricing bracket at $0.15/$0.60, at worst it's second to Gemma 4 26B A4B. The above countries have never had a model that is even close to being so. This particular Mistral Medium looks to be uncompetitive at that pr…
[deleted]
Re: Mistral Medium 3.5
#230Earlier quoted context omitted.
> The one thing I would want everyone curious about local LLMs to know is that being able to run a model and being able to run a model fast are two very different thresholds. You can get these models to run on a 128GB Mac, but we need to first tell if Q4 retains enough quality (models have different sensitivities to quantization) and how fast it runs. Very valid. This is an active area of research, and there are a lo…
Super interesting! > - People have successfully used TurboQuant to quantize model weights (TQ3_4S), not just the context KV, to achieve smaller sizes than Q4 (~3.5 bpw) with much better PPL and faster decoding. Where can I find more info on this? I’d like to convert models to onnx this way. > - Importance-weighted quantization (e.g. IQ4) also provides way better PPL, KDL, etc. at the same size as a Q4. Where can I fi…
For TurboQuant on model weights AFAIK it's currently a single person effort [1]. It needs his fork of llama.cpp, hasn't been upstreamed. He publishes his quantizations on HuggingFace but I'm not sure if he open-sourced the quantization pipeline.