Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral Medium 3.5

mistral.ai

Post reply on HN
Print view View original

201–210 of 248 posts

Re: Mistral Medium 3.5

#201

Post by varispeed »

simjnd wrote: post #52

I'm not sure what people are on in the comments. It doesn't beat the other models, but it sure competes despite its size. GLM 5.1 is an excellent model, but even at Q4 you're looking at ~400GB. Kimi K2.5 is really good too, and at Q4 quantization you're looking at almost ~600GB. This model? You can run it at Q4 with 70GB of VRAM. This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM…

> It doesn't beat the other models, but it sure competes despite its size.

But what is the rationale for running a dumb model? Because it can ocasionally produce something passable?

I don't get where is the value apart from mild entertainment, as in "I am somewhat of Anthropic myself".

Re: Mistral Medium 3.5

#202

Post by varispeed »

WhitneyLand wrote: post #180
simjnd wrote: post #52

I'm not sure what people are on in the comments. It doesn't beat the other models, but it sure competes despite its size. GLM 5.1 is an excellent model, but even at Q4 you're looking at ~400GB. Kimi K2.5 is really good too, and at Q4 quantization you're looking at almost ~600GB. This model? You can run it at Q4 with 70GB of VRAM. This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM…

“This beats the latest Sonnet while running locally” Not really. - The benchmarks are based on F8_E4M3 and you’re not running that on any Mac. - Sonnet has a 1M token context window. This is 256k but again you’re probably not even getting that locally. - Sonnet is fast over the wire. This is going to be much slower.

Not sure if 1M token window is meaningful with Sonnet/Opus. The models go dumb quickly as context increases making them unusable (that is if you get routed to actual Opus, otherwise they are just dumb regardless of context window).

Re: Mistral Medium 3.5

#203

Post by schipperai »

ac29 wrote: post #128
schipperai wrote: post #106

With most OSS releases being MoEs, and modern GPUs optimized for MoEs, can somebody with knowledge of the topic explain or speculate why Mistral might have opted for a dense model?

Modern GPUs aren't optimized for MoEs though? The advantage to a dense model like this Mistral one is that it is as smart as a much larger MoE model so it can fit on less GPUs. The tradeoff is that it is much slower since it has to read 100% of its weights for every token, MoE models typically only read about a tenth (though sparsity levels vary).

Thanks, makes sense. I meant Blackwell is explicitly optimized for MoEs.

Re: Mistral Medium 3.5

#204

Post by hadlock »

sroussey wrote: post #190
simjnd wrote: post #122

Earlier quoted context omitted.

> The one thing I would want everyone curious about local LLMs to know is that being able to run a model and being able to run a model fast are two very different thresholds. You can get these models to run on a 128GB Mac, but we need to first tell if Q4 retains enough quality (models have different sensitivities to quantization) and how fast it runs. Very valid. This is an active area of research, and there are a lo…

Super interesting! > - People have successfully used TurboQuant to quantize model weights (TQ3_4S), not just the context KV, to achieve smaller sizes than Q4 (~3.5 bpw) with much better PPL and faster decoding. Where can I find more info on this? I’d like to convert models to onnx this way. > - Importance-weighted quantization (e.g. IQ4) also provides way better PPL, KDL, etc. at the same size as a Q4. Where can I fi…

Google only released their TurboQuant paper barely a month ago, it is bleeding edge even by LLM standards

Re: Mistral Medium 3.5

#205

Post by pbgcp2026 »

nijave wrote: post #191
Aurornis wrote: post #82

Earlier quoted context omitted.

> This model? You can run it at Q4 with 70GB of VRAM. This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM for ~3500 USD). The one thing I would want everyone curious about local LLMs to know is that being able to run a model and being able to run a model fast are two very different thresholds. You can get these models to run on a 128GB Mac, but we need to first tell if Q4 retains…

>Almost every open weight model launch this year has come with claims that it matches or exceeds Sonnet. I've been trying a lot of them and I have yet to see it in practice, even when the benchmarks show a clear lead. This has been my experience as well. I've been testing an agent built with Strands Agents which receives a load balancer latency alert and is expected to query logs with AWS Athena (Trino) then drill do…

" it supports prompt caching" May I ask if you checked that? I use "{"cachePoint": { "type": "default" }" and I found 2 things: * 1) even if stated in the Doco, Bedrock Converse API does not allow 1hr expiry time, only 5m - gives error when attempted; * 2) Bedrock Converse API does accept up to 4 cachePoint's but does NOT cache and returns zeroes. LOL. It was confirmed by some other people on Github. (Note: VertexAI does cache properly reducing the bill drastically, so I use Vertex instead of OpenRouter.)

Re: Mistral Medium 3.5

#206

Post by pbgcp2026 »

zozbot234 wrote: post #200
parsimo2010 wrote: post #196

Earlier quoted context omitted.

> being able to run a model and being able to run a model fast are two very different thresholds Specifically speaking, on my Strix Halo machine with (theoretical) memory bandwidth of 256 GB/s, a 70 GB model can't generate faster than 256/70= 3.65 t/s. The logic here is that a dense model must do a full read of the weights for each token. So even if the GPU can keep up, the memory bandwidth is limiting. A Mac M5 Pro…

Recent models support multi-token prediction, which can guess multiple future tokens in a single decode step (using some subset of the model itself, not a separate drafting model) and then verify them all at once. It's an emerging feature still (not widely supported) and it's only useful for speeding up highly predictable token runs, but it's one way to do better in practice than the common-sense theoretical limit mi…

It seems to me it's only Grok 4.20 that does this currently? Which other models did you have in mind, if I may ask?

Re: Mistral Medium 3.5

#207

Post by signatoremo »

argsnd wrote: post #51
deaux wrote: post #37

Earlier quoted context omitted.

Where are the competitive models from Singapore, Japan, Taiwan, Korea, Russia, Canada, India, the UK? From anywhere that isn't China or the US? There are none. Mistral Small 4 is pareto-competitive in its pricing bracket at $0.15/$0.60, at worst it's second to Gemma 4 26B A4B. The above countries have never had a model that is even close to being so. This particular Mistral Medium looks to be uncompetitive at that pr…

DeepMind, which is headquartered in London, probably had a significant role in the development of the Gemini and Gemma models. Yes, it might be a problem that the UK allows companies like this to be bought up by foreign countries.

Yet ASML is always cited as a great Europe great achievement, but it's hardly ever mentioned that without American's EUV research and patents, and without Cymer there would be no AMSL as we known of.

In all honesty I believe ASML's success is mostly their own. Still, lamenting "being bought up by foreign countries" is a lame excuse.

Re: Mistral Medium 3.5

#208

Post by trueno »

simjnd wrote: post #153
ksubedi wrote: post #142

Earlier quoted context omitted.

Let's not forget Qwen 35B A3B MoE. It gets better performance than this in all the metrics for a fraction of the memory / compute footprint. Sad to see all the non Chinese open source models being at least one generation behind.

Qwen3.6 27B is even more impressive IMO. Dense so it doesn't run as fast but it's so good.

im kinda torn on which to download. i have the headroom to run either, mostly just want the occasional "do a coding thing im too lazy to do"

Re: Mistral Medium 3.5

#209

Post by sroussey »

hadlock wrote: post #204
sroussey wrote: post #190

Earlier quoted context omitted.

Super interesting! > - People have successfully used TurboQuant to quantize model weights (TQ3_4S), not just the context KV, to achieve smaller sizes than Q4 (~3.5 bpw) with much better PPL and faster decoding. Where can I find more info on this? I’d like to convert models to onnx this way. > - Importance-weighted quantization (e.g. IQ4) also provides way better PPL, KDL, etc. at the same size as a Q4. Where can I fi…

Google only released their TurboQuant paper barely a month ago, it is bleeding edge even by LLM standards

Actually, they published a year ago. Recent was being on official Google blog.

https://arxiv.org/abs/2504.19874

https://research.google/blog/turboquant-redefining-ai-effici...

Re: Mistral Medium 3.5

#210

Post by notatoad »

Aurornis wrote: post #82
simjnd wrote: post #52

I'm not sure what people are on in the comments. It doesn't beat the other models, but it sure competes despite its size. GLM 5.1 is an excellent model, but even at Q4 you're looking at ~400GB. Kimi K2.5 is really good too, and at Q4 quantization you're looking at almost ~600GB. This model? You can run it at Q4 with 70GB of VRAM. This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM…

> This model? You can run it at Q4 with 70GB of VRAM. This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM for ~3500 USD). The one thing I would want everyone curious about local LLMs to know is that being able to run a model and being able to run a model fast are two very different thresholds. You can get these models to run on a 128GB Mac, but we need to first tell if Q4 retains…

being able to run a model fast is definitely more useful, but being able to run a model slowly for free is still super useful. agentic workflows are maturing all the time.

yes, if i'm directly interacting with the LLM, i want it to be reasonably fast. but lately i've been queueing up a bunch of things when i go for lunch, or leaving things running when i go home at the end of the day. and claude doesn't keep working on that all night, it runs for an hour or so, gets to a point where it needs more input from me, and gives me some stuff to review in the morning. that could run 16x slower and still be just as useful for me.

Post reply on HN