Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral Medium 3.5

mistral.ai

Post reply on HN
Print view View original

231–240 of 248 posts

Re: Mistral Medium 3.5

#231

Post by simjnd »

tredre3 wrote: post #215
simjnd wrote: post #122

Earlier quoted context omitted.

> The one thing I would want everyone curious about local LLMs to know is that being able to run a model and being able to run a model fast are two very different thresholds. You can get these models to run on a 128GB Mac, but we need to first tell if Q4 retains enough quality (models have different sensitivities to quantization) and how fast it runs. Very valid. This is an active area of research, and there are a lo…

> - Importance-weighted quantization (e.g. IQ4) also provides way better PPL, KDL, etc. at the same size as a Q4. All the Q quants from big quant providers are importance-weighted (imatrix) nowadays. The main (possibly only?) difference between Q and IQ today is that IQ uses a lookup table to achieve better compression. That is also why IQ suffers more when it can't fully fit into VRAM. It's important to teach people…

Thanks for bringing this up I looked into it, and if I understood correctly:

- Q4_0 (not K quant) is the traditional flat quantization - Q4_K (4-bit K quant) uses an imatrix and important weights get higher precision (5-6 bits instead of 4, but still largely 4 bits) - IQ4 uses an imatrix and important weights get an optimized scale to avoid clipping at 4-bit, but all the weights are still 4-bit

And yeah most quants nowadays are K quants which are importance weighted

Re: Mistral Medium 3.5

#232

Post by Giorgi »

kergonath wrote: post #136
Giorgi wrote: post #50

Oh they are still a thing?! Completely forgot about Mistral. I am assuming they are still burning trough investor money.

> they are still burning trough investor money Difficult to say, this information is not really public. That said, those investors include EU agencies and European multinational companies and governments. It’s not as flashy as the ridiculous sums OpenAI is getting but it should be enough to keep them going for a while. They also have a different business model. They are selling their expertise to fine tune and adapt…

Ooh, ok so people got all worked up because it is EU vs USA thing.

Re: Mistral Medium 3.5

#233

Post by nijave »

pbgcp2026 wrote: post #205
nijave wrote: post #191

Earlier quoted context omitted.

>Almost every open weight model launch this year has come with claims that it matches or exceeds Sonnet. I've been trying a lot of them and I have yet to see it in practice, even when the benchmarks show a clear lead. This has been my experience as well. I've been testing an agent built with Strands Agents which receives a load balancer latency alert and is expected to query logs with AWS Athena (Trino) then drill do…

" it supports prompt caching" May I ask if you checked that? I use "{"cachePoint": { "type": "default" }" and I found 2 things: * 1) even if stated in the Doco, Bedrock Converse API does not allow 1hr expiry time, only 5m - gives error when attempted; * 2) Bedrock Converse API does accept up to 4 cachePoint's but does NOT cache and returns zeroes. LOL. It was confirmed by some other people on Github. (Note: VertexAI…

I had Claude Code pull the OTEL trace and calculate cost based on token counts in the responses. I'll double check later today tho if I remember

Edit: I do see the first request shows 0 cache read, 7k cache write tokens. The next request shows 7k cache read, 900 cache write tokens. The agent run summary is:

usage {

cache_read_input_tokens 244586

cache_write_input_tokens 38399

completion_tokens 8131

input_tokens 1172

output_tokens 8131

prompt_tokens 1172

total_tokens 292288

}

I do see a recent issue in the Strands Agent issue tracker about 1hr TTL getting ignored and defaulting to 5m TTL. I haven't validated cache TTL but these agent runs take ~2-3m so a 5m TTL is sufficient.

I also checked the AWS bill and see separate Usage SKUs

USE1-MP:USE1_CacheWriteInputTokenCount-Units $0.34

USE1-MP:USE1_OutputTokenCount-Units $0.27

USE1-MP:USE1_CacheReadInputTokenCount-Units $0.16

USE1-MP:USE1_InputTokenCount-Units $0.01

Re: Mistral Medium 3.5

#234

Post by parsimo2010 »

zozbot234 wrote: post #200
parsimo2010 wrote: post #196

Earlier quoted context omitted.

> being able to run a model and being able to run a model fast are two very different thresholds Specifically speaking, on my Strix Halo machine with (theoretical) memory bandwidth of 256 GB/s, a 70 GB model can't generate faster than 256/70= 3.65 t/s. The logic here is that a dense model must do a full read of the weights for each token. So even if the GPU can keep up, the memory bandwidth is limiting. A Mac M5 Pro…

Recent models support multi-token prediction, which can guess multiple future tokens in a single decode step (using some subset of the model itself, not a separate drafting model) and then verify them all at once. It's an emerging feature still (not widely supported) and it's only useful for speeding up highly predictable token runs, but it's one way to do better in practice than the common-sense theoretical limit mi…

If Mistral Medium 3.5 supports it, that might get it to 10 t/s. It will still be fairly slow.

Re: Mistral Medium 3.5

#235

Post by Ancapistani »

DeathArrow wrote: post #92
simjnd wrote: post #52

I'm not sure what people are on in the comments. It doesn't beat the other models, but it sure competes despite its size. GLM 5.1 is an excellent model, but even at Q4 you're looking at ~400GB. Kimi K2.5 is really good too, and at Q4 quantization you're looking at almost ~600GB. This model? You can run it at Q4 with 70GB of VRAM. This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM…

>This model? You can run it at Q4 with 70GB of VRAM. >This beats the latest Sonnet while running locally Not sure it will beat Sonet at Q4. >This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM for ~3500 USD). For $3500 I can get 7-8 years of GLM using coding plans, have a faster model and much better code quality.

> For $3500 I can get 7-8 years of GLM using coding plans, have a faster model and much better code quality.

I know HN's distaste for crypto, but I do my inference (for personal stuff - not my employer) through Venice. I was in the airdrop for VVV, and kept as much of it staked as I could. I have ~$40/day in inference as long as that service lasts.

These days the multiplier is about 1000x last I checked; if you want $10/day in inference and can lock up $10k in VVV, you get ~$10/day in inference plus (currently) ~16% APY in the form of more VVV.

I'm not sure I'd want to invest that much if I had to today, but it's a reasonable option. The risk of VVV going to $0 seems pretty small to me.

Re: Mistral Medium 3.5

#237

Post by ffsm8 »

Mashimo wrote: post #61

Compared to all other hosted LLMs that I have tested, Mistral seems to be the only one with rather strict CSP headers. When you ask them to create a website with some javascript library it will not preview, even though le chat offers canvas mode. Sometimes when a new release comes around from any provider I just want to test it a bit on the web. without paying and using an agent harness. Why are they like this ;_; Ed…

Are those cherries overlayed ontop of boobs with a bus to the side, driving towards a rock? Scnr

Re: Mistral Medium 3.5

#239

Post by EntityDeletr »

trueno wrote: post #208
simjnd wrote: post #153

Earlier quoted context omitted.

Qwen3.6 27B is even more impressive IMO. Dense so it doesn't run as fast but it's so good.

im kinda torn on which to download. i have the headroom to run either, mostly just want the occasional "do a coding thing im too lazy to do"

Then go with Qwen3.6 35B A3B. It's way faster (up to 5x) and it is 80% as capable as the 27B. The 27B is for serious people looking for one shot coding. The 35B is for iterative and quicker coding. I am in the same situation as you (making something I don't want to do myself) and I use the 35B at Q5_K_M.

Re: Mistral Medium 3.5

#240

Post by EntityDeletr »

pbgcp2026 wrote: post #220
simjnd wrote: post #124

Earlier quoted context omitted.

DeepSeek v4 Flash is still over 100GB at Q4 IIRC, and Q4 has generally been the sweet spot. Although it's an MoE so it might run a lot faster that this dense Mistral model if you have the RAM.

"Q4 has generally been the sweet spot" for self-hosting, yes. For any real meaningful work it's dumb AF. The only way to get reasonable intelligence from mid-size Gemma or Qwen is to run full precision BF16. Anything else is just an emulation of AI.

I would disagree. I have 8 GB of VRAM and 32 GB of RAM. I can either run a 4B BF16 dense model fully on GPU at around 30 t/s or Qwen3.6 35B A3B Q5_K_M at 20 t/s with GPU offload. Which one would I choose?
Post reply on HN