Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral ships Le Chat – enterprise AI assistant that can run on prem

mistral.ai

Post reply on HN
Print view View original

81–90 of 166 posts

Re: Mistral ships Le Chat – enterprise AI assistant that can run on prem

#81

Post by mistrial9 »

jstummbillig wrote: post #59
fhd2 wrote: post #21

Earlier quoted context omitted.

You might be able to get your clients to sign something to allow usage, but if you don't, as you say, it doesn't seem wise to vibe code for them. For two reasons: 1. A typical contract transfers the rights to the work. The ownership of AI generated code is legally a wee bit disputed. If you modify and refactor generated code heavily it's probably fine, but if you just accept AI generated code en masse, making your cl…

Nobody is seriously disputing the ownership of AI generated code. A serious dispute would be a considerable, concerted effort to stop AI code generation in any jurisdiction, that provides a contrast to the enormous , ongoing efforts by multiple large players with eye-watering investments to make code generation bigger and better. Note, that this is not a statement about the fairness or morality of LLM building, but t…

this is "Kool-aid" from the supply side of LLMs for coding IMO. Plenty of people are plenty upset about the capture of code at Github corral, fed into BigCorp$ training systems.

parent statement reminds me of smug French in a castle north of London circa 1200, with furious locals standing outside the gates, dressed in rags with farm tools as weapons. One well-equipped tower guard says to another "no one is seriously disputing the administration of these lands"

Re: Mistral ships Le Chat – enterprise AI assistant that can run on prem

#82

Post by tootie »

starik36 wrote: post #66

I don't see any mention of hardware requirements for on prem. What GPUs? How many? Disk space?

I'm guessing it's flexible. Mistral makes small models capable of running on consumer hardware so they can probably scale up and down based on needs. And what is available from hosts.

Re: Mistral ships Le Chat – enterprise AI assistant that can run on prem

#83

Post by tootie »

iamnotagenius wrote: post #11

Mistral models though are not interesting as models. Context handling is weak, language is dry, coding mediocre; not sure why would anyone chose it over Chinese (Qwen, GLM, Deepseek) or American models (Gemma, Command A, Llama).

I flip back and forth with Claude and Le Chat and find them comparable. Le Chat always feels very quick and concise. That's just vibes not benchmarks.

Re: Mistral ships Le Chat – enterprise AI assistant that can run on prem

#84

Post by codetrotter »

simonw wrote: post #69
indigodaddy wrote: post #68

Earlier quoted context omitted.

Simon, can you recommend some small models that would be usable for coding on a standard M4 Mac Mini (only 16G ram) ?

That's pretty tough - the problem is that you need to have RAM left over to run actual applications! Qwen 3 8B on MLX runs in just 5GB of RAM and can write basic code but I don't know if it would be good enough for anything interesting: https://simonwillison.net/2025/May/2/qwen3-8b/ Honestly though with that little memory I'd stick to running against hosted LLMs - Claude 3.7 Sonnet, Gemini 2.5 Pro, o4-mini are all ch…

How about on an MacBook Pro M2 Max with 64GB RAM? Any recommendations for local models for coding on that?

I tried to run some of the differently sized DeepSeek R1 locally when those had recently come out, but couldn’t manage at the time to run any of them. And I had to download a lot of data to try those. So if you know a specific size of DeepSeek R1 that will work on 64GB RAM on MacBook Pro M2 Max, or another great local LLM for coding on that, that would be super appreciated

Re: Mistral ships Le Chat – enterprise AI assistant that can run on prem

#85

Post by jychang »

indigodaddy wrote: post #68
simonw wrote: post #58

Earlier quoted context omitted.

There are plenty of other ways to run Mistral models on a Mac. I'm a big fan of Mistral Small 3.1. I've run that using both Ollama (easiest) and MLX. Here are the Ollama models: https://ollama.com/library/mistral-small3.1/tags - the 15GB one works fine. For MLX https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-I... and https://huggingface.co/mlx-community/Mistral-Small-3.1-24B-I... should work, I use the 8bi…

Simon, can you recommend some small models that would be usable for coding on a standard M4 Mac Mini (only 16G ram) ?

16GB on a mac with unified memory is too small for good coding models. Anything on that machine is severely compromised. Maybe in ~1 year we will see better models that fit in ~8gb vram, but not yet.

Right now, for a coding LLM on a Mac, the standard is Qwen 3 32b, which runs great on any M1 mac with 32gb memory or better. Qwen 3 235b is better, but fewer people have 128gb memory.

Anything smaller than 32b, you start seeing a big drop off in quality. Qwen 3 14b Q4_K_M is probably your best option at 16gb memory, but it's significantly worse in quality than 32b.

Re: Mistral ships Le Chat – enterprise AI assistant that can run on prem

#86

Post by _bin_ »

v3ss0n wrote: post #55
codingbot3000 wrote: post #3

I think this is a game changer, because data privacy is a legitimate concern for many enterprise users. Btw, you can also run Mistral locally within the Docker model runner on a Mac.

What's the point when we can run much powerful models now? Qwen3 , Deepseek

It would be short-termist for Americans or euros to use chinese-made models. Increasing their popularity has an indirect but significant cost in the long term. china "winning AI" should be an unacceptable outcome for America or europe by any means necessary.

Re: Mistral ships Le Chat – enterprise AI assistant that can run on prem

#87

Post by freeqaz »

codetrotter wrote: post #84
simonw wrote: post #69

Earlier quoted context omitted.

That's pretty tough - the problem is that you need to have RAM left over to run actual applications! Qwen 3 8B on MLX runs in just 5GB of RAM and can write basic code but I don't know if it would be good enough for anything interesting: https://simonwillison.net/2025/May/2/qwen3-8b/ Honestly though with that little memory I'd stick to running against hosted LLMs - Claude 3.7 Sonnet, Gemini 2.5 Pro, o4-mini are all ch…

How about on an MacBook Pro M2 Max with 64GB RAM? Any recommendations for local models for coding on that? I tried to run some of the differently sized DeepSeek R1 locally when those had recently come out, but couldn’t manage at the time to run any of them. And I had to download a lot of data to try those. So if you know a specific size of DeepSeek R1 that will work on 64GB RAM on MacBook Pro M2 Max, or another great…

I imagine that this in quantized form would fit pretty well and be decent. (Qwen R1 32b[1] or Qwen 3 32b[2])

Specifically the `Q6_K` quant looks solid at ~27gb. That leaves enough headroom on your 64gb Macbook that you can actually load a decent amount of context. (It takes extra VRAM for every token of context you need)

Rough math, based on this[0] calculator is that it's around ~10gb per 32k tokens of context. And that doesn't seem to change based on using a different quant size -- you just have to have enough headroom.

So with 64gb:

- ~25gb for Q6 quant

- 10-20gb for context of 32-64k

That leaves you around 20gb for application memory and _probably_ enough context to actually be useful for larger coding tasks! (It just might be slow, but you can use a smaller quant to get more speed.)

I hope that helps!

0: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul...

1: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32...

2: https://huggingface.co/Qwen/Qwen3-32B-GGUF

Re: Mistral ships Le Chat – enterprise AI assistant that can run on prem

#89

Post by simonw »

codetrotter wrote: post #84
simonw wrote: post #69

Earlier quoted context omitted.

That's pretty tough - the problem is that you need to have RAM left over to run actual applications! Qwen 3 8B on MLX runs in just 5GB of RAM and can write basic code but I don't know if it would be good enough for anything interesting: https://simonwillison.net/2025/May/2/qwen3-8b/ Honestly though with that little memory I'd stick to running against hosted LLMs - Claude 3.7 Sonnet, Gemini 2.5 Pro, o4-mini are all ch…

How about on an MacBook Pro M2 Max with 64GB RAM? Any recommendations for local models for coding on that? I tried to run some of the differently sized DeepSeek R1 locally when those had recently come out, but couldn’t manage at the time to run any of them. And I had to download a lot of data to try those. So if you know a specific size of DeepSeek R1 that will work on 64GB RAM on MacBook Pro M2 Max, or another great…

I really like Mistral Small 3.1 (I have a 64GB M2 as well). Qwen 3 is worth trying in different sizes too.

I don't know if they'll be good enough for general coding tasks though - I've been spoiled by API access to Claude 3.7 Sonnet and o4-mini and Gemini 2.5 Pro.

Re: Mistral ships Le Chat – enterprise AI assistant that can run on prem

#90

Post by the_clarence »

_pdp_ wrote: post #10

While I am rooting for Mistral, having access to a diverse set of models is the killer app IMHO. Sometimes you want to code. Sometimes you want to write. Not all models are made equal.

Tbh I think the one general model approach is winning. People don't want to figure out which model is better at what unless its for a very specific task.
Post reply on HN