8x22b. If this is as good as Mixtral 8x7b we are in for a wonderful time.
I've heard command-r is first opensource to beat gpt4 in benchmarks
It's "Command R+". "Command R" is a smaller model.
Mistral AI Launches New 8x22B MOE Model
41–50 of 161 posts
Re: Mistral AI Launches New 8x22B MOE Model
#42The lack of a corresponding announcement on their blog makes me worry about a Twitter account compromise and a malicious model. Any way to verify it’s really from them?
Their https://twitter.com/MistralAI account has 5 tweets since the account opened, three of which were model release magnet links.
https://twitter.com/MistralAILabs is their other Twitter account, which is very slightly more useful though still very low traffic.
Re: Mistral AI Launches New 8x22B MOE Model
#43Earlier quoted context omitted.
For those of us without twitter, how many GB is the model?
262 gb
Ooof. I really need to pick up another HD, these model sizes are killer.
Lacking a godly GPU, I will probably hold off for a quanitized version which has the potential to run okish on CPU or my modest GPU, but really appreciate the info.
Re: Mistral AI Launches New 8x22B MOE Model
#444 bit quants should require 85GB VRAM, so this will fit nicely on 4x 24G consumer GPUs, plus some leftover for KV cache optimization.
I've found the 2 bit quant of Mixtral 8x7B is usable for some purposes with an 8GB GPU. I'm curious how this new model will work in similar cheap 8-16GB GPU configurations.
Re: Mistral AI Launches New 8x22B MOE Model
#45Re: Mistral AI Launches New 8x22B MOE Model
#46Earlier quoted context omitted.
It beats the old GPT4 version in lmsys benchmark you can check it out here https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... but Command R is commercially licensed We can assume that mistral will do a better job.
> but Command R is commercially licensed It is licensed under CC-BY-NC-4.0. That license means you are free to use, modify and redistribute it, so long as you aren't doing so "commercially". What exactly counts as "commercial" use is a complex legal question, and the answer may vary from jurisdiction to jurisdiction (different courts may interpret the phrase differently). But, for example, if you are just using it at…
I am not a lawyer, but lately, I have been wondering whether the contra proferentem rule interacts with these licenses.
Re: Mistral AI Launches New 8x22B MOE Model
#47To this day 8x7b Mixtral remains the best model you can run on a single 48GB GPU. This has the potential to become the best model you can run on two such GPUs, or on an MBP with maxed out RAM, when 4-bit quantized.
[deleted]
Re: Mistral AI Launches New 8x22B MOE Model
#48Earlier quoted context omitted.
> but Command R is commercially licensed It is licensed under CC-BY-NC-4.0. That license means you are free to use, modify and redistribute it, so long as you aren't doing so "commercially". What exactly counts as "commercial" use is a complex legal question, and the answer may vary from jurisdiction to jurisdiction (different courts may interpret the phrase differently). But, for example, if you are just using it at…
I am not a lawyer, but lately, I have been wondering whether the contra proferentem rule interacts with these licenses.
I think the correct answer to your question is almost certainly some combination of "it depends on the jurisdiction" and also (in many cases) "nobody can be entirely sure because no court has considered the issue yet"
There have been a handful of court decisions on what "non-commercial" use means – the Creative Commons legal case database records [0] records three cases involving non-commercial CC licenses in the US, one in Belgium, one in Israel, plus I also know of one in Germany [1] which their database seems to be missing. I don't know if any of them addressed the contra proferentem rule which you mention.
The German and US cases on this topic appear contradictory – from what I understand, the German case assumed that all government use is commercial, interpreting "non-commercial" to basically mean "private home use", whereas two of the US cases (Great Minds v FedEx Office and Great Minds v Office Depot) were about use by commercial entities acting under contract to public school districts, and the holdings of those cases assume that government-operated schools are "non-commercial" (and furthermore, the commercial entities were engaging in "non-commercial" use, even though they were acting commercially, because they were doing so on behalf of a "non-commercial" customer). That said, all these cases have somewhat limited precedential value – the US cases are binding precedent in two federal judicial circuits (2nd and 9th) but have merely persuasive value in the remainder of the US; I don't know what the ultimate outcome of the German case was (Deutschlandradio said they were going to appeal but I don't know if they did and what the outcome was if they did), and German law doesn't view precedent as "binding" in quite the same sense that common law systems do anyway
[0] https://legaldb.creativecommons.org/cases/?keywords=&tags%5B...
[1] https://www.techdirt.com/2014/03/27/german-court-says-creati... and if you can read German, here is the actual court judgement: https://netzpolitik.org/wp-upload/OLG-K%C3%B6ln-CC-NC-Entsch...
Re: Mistral AI Launches New 8x22B MOE Model
#49What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library? Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?
I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format). You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile? [0] https://github.com/Mozilla…
llamafile author here. I'm downloading Mixtral 8x22b right now. I can't say for certain it'll work until I try it, but let's keep our fingers crossed! If not, we'll be shipping a release as soon as possible that gets it working.
My recent work optimizing CPU evaluation https://justine.lol/matmul/ may have come at just the right time. Mixtral 8x7b always worked best at Q5_K_M and higher, which is 31GB. So unless you've got 4x GeForce RTX 4090's in your computer, CPU inference is going to be the best chance you've got at running 8x22b at top fidelity.
Re: Mistral AI Launches New 8x22B MOE Model
#50Why are some of their models open, and others closed? What is their strategy?
My personal speculation is that their closed models are based on other companies' models.
For example on EQbench[0], Miqu[1], a leaked continued pretrain based on LLama2, performs extremely similar to the mistral medium model their API offers.
Maybe they're thinking it'd be bad PR for them to release models they didn't create from scratch, or there is some contractual obligation preventing the release.