https://huggingface.co/someone13574/mixtral-8x7b-32kseqlen
[dead]
Mistral "Mixtral" 8x7B 32k model [magnet]
241–250 of 255 posts
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#242Earlier quoted context omitted.
It's actually ~13B class at runtime. 2B for attention is shared across each expert and then it runs 2 experts at a time. So 2B for attention + 5Bx2 for inference = 12B in RAM at runtime.
Yeah. I just mean in terms of VRAM usage.
Yes, that's what I mean as well.
It's between 7B and 13B in terms of VRAM usage and 70B in terms of performance.
Tim Dettmers (QLoRA creator) released code to run Mixtral 8x7b in 4GB of VRAM. (But it benchmarks better than Llama-2 70B).
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#243Earlier quoted context omitted.
Given the config parametes posted, its 2 experts per token, so the conputation cost per token should be the cost of the conponent that selects experts + 2× cost of a 7B model.
Yes, but I also care about "can I load this onto my home GPU?" where, if I need all experts for this to run, the answer is "no".
The answer is yes if you have a 24GB GPU. Just wait for 4bit quantization.
Or watch Tim Dettmers, who is releasing code to run Mixtral 8x7b in just 4GB of RAM.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#244Earlier quoted context omitted.
Aren't LLM benchmarks at best irrelevant, at worst lying, at this point?
If you don't like machine evaluations, you can take a look at the lmsys chatbot arena. You give a prompt, two chatbots answer anonymously, and you pick which answer is better: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... On the human ratings, three different 7B LLMs (Two different Openchat models and a Mistral fine tune) beat a version of GPT-3.5. (The top 9 chatbots are GPT and Claude versions. Te…
Pretty sure openchat-3.5 is a mistral fine tune as well.
The trick is not this neural alignment - it is training on many, many more tokens than Chinchilla recommends.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#245In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…
DPO is pretty good as well.
I think that the '7b beating 70b' is mostly due to the fact that Mistral is likely trained on considerably more tokens than Chinchilla optimal. So is llama-70b, but not to the same degree.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#246Mistral sure does not bother too much with explanations, but this style gives me much more confidence in the product than Google's polished, corporate, soulless announcement of Gemini!
I will take weights over docs. Its does remind me how some Google employee was bragging that they disclosed the weights for the Gemini, and only the small mobile Gemini, as if that's a generous step over other companies.
they did not disclose the weights for any gemini, you must have misunderstood
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#247Earlier quoted context omitted.
My years-old M1 macbook with 16GB of ram runs them just fine. Several Geforce 40-series cards have at least 16GB of vram. Macbook pros go up to 128GB of ram and the mac studio goes up to 192GB. Running regular CPU inference on lots of system ram is cheap-ish and not intolerably slow. These aren't totally common configurations, but they're not totally out of reach like buying an H100 for personal use.
1. I wouldn't consider Mac Studio ($7,000) a customer product. 2. Yes, and my MBP M1 Pro can run quantized 34b models. My point was that when you do MoE, memory requirements suddenly become too challenging. A 7b Q8 is roughly 7GB (7b parameters × 8 bits each). But 8x of that would be 56GB, and all of that must be in memory to run.
Why? $7k for a Mac Studio isn't much if we consider the original IBM PC adjusted for inflation cost $6k.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#248Earlier quoted context omitted.
Google and Facebook were, up until just a couple of days ago, in a cold war within the Canadian government.
Google made a deal to pay 100M/year to news organizations in Canada but Meta is continuing to block news links.
You'd think Google would devote its resources to improve search, but instead they're paying journalists (who hate tech) to forge the very chains that bind us.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#249Earlier quoted context omitted.
How so? I'm only getting 12 t/s using Mistral in LM Studio.
The lazy way is to use text-generation-webui, use an exllamav2 conversion of your model, and turn down context length until it fits (and tick the 8 bit cache option). If you go over your vram it will cut your speed substantially. Like 60/s down to 15/s for an extra 500 context length over what fits. Similar idea applies to any other backends, but you need to shove all the layers into vram if you want decent tok/s. To…
Just tried it - doesn't seem to be working. In fact, I'm getting 1.4 t/s with a Quadro P4000 (8 GB) running a 7B at 3 bits per weight. Are you changing anything other than the 8 bit cache and context?
For reference, I'm getting 10 t/s with a Q5_K_M Mistral GGUF model.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#250Kinda following all this stuff from outside w/o really understanding, but why are these things released like this, instead of "competing ChatGPTs apps" with higher and higher quality/costs? Could be open sourced but also hosted version that is maybe 5 usd/minute - if the results are great I guess people would pay the fair price...
Is it mainly because its hard to apply the limitations so that it doesn't spit out bomb making instructions?