No indication that this a MoE (Mistral not Mixtral). Very exciting nevertheless, here’s hoping the bless the OS community once again!
It is indicated by the magnet link which they have posted on twitter (see comments above).
Mistral AI launches Mixtral-Next
51–55 of 55 posts
Re: Mistral AI launches Mixtral-Next
#52Earlier quoted context omitted.
The clincher for me will be if the API to it is not so exorbitantly priced as GPT4, and if mistral can make using LoRAs economical.
Curious what your use case is that API tokens are actually significant expense at this time. I get it if you’re reselling a product but myself: abusively and naively messing around, I don’t see major prices as an end user. I sometimes hit the rate limit in ChatGPT, maybe like twice a week, but otherwise my monthly API bill in the last 6 months usually is in the dollars frame. Not tens or thousands of dollars, just do…
GPT4 is very cheap when you're hitting the api with common knowledge questions or reasoning about known things, and asking for short replies. It's very expensive when you're loading up the context with a ton of data and asking it to convert some data points into a narrative structure.
Re: Mistral AI launches Mixtral-Next
#53I used this but, upon asking which model it is, it replied as being a "fine-tuned version of GPT 3.5". Any clue why? In a second chat it replied "You're chatting with one of the fine-tuned versions of the OpenAssistant model!".
Models don't "know" what they are. Usually the system prompt contains that imformation. If it doesn't, it will hallucinate an answer for you.
Re: Mistral AI launches Mixtral-Next
#54Earlier quoted context omitted.
And how does Mistral do "accurate long content retrieval"?
see the long range performance piece here https://arxiv.org/pdf/2401.04088.pdf
I don't think they explain it in the paper, do they? They just mention the result it seems. I am really curious to know too. Maybe they imply it's the result of their Mixture Of Experts architecture? Or maybe they just don't wanna say, idk
Re: Mistral AI launches Mixtral-Next
#55No indication that this a MoE (Mistral not Mixtral). Very exciting nevertheless, here’s hoping the bless the OS community once again!
It is indicated by the magnet link which they have posted on twitter (see comments above).
There's no magnet link for this new (Mistral Next) model as of right now.