Slightly related question: what's a good coding LLM to run on a 4070 12GB card? Also, do coding LLMs use treesitter to "understand" code?
Try deepseek 6.7B
Mistral AI launches Mixtral-Next
31–40 of 55 posts
Re: Mistral AI launches Mixtral-Next
#32The Together.AI logo at the bottom is very hard to read... (Dark gray on black)
Re: Mistral AI launches Mixtral-Next
#33As someone who has only been using GPT-4 since its release, I am pleasantly surprised by how far open LLMs have come.
Re: Mistral AI launches Mixtral-Next
#34Slightly related question: what's a good coding LLM to run on a 4070 12GB card? Also, do coding LLMs use treesitter to "understand" code?
> do coding LLMs use treesitter...?
It's up to the app to put that into the context. Generally, coding LLMs do well if you provide them the source tree, graph, search results, notable files, etc in the context. The is how Sourcegraph's Cody product works, for example.
Re: Mistral AI launches Mixtral-Next
#35Note that it's actually "Mistral Next" not "Mixtral Next" - so it isn't necessarily a MoE. For example, an early version of Mistral Medium (Miqu) was not a MoE but instead a Llama 70B model. I wonder how many parameters this one has
I know what they were going for with the Mixtral name but every time I come across it I wonder if they considered just how easily the two might be confused. It seems like a poor branding decision - what if some expected the Mixtral performance but accidentally uses a Mistral model? What if someone wants the low resource usage of e.g. Mistral 7B but tries out Mixtral 8x7B instead? It's especially hard when your collea…
I feel like this is not really an issue. I personally lost track of all the llamas, gpts, etc - but if somebody is going to seriously use a certain model, they'll find out soon enough if they're using the wrong one.
Re: Mistral AI launches Mixtral-Next
#36Note that it's actually "Mistral Next" not "Mixtral Next" - so it isn't necessarily a MoE. For example, an early version of Mistral Medium (Miqu) was not a MoE but instead a Llama 70B model. I wonder how many parameters this one has
I know what they were going for with the Mixtral name but every time I come across it I wonder if they considered just how easily the two might be confused. It seems like a poor branding decision - what if some expected the Mixtral performance but accidentally uses a Mistral model? What if someone wants the low resource usage of e.g. Mistral 7B but tries out Mixtral 8x7B instead? It's especially hard when your collea…
I agree. I also think the Llama naming was confusing - versioning by capitalization? (LLaMA vs Llama)
Re: Mistral AI launches Mixtral-Next
#37AIExplained on youtube has guessed that Gemini 1.5 pro is taking Mistral’s accurate long content retrieval and Google just scaled it as much as they could. The Gemini 1.5 pro paper has a citation back to the last mistral paper in 2024.
And how does Mistral do "accurate long content retrieval"?
see the long range performance piece here
https://arxiv.org/pdf/2401.04088.pdf
Re: Mistral AI launches Mixtral-Next
#38It's quite funny to use! It is better when speaking French than chat gpt3.5 on my opinion
It is a French company so maybe they have extra French datasets? I've been quite disappointed by French LLMs on Huggingface when I tried a month ago.
Mistral models tend to be quite good at non-English languages. French of course, but also Spanish, German and Italian. From what I have read it’s something they consider important when training their models.
Re: Mistral AI launches Mixtral-Next
#39[deleted]
Re: Mistral AI launches Mixtral-Next
#40Earlier quoted context omitted.
I know what they were going for with the Mixtral name but every time I come across it I wonder if they considered just how easily the two might be confused. It seems like a poor branding decision - what if some expected the Mixtral performance but accidentally uses a Mistral model? What if someone wants the low resource usage of e.g. Mistral 7B but tries out Mixtral 8x7B instead? It's especially hard when your collea…
I feel like this is not really an issue. I personally lost track of all the llamas, gpts, etc - but if somebody is going to seriously use a certain model, they'll find out soon enough if they're using the wrong one.
It has definitely affected myself and colleagues, perhaps we didn't waste much time but it's annoying. Even if it isn't a problem, it really cannot hurt to make the naming easier to understand.