131–140 of 161 posts
What is the excitement around models that arent as good as llama? This is clearly an inferior model that they are willing to share for marketing purposes. If it was an improvement over llama, sure, but it seems like just an ad for bad AI.
What are the advantages compared to GPT? Looking forward to using it!
It actually does what you tell it, and won't try to silently change your prompt to conform to a specific flavor of Californian hysterics, which is what OpenAI's products do.
Also, since it's a local model, your queries aren't being datamined nor can access to the service be revoked on a whim.
What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library? Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?
There is a user called The Bloke on hugging face- they release pre quantized models pretty soon after the full size drop. Just watch their page and pray you can fit the 4 bit in your GPU. I’m sure they are already working on it.
Very important to note that this is a base model, not an instruct model. Instruct fine-tuned models are what's useful for chat.
Earlier quoted context omitted.
262 GB is not exactly small. But yes it seems they're all getting them out the door in case they end up being worse than llama-3 in which case it'll be too embarrassing to release later.
Since it’s a MOE model it will only need to load a few of the 8 sub models into vram in order to answer a query. So it may look large, but I think a quantized model will easily fit on a Mac with 64GB of memory and maybe even a bit fewer bits and it’ll fit into 32GB. I think it might be the end for 24GB 4090 cards though :(
But, still, its going to need 262GB for weights + a variable amount based on context without quantization, and 66GB+ at 4-bit quantization.
Very important to note that this is a base model, not an instruct model. Instruct fine-tuned models are what's useful for chat.
What's the feeling of playing with a powerful base model? Will it just complete the prompt text like a continuation of it?
This model is apparently surprisingly good at chat, even though it is a base model, and will take part it it to some extent. It should be really interesting once it's fine-tuned.
What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library? Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?
There is a user called The Bloke on hugging face- they release pre quantized models pretty soon after the full size drop. Just watch their page and pray you can fit the 4 bit in your GPU. I’m sure they are already working on it.
Earlier quoted context omitted.
Wish it had GPT-4, that’s the one to beat still.
It is there, not for all the benchmarks, but for those where it is included, GPT-4 scores much higher. Not surprising since GPT-4 is still state-of-the-art and much bigger. Where Mistral has been particularly impressive is when you take the size of the model into account.
Earlier quoted context omitted.
262 GB is not exactly small. But yes it seems they're all getting them out the door in case they end up being worse than llama-3 in which case it'll be too embarrassing to release later.
Since it’s a MOE model it will only need to load a few of the 8 sub models into vram in order to answer a query. So it may look large, but I think a quantized model will easily fit on a Mac with 64GB of memory and maybe even a bit fewer bits and it’ll fit into 32GB. I think it might be the end for 24GB 4090 cards though :(