121–130 of 255 posts
Earlier quoted context omitted.
unfortunately too big for the broader community to test. Will be very interesting to see how well it performs compared to the large models
Not really, looks like a ~40B class model which is very runnable.
So 2B for attention + 5Bx2 for inference = 12B in RAM at runtime.
Who know if I can run this on MBC Pro M3 max 128gb? at what TPS?
Who know if I can run this on MBC Pro M3 max 128gb? at what TPS?
Holy shit, this is some clever marketing.
Kinda wonder if any of their employees were part of the warez scene at some point.
Earlier quoted context omitted.
> I can run it on my Macbook Air at 12tkps, can't wait to try this on my desktop. That seems kinda low, are you using Metal GPU acceleration with llama.cpp? I don't have a macbook, but saw some of the llama.cpp benchmarks that suggest it can reach close to 30tk/s with GPU acceleration.
Thanks for the tip. I'm on the M2 Air with 16 GB's of ram. If anyone has faster than 12tkps on Air's let me know. I'm using the LM Studio GUI over llama.cpp with the "Apple Metal GPU" option. Increasing CPU threads seemingly does nothing either without metal. Ram usage hovers at 5.5GB with a q5_k_m of Mistral.
Who know if I can run this on MBC Pro M3 max 128gb? at what TPS?
RAM Wise, you can easily run a 70b with 128GB, 8x7B is obviously less than that.
Compute wise, I suppose it would be a bit slower than running a 13b.
edit: "actually", I think it might be faster than a 13b. 8 random 7b ~= 115GB, Mixtral is under 90. I will have to wait for more info/understanding.
Mistral sure does not bother too much with explanations, but this style gives me much more confidence in the product than Google's polished, corporate, soulless announcement of Gemini!
I will take weights over docs. Its does remind me how some Google employee was bragging that they disclosed the weights for the Gemini, and only the small mobile Gemini, as if that's a generous step over other companies.
I am 100% in agreement with your viewpoint, but feel squeamish seeing an un-needed lie coupled to it to justify it. Just so much Othering these days.
Hot take but Mistral 7B is the actual state of the art of LLM's. ChatGPT 4 is amazing yes and i've been a day 1 subscriber, but it's huge, runs on server farms far away and is more or less a black box. Mistral is tiny, and amazingly coherent and useful for it's size for both general questions and code, uncensored, and a leap i wouldn't have believed possible in just a year. I can run it on my Macbook Air at 12tkps, c…