91–100 of 161 posts
Earlier quoted context omitted.
With open models, yes we are at the performance of at least the first release of ChatGPT 4.
Could you recommend one or a few in particular?
Earlier quoted context omitted.
I hope i get it to run on my 96gb m2 in q4.
It actually does, in case anybody wonders. But it seems as if it's not fine tuned to chat, or i'm doing it wrong at the moment. Getting a lot of duplicates and non useful answers.
Earlier quoted context omitted.
Since it’s a MOE model it will only need to load a few of the 8 sub models into vram in order to answer a query. So it may look large, but I think a quantized model will easily fit on a Mac with 64GB of memory and maybe even a bit fewer bits and it’ll fit into 32GB. I think it might be the end for 24GB 4090 cards though :(
Unless something has changed, it needs to load the full 8 models at the same time. During inference it performs like a 2 x base model. Mixtral 7B @ 5 bit takes up over 30gb on my M3 Max. That's over 90 for this at the same quantization. Realistically you probably need a 128gb machine to run this with good results.
Earlier quoted context omitted.
LM Studio is a great way to test out LLMs on my MacBook: https://lmstudio.ai/ Really easy to search huggingface for new models to test directly in the app.
Make sure you get the prompt template set correctly, the defaults are wrong for a lot of models.
Earlier quoted context omitted.
Exploit a memory safety issue in the tokenizer/or other parts of your LLM infra written in a native language.
??? With weights?
Earlier quoted context omitted.
Make sure you get the prompt template set correctly, the defaults are wrong for a lot of models.
Could you explain how to do this properly ? I've been having problems with the app and am wondering if this is ehy
They Just announced their new model on Twitter, which you can download using torrent
Earlier quoted context omitted.
It beats the old GPT4 version in lmsys benchmark you can check it out here https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... but Command R is commercially licensed We can assume that mistral will do a better job.
> but Command R is commercially licensed It is licensed under CC-BY-NC-4.0. That license means you are free to use, modify and redistribute it, so long as you aren't doing so "commercially". What exactly counts as "commercial" use is a complex legal question, and the answer may vary from jurisdiction to jurisdiction (different courts may interpret the phrase differently). But, for example, if you are just using it at…
Morally I feel 100% fine because the app would be just as appealing without it, and subscribing means you get sync, you could theoretically not pay me and use Command R
Earlier quoted context omitted.
16GB will be way too small unfortunately — this has over 3x the param count, so at best you're looking at a 24GB card with extreme 2bit quantization. Really though if you're just looking to run models personally and not finetune (which requires monstrous amounts of VRAM), Macs are the way to go for this kind of mega model: Macs have unified memory between the GPU and CPU, and you can buy them with a lot of RAM. It'll…
Reasonably priced Epyc systems with up to 12 memory channels and support for several TB of system memory are now available. Used datacenter hardware is even less expensive. They are on par with the memory bandwidth available to any one of the CPU, GPU, or NPU in the highest end Macs, but capable of driving MUCH more memory. And much simpler to run Linux or Windows on.
Earlier quoted context omitted.
16GB will be way too small unfortunately — this has over 3x the param count, so at best you're looking at a 24GB card with extreme 2bit quantization. Really though if you're just looking to run models personally and not finetune (which requires monstrous amounts of VRAM), Macs are the way to go for this kind of mega model: Macs have unified memory between the GPU and CPU, and you can buy them with a lot of RAM. It'll…
You can QLoRA decent models on 24GB VRAM. There’s also optimised kernels like Unsloth that are really VRAM efficient and good for hobbyists.