Why is there so much focus on running GPT models on Mac OS? Is there something special about Apple's new chip, or Mac OS?
[deleted]
State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
41–50 of 88 posts
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#42MacBook with M1 chip here.python installed with homebrew tried to install with: pip install fschat then tried to run it with: python3 -m fastchat.serve.cli --model -name vicuna-7b --device mps --load-8bit got this: traceback (most recent call last): File " ", line 198, in _run_module_as_main File " ", line 88, in _run_code File "/opt/homebrew/lib/python3.11/site-packages/fastchat/serve/cli.py", line 9, in from transf…
It looks like you're on python 3.11 which has some issues with Pytorch. Downgrade to python 3.10 and try running it again.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#43Why is there so much focus on running GPT models on Mac OS? Is there something special about Apple's new chip, or Mac OS?
The shared ram and neural engine make for an interesting/powerful platform if people are willing to port to it.
Are the neural engines able to be leveraged by 3rd parties yet? I thought there was no API available yet.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#44I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?
I have not experienced that problem, but it sounds both annoying and funny. How often do you encounter it? A few times a day but it varies quite a bit. Have you been able to tell what causes it? Shorter prompts sometimes cause it but I have seen it on longer prompts as well. Is there a new version that fixes the issue? Not that I've seen released but it is worth checking.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#45I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?
I have not experienced that problem, but it sounds both annoying and funny. How often do you encounter it? A few times a day but it varies quite a bit. Have you been able to tell what causes it? Shorter prompts sometimes cause it but I have seen it on longer prompts as well. Is there a new version that fixes the issue? Not that I've seen released but it is worth checking.
It happens randomly, and I tried adjusting the gradio+model settings to match FastChat. I should start taking some screenshots of these cause some are funny. I asked how do I update a git repo and it answered correctly with git pull. Then it added HUMAN: how do I delete the whole folder and start over and answered ASSISTANT: try using git reset —hard, if not use rm -rf (paraphrasing here).
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#46So if I have a 32GB RAM Macbook Pro, and the instructions say this: "Vicuna-13B This conversion command needs around 60 GB of CPU RAM." Does this mean I simply cannot run that model at all? Or will it rip into HD swap or something to make the model weights and just take forever?
You can try the smaller 7B version.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#47I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?
I have not experienced that problem, but it sounds both annoying and funny. How often do you encounter it? A few times a day but it varies quite a bit. Have you been able to tell what causes it? Shorter prompts sometimes cause it but I have seen it on longer prompts as well. Is there a new version that fixes the issue? Not that I've seen released but it is worth checking.
That was brilliant. Thanks! You're welcome.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#48I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?
I've gotten that which alpaca.cpp. It'll start asking itself questions and then answer them.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#49I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?
I've been testing quite a few of these models lately.
For me, the absolute best is still the 65B 4-bit quantized llama model with the correct prompt and parameters, both for programming, language and general questions.
I am actually getting about 2 tokens/second with the latest llama.cpp using 16 threads on a 5950x with 64 gb of ram. 16 threads seems to be the sweet spot, any higher and it slows down, any lower and it is less consistent in the time to produce a token.
I am 100% convinced that the AI "market" will be a local thing. Running this and having access to all the information stored in the weights easily and without internet is just so great I think :)
Edit: the responding to itself "bug" is most likely an issue with the prompt you issue. The recent llama.cpp has a good starting point in the examples/chat-13b.sh I am using a modified version of that where I set the 65B model, change the moscow stuff to cairo and the node.js to a small C program.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#50So if I have a 32GB RAM Macbook Pro, and the instructions say this: "Vicuna-13B This conversion command needs around 60 GB of CPU RAM." Does this mean I simply cannot run that model at all? Or will it rip into HD swap or something to make the model weights and just take forever?
I just reached that step on my Linux laptop which has 32GB of RAM. I'm about to give it a try anyway, but I'm not hopeful based on that comment.
I'm wondering if anyone is torrenting these Vicuna-13B weights?