State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration • Hacker News Board

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#51

Post by rnosov » Thu, Apr 06, 2023, 11:47 PM UTC

steve_adams_86 wrote: post #43

matwood wrote: post #29

Earlier quoted context omitted.

The shared ram and neural engine make for an interesting/powerful platform if people are willing to port to it.

Are the neural engines able to be leveraged by 3rd parties yet? I thought there was no API available yet.

They are leveraging Apple’s Metal Performance Shaders[1] not the neural engine. From the chart, it looks like you might get ~20x max boost on inference over plain CPU. Obviously, it's not like having RTX 4090 but better than nothing.

[1] https://pytorch.org/blog/introducing-accelerated-pytorch-tra...

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#52

Post by [deleted] » Thu, Apr 06, 2023, 11:56 PM UTC

JohnFen wrote: post #36

ReptileMan wrote: post #33

Earlier quoted context omitted.

The Silicon Valley ethos has always been - do it first worry about legality later. If you go bust - nobody will care. If you become small - you will be ignored. If you go big - lawyers will figure something out to cut a deal.

That is a thoroughly bankrupt ethos that should be denounced every time it pops up. It is literally condoning criminality.

[deleted]

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#53

Post by taf2 » Fri, Apr 07, 2023, 12:01 AM UTC

syntaxing wrote: post #40

I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?

I got this and after I redownloaded the model/ggml file it was fixed... could be some corruption in the model file?

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#54

Post by siraben » Fri, Apr 07, 2023, 12:07 AM UTC

syntaxing wrote: post #40

I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?

Haven't used the text generation web UI, but if you're using the CLI, use the "reverse prompt" option to hand control back to the user.

  ./bin/main -i --interactive-first -r '### Human:' -t 8 -n 512 --instruct -m ./models/vicuna-13B/ggml-vicuna-13b-4bit.bin --color

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#55

Post by wmf » Fri, Apr 07, 2023, 12:14 AM UTC

steve_adams_86 wrote: post #43

matwood wrote: post #29

Earlier quoted context omitted.

The shared ram and neural engine make for an interesting/powerful platform if people are willing to port to it.

Are the neural engines able to be leveraged by 3rd parties yet? I thought there was no API available yet.

CoreML is the API.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#56

Post by syntaxing » Fri, Apr 07, 2023, 12:49 AM UTC

tyfon wrote: post #49

syntaxing wrote: post #40

I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?

I've been testing quite a few of these models lately. For me, the absolute best is still the 65B 4-bit quantized llama model with the correct prompt and parameters, both for programming, language and general questions. I am actually getting about 2 tokens/second with the latest llama.cpp using 16 threads on a 5950x with 64 gb of ram. 16 threads seems to be the sweet spot, any higher and it slows down, any lower and i…

I really want to try out the 65B model, I hear great things! Sadly none of my computers can handle it, slightly tempted to get more RAM but I’m on a 8 core i7.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#57

Post by zhwu » Fri, Apr 07, 2023, 12:50 AM UTC

bawana wrote: post #28

MacBook with M1 chip here.python installed with homebrew tried to install with: pip install fschat then tried to run it with: python3 -m fastchat.serve.cli --model -name vicuna-7b --device mps --load-8bit got this: traceback (most recent call last): File " ", line 198, in _run_module_as_main File " ", line 88, in _run_code File "/opt/homebrew/lib/python3.11/site-packages/fastchat/serve/cli.py", line 9, in from transf…

You need to use the transformers from the main branch instead of the pypi version, because the llama support is recently added. According to the readme of the repo, you need to install transformers with: pip3 install git+https://github.com/huggingface/transformers

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#58

Post by acchow » Fri, Apr 07, 2023, 1:31 AM UTC

dchuk wrote: post #39

So if I have a 32GB RAM Macbook Pro, and the instructions say this: "Vicuna-13B This conversion command needs around 60 GB of CPU RAM." Does this mean I simply cannot run that model at all? Or will it rip into HD swap or something to make the model weights and just take forever?

Can someone explain why computing a delta needs to hold the entire model at once? Can't it just do one layer at time?

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#59

Post by ThorsBane » Fri, Apr 07, 2023, 1:35 AM UTC

This is cool, but how can we get Facebook’s fingers out of the pie with open source weights?

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#60

Post by forgingahead » Fri, Apr 07, 2023, 2:00 AM UTC

rnosov wrote: post #42

bawana wrote: post #28

MacBook with M1 chip here.python installed with homebrew tried to install with: pip install fschat then tried to run it with: python3 -m fastchat.serve.cli --model -name vicuna-7b --device mps --load-8bit got this: traceback (most recent call last): File " ", line 198, in _run_module_as_main File " ", line 88, in _run_code File "/opt/homebrew/lib/python3.11/site-packages/fastchat/serve/cli.py", line 9, in from transf…

It looks like you're on python 3.11 which has some issues with Pytorch. Downgrade to python 3.10 and try running it again.

This is incredible to me (not your comment per se, but what you're referencing). I really don't understand how brittle and fragile Python is with all its dependencies. It's crazy to me that a simple bump from 3.10 to 3.11 can break Pytorch. This is like bumping your Ruby version up one level and suddenly Rails doesn't work.

Why on earth is Python like this? It's so frustrating coming from other languages where the dependency management plan isn't just so YOLO and free-for-all.