State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
1–10 of 88 posts
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#2No llama.cpp nor any compilation complexity. Run with two Python commands!
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#3So far I think what these models lack is memory of people and other things. Especially if not as popular. And probably a ton more.
E.g. try asking it "Who is Tyler Volk?"
Then try asking GPT-4 "Who is Tyler Volk?"
Then check who he is online.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#4Anyone here who's used both this and GPT4All? Any thoughts/input on how they compare?
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#5So far I think what these models lack is memory of people and other things. Especially if not as popular. And probably a ton more. E.g. try asking it "Who is Tyler Volk?" Then try asking GPT-4 "Who is Tyler Volk?" Then check who he is online.
Probably because the parameter count is way lower so it's less able to memorize things
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#6No llama.cpp nor any compilation complexity. Run with two Python commands!
I think you have it backwards. The python (ie, huggingface, etc) implementations of transformers are the complex ones with dependency hell so bad even there's even a layer of package manager / env hell. This version of fastchat (there's 2) required a particular commit of huggingface libs for quite a while. Something that only changed recently. And it'll happen again in the future. Python just hides this complexity... until it doesn't. Like beautiful but rapidly rotting fruit.
llama.cpp will remain a single two line project (git clone https://github.com/ggerganov/llama.cpp, make -j) that will compile easily and run on anything. No external deps to pin to a particular commit (that will only have a lifetime of some months) as things change rapidly.
That said, the changes in the ggml weights format the last 2 weeks were annoying, but now that the mmap-style weights are settled on it should be less converting. In that sense huggingface wins, it only has two incompatible weights formats. llama.cpp's ggml has had 3.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#7No llama.cpp nor any compilation complexity. Run with two Python commands!
I think you have it backwards. The python (ie, huggingface, etc) implementations of transformers are the complex ones with dependency hell so bad even there's even a layer of package manager / env hell. This version of fastchat (there's 2) required a particular commit of huggingface libs for quite a while. Something that only changed recently. And it'll happen again in the future. Python just hides this complexity...…
This has been my experience so far as well. GPT4All feels pretty fragile with all its dependencies.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#8So far I think what these models lack is memory of people and other things. Especially if not as popular. And probably a ton more. E.g. try asking it "Who is Tyler Volk?" Then try asking GPT-4 "Who is Tyler Volk?" Then check who he is online.
The language is also quite unnatural feeling.
Neat none the less but hardly a standout in my opinion.
Everything is state of the art at the moment I guess so can't criticise that too much.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#9Anyone here who's used both this and GPT4All? Any thoughts/input on how they compare?
I asked GPT4All one of Vicuna's benchmark questions:
"What if the Internet had been invented during the Renaissance period?"
Check out their responses: https://imgur.com/a/mPrdZ1W More questions here: https://vicuna.lmsys.org/eval/
Note: not an apple-to-apple comparison but that's the model checkpoint I found on their git repo.
Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
#10Anyone here who's used both this and GPT4All? Any thoughts/input on how they compare?
My one take away after playing with both chat mode and text completion modes is that gpt4all 7B 4bit stays on the chat rails (doesn't start taking the role of the user, or spewing fine tuning boilerplate) much better than vicuna 7B 4bit. In text completion they're about the same but I'd still prefer the vanilla llama 7B in that case.
There are a couple versions of gpt4all fine-tuned llama 7B and my favorite is the unfiltered one (gpt4all-lora-unfiltered-quantized.bin). https://github.com/nomic-ai/gpt4all#try-it-yourself