Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

twitter.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#61

Post by satvikpendem »

tyfon wrote: post #49
syntaxing wrote: post #40

I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?

I've been testing quite a few of these models lately. For me, the absolute best is still the 65B 4-bit quantized llama model with the correct prompt and parameters, both for programming, language and general questions. I am actually getting about 2 tokens/second with the latest llama.cpp using 16 threads on a 5950x with 64 gb of ram. 16 threads seems to be the sweet spot, any higher and it slows down, any lower and i…

Could you share your prompts and parameters? I tried it and I didn't really seem to get much better results than ChatGPT or others.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#62

Post by lhl »

syntaxing wrote: post #40

I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?

I'm in the process of testing the various self-hosted LLMs. I have an M2 MBA laptop and a 5950X w/ 64GB RAM and an RTX 4090 (24GB VRAM).

I've used ChatGPT 3.5 and 4 quite a bit, and have done a bunch of comparisons w/ nat.dev's Playground between a variety of models (claude-instant provides gpt-3.5-turbo level output and is about 3-4X faster; gpt-3.5-turbo, text-davinci-003 to me are about equal and about the cutoff level of where they are generally useful for me - reliability as an end user for summarizations, Q&A, code assistance, etc).

I found all the raw LLaMA variants I could run (up to 30B) to not be very coherent or useful. pythia, gpt-j, gpt-neox, chatglm and the other open raw models I found to be much worse than what the various eval scores would suggest (PIQA, HellaSwag, WinoGrande, ARC-e, etc)? I did a fair amount of playing w/ inference hyper-parameters early on to no avail, but did not do much k-shot learning or proper prompts (like the one's Scale AI uses for training).

I tried a bunch of other Alpaca/instruction-tuned models and they're better, but IMO still not very good. GPT4All w/ the unfiltered checkpoint was the only one that did OK until I tried Vicuna (13B load-8-bit on GPU; I tried Baiz but wasn't impressed, have yet to try Koala, but don't have high expectations). Vicuna does a better job than GPT4All, but I did notice some of the going off the rails/not stopping - it however strongly leans on "as an AI language model..." responses - IMO, any fine-tune based on ChatGPT output really should filter that out, it really knee caps the responses.

One surprise, while it generally doesn't perform quite as well, it tends to be more lucid and in some cases does a significantly better job, is RWKV Raven (ChatRWKV is pretty easy to get going; I can run the v7 14B fw/ fp16int8 in about 16GB of VRAM).

The rate of advancement over just a few weeks is really impressive and it's been really fun catching up on the state of the art on LLMs (I wasn't paying much attention before, despite playing around a bunch w/ SD image generation models previously) and I'm still learning, but after poking around w/ these "smaller" self-hosted models makes me wonder if there's some threshold (50B+ params?) or other secret sauce that captures the "magic" that gpt-3.5 seems to reach (from benchmarks LLaMA 65B is supposed to outperform Chichilla 70B, Gopher 280B, and even match PaLM 540B - gpt-3.5 is ~175-200B, gpt-4 is estimated at 1T parameters).

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#63

Post by fortyseven »

JohnFen wrote: post #36
ReptileMan wrote: post #33

Earlier quoted context omitted.

The Silicon Valley ethos has always been - do it first worry about legality later. If you go bust - nobody will care. If you become small - you will be ignored. If you go big - lawyers will figure something out to cut a deal.

That is a thoroughly bankrupt ethos that should be denounced every time it pops up. It is literally condoning criminality.

"Won't somebody think of the poor defenseless corporations?!"

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#64

Post by baq »

JohnFen wrote: post #36
ReptileMan wrote: post #33

Earlier quoted context omitted.

The Silicon Valley ethos has always been - do it first worry about legality later. If you go bust - nobody will care. If you become small - you will be ignored. If you go big - lawyers will figure something out to cut a deal.

That is a thoroughly bankrupt ethos that should be denounced every time it pops up. It is literally condoning criminality.

No crimes in this case, just license breaches. After a few training iterations, it’ll be very muddled anyways.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#65

Post by baq »

forgingahead wrote: post #60
rnosov wrote: post #42

Earlier quoted context omitted.

It looks like you're on python 3.11 which has some issues with Pytorch. Downgrade to python 3.10 and try running it again.

This is incredible to me (not your comment per se, but what you're referencing). I really don't understand how brittle and fragile Python is with all its dependencies. It's crazy to me that a simple bump from 3.10 to 3.11 can break Pytorch . This is like bumping your Ruby version up one level and suddenly Rails doesn't work. Why on earth is Python like this? It's so frustrating coming from other languages where the d…

C API changes between minor versions. It’s one of the bigger reasons to bump the minor version.

I agree though, Python packaging is consistently hell in an otherwise pleasant environment… saying that as a Python user since 1.5.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#66

Post by GaggiX »

dchuk wrote: post #39

So if I have a 32GB RAM Macbook Pro, and the instructions say this: "Vicuna-13B This conversion command needs around 60 GB of CPU RAM." Does this mean I simply cannot run that model at all? Or will it rip into HD swap or something to make the model weights and just take forever?

Someone really needs to write a script that does not load both entire models into memory to do this.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#67

Post by lhl »

tyfon wrote: post #49
syntaxing wrote: post #40

I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?

I've been testing quite a few of these models lately. For me, the absolute best is still the 65B 4-bit quantized llama model with the correct prompt and parameters, both for programming, language and general questions. I am actually getting about 2 tokens/second with the latest llama.cpp using 16 threads on a 5950x with 64 gb of ram. 16 threads seems to be the sweet spot, any higher and it slows down, any lower and i…

Sounds like you've had some more success w/ w/ raw LLaMA - would def be interested in how you're prompting it.

BTW, for those interested (looks like the markdown rendering is a bit messed up) but here are some notes I'm taking for some of the nuts and bolts for the local models I'm running: https://mostlyobvious.org/?link=%2FReference%2FSoftware%2FGe...

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#68

Post by wensheng »

lhl wrote: post #67
tyfon wrote: post #49

Earlier quoted context omitted.

I've been testing quite a few of these models lately. For me, the absolute best is still the 65B 4-bit quantized llama model with the correct prompt and parameters, both for programming, language and general questions. I am actually getting about 2 tokens/second with the latest llama.cpp using 16 threads on a 5950x with 64 gb of ram. 16 threads seems to be the sweet spot, any higher and it slows down, any lower and i…

Sounds like you've had some more success w/ w/ raw LLaMA - would def be interested in how you're prompting it. BTW, for those interested (looks like the markdown rendering is a bit messed up) but here are some notes I'm taking for some of the nuts and bolts for the local models I'm running: https://mostlyobvious.org/?link=%2FReference%2FSoftware%2FGe...

super helpful, thank you.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#69

Post by egorfine »

tyfon wrote: post #49
syntaxing wrote: post #40

I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?

I've been testing quite a few of these models lately. For me, the absolute best is still the 65B 4-bit quantized llama model with the correct prompt and parameters, both for programming, language and general questions. I am actually getting about 2 tokens/second with the latest llama.cpp using 16 threads on a 5950x with 64 gb of ram. 16 threads seems to be the sweet spot, any higher and it slows down, any lower and i…

How do you talk to llama? It doesn't respond to instructions, so it's a bit complicated to have it extract keywords and/or summarize texts. Can you please share examples of llama prompts?

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#70

Post by matwood »

forgingahead wrote: post #60
rnosov wrote: post #42

Earlier quoted context omitted.

It looks like you're on python 3.11 which has some issues with Pytorch. Downgrade to python 3.10 and try running it again.

This is incredible to me (not your comment per se, but what you're referencing). I really don't understand how brittle and fragile Python is with all its dependencies. It's crazy to me that a simple bump from 3.10 to 3.11 can break Pytorch . This is like bumping your Ruby version up one level and suddenly Rails doesn't work. Why on earth is Python like this? It's so frustrating coming from other languages where the d…

I have despised Python ever since the 2=>3 transition for the reasons you say. Tools like pyenv help, but it's still a mess. It makes me sad that all the popular ML tooling ends up built in Python.
Post reply on HN