State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration • Hacker News Board

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#11

Post by sterlind » Thu, Apr 06, 2023, 9:31 PM UTC

superkuh wrote: post #6

zhisbug wrote: post #2

No llama.cpp nor any compilation complexity. Run with two Python commands!

I think you have it backwards. The python (ie, huggingface, etc) implementations of transformers are the complex ones with dependency hell so bad even there's even a layer of package manager / env hell. This version of fastchat (there's 2) required a particular commit of huggingface libs for quite a while. Something that only changed recently. And it'll happen again in the future. Python just hides this complexity...…

I've spent the past couple days packaging an LLM playground environment as a Nix expression. it's been pure hell.

also nice to see you again, superkuh. I frequented your IRC channel about a decade ago.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#12

Post by siraben » Thu, Apr 06, 2023, 9:35 PM UTC

sterlind wrote: post #11

superkuh wrote: post #6

Earlier quoted context omitted.

I think you have it backwards. The python (ie, huggingface, etc) implementations of transformers are the complex ones with dependency hell so bad even there's even a layer of package manager / env hell. This version of fastchat (there's 2) required a particular commit of huggingface libs for quite a while. Something that only changed recently. And it'll happen again in the future. Python just hides this complexity...…

I've spent the past couple days packaging an LLM playground environment as a Nix expression. it's been pure hell. also nice to see you again, superkuh. I frequented your IRC channel about a decade ago.

Have you been successful in getting the LLM playground up with Nix?

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#13

Post by zhisbug » Thu, Apr 06, 2023, 9:35 PM UTC

superkuh wrote: post #10

Casteil wrote: post #4

Anyone here who's used both this and GPT4All? Any thoughts/input on how they compare?

My one take away after playing with both chat mode and text completion modes is that gpt4all 7B 4bit stays on the chat rails (doesn't start taking the role of the user, or spewing fine tuning boilerplate) much better than vicuna 7B 4bit. In text completion they're about the same but I'd still prefer the vanilla llama 7B in that case. There are a couple versions of gpt4all fine-tuned llama 7B and my favorite is the un…

Lmsys hasn't released any official 4-bit version. It might be a better idea to wait for the official 4-bit version. But it is interesting to learn that the third-party 4bit version has performance degeneration.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#14

Post by superkuh » Thu, Apr 06, 2023, 9:36 PM UTC

sterlind wrote: post #11

superkuh wrote: post #6

Earlier quoted context omitted.

I think you have it backwards. The python (ie, huggingface, etc) implementations of transformers are the complex ones with dependency hell so bad even there's even a layer of package manager / env hell. This version of fastchat (there's 2) required a particular commit of huggingface libs for quite a while. Something that only changed recently. And it'll happen again in the future. Python just hides this complexity...…

I've spent the past couple days packaging an LLM playground environment as a Nix expression. it's been pure hell. also nice to see you again, superkuh. I frequented your IRC channel about a decade ago.

Using nix and then complaining about having to set up your compilation environment libs/etc is kind of like sticking a rod in your bike's wheel spokes and complaining about crashing. Don't give up on the idea of system libraries (ie, use nix) and this doesn't happen.

Also, hi? I don't recall you by that nick but the internet is a small place sometimes.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#15

Post by zhisbug » Thu, Apr 06, 2023, 9:36 PM UTC

superkuh wrote: post #6

zhisbug wrote: post #2

No llama.cpp nor any compilation complexity. Run with two Python commands!

I think you have it backwards. The python (ie, huggingface, etc) implementations of transformers are the complex ones with dependency hell so bad even there's even a layer of package manager / env hell. This version of fastchat (there's 2) required a particular commit of huggingface libs for quite a while. Something that only changed recently. And it'll happen again in the future. Python just hides this complexity...…

no, the requirement on a particular HF commit has been fixed. It is no longer needed.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#16

Post by zhisbug » Thu, Apr 06, 2023, 9:38 PM UTC

zhisbug wrote: post #15

superkuh wrote: post #6

Earlier quoted context omitted.

I think you have it backwards. The python (ie, huggingface, etc) implementations of transformers are the complex ones with dependency hell so bad even there's even a layer of package manager / env hell. This version of fastchat (there's 2) required a particular commit of huggingface libs for quite a while. Something that only changed recently. And it'll happen again in the future. Python just hides this complexity...…

no, the requirement on a particular HF commit has been fixed. It is no longer needed.

it is really a matter of having faith on pytorch (or JAX) or on third-party cross-platform supports like llama-cpp. Apparently pytorch reduces a lot of complexity and grows extremely faster on cross-platform supports.

And, PyTorch does so well on GPUs!

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#17

Post by sterlind » Thu, Apr 06, 2023, 9:42 PM UTC

superkuh wrote: post #14

sterlind wrote: post #11

Earlier quoted context omitted.

I've spent the past couple days packaging an LLM playground environment as a Nix expression. it's been pure hell. also nice to see you again, superkuh. I frequented your IRC channel about a decade ago.

Using nix and then complaining about having to set up your compilation environment libs/etc is kind of like sticking a rod in your bike's wheel spokes and complaining about crashing. Don't give up on the idea of system libraries (ie, use nix) and this doesn't happen. Also, hi? I don't recall you by that nick but the internet is a small place sometimes.

oh, I'm very aware that I've brought this upon myself, but I'm sticking out for the greater good (and stubbornness.)

specifically, I'm trying to benchmark a bunch of different GPU configurations on different workloads on vast.ai, which uses Docker containers. I abhor Dockerfiles and my experience building containers with nix has been pleasant, so that's what I'm doing and why. fortunately I think I'm getting past the learning curve.

did our channel survive the demise of freenode? I was andares, I think I used to be annoying but I've gotten better.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#18

Post by superkuh » Thu, Apr 06, 2023, 9:44 PM UTC

zhisbug wrote: post #13

superkuh wrote: post #10

Earlier quoted context omitted.

My one take away after playing with both chat mode and text completion modes is that gpt4all 7B 4bit stays on the chat rails (doesn't start taking the role of the user, or spewing fine tuning boilerplate) much better than vicuna 7B 4bit. In text completion they're about the same but I'd still prefer the vanilla llama 7B in that case. There are a couple versions of gpt4all fine-tuned llama 7B and my favorite is the un…

Lmsys hasn't released any official 4-bit version. It might be a better idea to wait for the official 4-bit version. But it is interesting to learn that the third-party 4bit version has performance degeneration.

Lmsys hasn't released any official weights for anything. They've released "deltas" and other people have applied those deltas to the appropriate llama weights and done the quantization.

I reject your premise that the 8 to 4 bit quantization is the cause of the vicuna fine-tuned llamas very average performance though. This hasn't been the case for any of the other 8 to 4 bit quantizations. It would be a unique outlier. And so I don't think this is the "cause" here.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#19

Post by sterlind » Thu, Apr 06, 2023, 9:45 PM UTC

siraben wrote: post #12

sterlind wrote: post #11

Earlier quoted context omitted.

I've spent the past couple days packaging an LLM playground environment as a Nix expression. it's been pure hell. also nice to see you again, superkuh. I frequented your IRC channel about a decade ago.

Have you been successful in getting the LLM playground up with Nix?

yes, almost! I used poetry2nix and grafted a bunch of overrides to fix the torch-2.0 build, and I just got cuda working with it. I'm testing triton now. I'll submit my PR to poetry2nix so watch that space if you want it.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#20

Post by superkuh » Thu, Apr 06, 2023, 9:46 PM UTC

zhisbug wrote: post #15

superkuh wrote: post #6

Earlier quoted context omitted.

I think you have it backwards. The python (ie, huggingface, etc) implementations of transformers are the complex ones with dependency hell so bad even there's even a layer of package manager / env hell. This version of fastchat (there's 2) required a particular commit of huggingface libs for quite a while. Something that only changed recently. And it'll happen again in the future. Python just hides this complexity...…

no, the requirement on a particular HF commit has been fixed. It is no longer needed.

Right. That particular problem has been fixed. But the fact that it was needed indicates it will happen again. It exposes the underlying complexity of the huggingface transformer stack. It's wonderful code, don't get me wrong. It's just the furthest thing possible from the least complex.