State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration • Hacker News Board

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#31

Post by alwayslikethis » Thu, Apr 06, 2023, 10:39 PM UTC

wejick wrote: post #25

Seems like llama derived model are flourishing. However with llama is licensed as academic only and noncommercial model, what is the path for bringing this to production of for profit purpose? I certainly interested doing so.

Copilot style. Train a distilled model based on it, and now it's a new model unencumbered by copyright.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#32

Post by 19h » Thu, Apr 06, 2023, 10:40 PM UTC

tric wrote: post #27

Why is there so much focus on running GPT models on Mac OS? Is there something special about Apple's new chip, or Mac OS?

Unified memory allows both CPU and GPU to use the same memory, effectively giving a MacBook with 96GB of memory 96GB of VRAM (minus OS overhead obv).

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#33

Post by ReptileMan » Thu, Apr 06, 2023, 10:40 PM UTC

wejick wrote: post #25

Seems like llama derived model are flourishing. However with llama is licensed as academic only and noncommercial model, what is the path for bringing this to production of for profit purpose? I certainly interested doing so.

The Silicon Valley ethos has always been - do it first worry about legality later. If you go bust - nobody will care. If you become small - you will be ignored. If you go big - lawyers will figure something out to cut a deal.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#34

Post by nickthegreek » Thu, Apr 06, 2023, 10:48 PM UTC

tric wrote: post #27

Why is there so much focus on running GPT models on Mac OS? Is there something special about Apple's new chip, or Mac OS?

I can run the 30b 4bit model on my m2 air that has 24gb of ram.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#35

Post by avereveard » Thu, Apr 06, 2023, 10:48 PM UTC

wejick wrote: post #25

Seems like llama derived model are flourishing. However with llama is licensed as academic only and noncommercial model, what is the path for bringing this to production of for profit purpose? I certainly interested doing so.

The methodology for alpaca has proven powerful and it's being applied to model with better licensing. It's hard to track lineage, but I think openassistant models are the most permissive at the moment, they use a openly sourced set of data to build an instruct model on top of phiia, which itself is a gptneox trained on a duplicated version of the famous the pile dataset.

The problem is verifying the licensing claims for these composed solutions is becoming exceedingly hard.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#36

Post by JohnFen » Thu, Apr 06, 2023, 10:54 PM UTC

ReptileMan wrote: post #33

wejick wrote: post #25

Seems like llama derived model are flourishing. However with llama is licensed as academic only and noncommercial model, what is the path for bringing this to production of for profit purpose? I certainly interested doing so.

The Silicon Valley ethos has always been - do it first worry about legality later. If you go bust - nobody will care. If you become small - you will be ignored. If you go big - lawyers will figure something out to cut a deal.

That is a thoroughly bankrupt ethos that should be denounced every time it pops up. It is literally condoning criminality.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#37

Post by messe » Thu, Apr 06, 2023, 11:05 PM UTC

wejick wrote: post #25

Seems like llama derived model are flourishing. However with llama is licensed as academic only and noncommercial model, what is the path for bringing this to production of for profit purpose? I certainly interested doing so.

> llama is licensed as academic only and noncommercial model

Are weights even copyrightable? I was under the impression that they weren't (although it hasn't been tested, and there's a chance they may run afoul of database rights).

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#38

Post by messe » Thu, Apr 06, 2023, 11:08 PM UTC

tric wrote: post #27

Why is there so much focus on running GPT models on Mac OS? Is there something special about Apple's new chip, or Mac OS?

> Why is there so much focus on running GPT models on Mac OS?

Because a MacBook with 96GB of RAM is cheaper than a GPU with anything close to that.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#39

Post by dchuk » Thu, Apr 06, 2023, 11:09 PM UTC

So if I have a 32GB RAM Macbook Pro, and the instructions say this:

"Vicuna-13B This conversion command needs around 60 GB of CPU RAM."

Does this mean I simply cannot run that model at all? Or will it rip into HD swap or something to make the model weights and just take forever?

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#40

Post by syntaxing » Thu, Apr 06, 2023, 11:10 PM UTC

I’ve been using the GPTQ 4 bit quantized 13B with Text generation web UI and it’s been amazing. Probably the closest to ChatGPT I have used so far. I still get an issue where it keeps on talking to itself by generating its own prompt and then answering it. Has anyone experienced the same thing?