Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

twitter.com

Post reply on HN
Print view View original

81–88 of 88 posts

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#81

Post by sterlind »

anotherhue wrote: post #72
sterlind wrote: post #17

Earlier quoted context omitted.

oh, I'm very aware that I've brought this upon myself, but I'm sticking out for the greater good (and stubbornness.) specifically, I'm trying to benchmark a bunch of different GPU configurations on different workloads on vast.ai, which uses Docker containers. I abhor Dockerfiles and my experience building containers with nix has been pleasant, so that's what I'm doing and why. fortunately I think I'm getting past the…

Care to share some of your progress? I have similar (stronger?) feelings regarding Dockerfile's big-ball-of-state nonsense. (The irony of holding this opinion while dealing with pre-trained AI models is not lost)

I finished my work on poetry2nix and submitted a PR which works perfectly (at least with preferWheels=true.) now I have a wonderful live environment with torch, triton, transformers, etc. Docker builds are fast and lightweight since I use buildLayeredImage. it is, truly, the promised land my forefathers prophesized.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#82

Post by rnk »

messe wrote: post #38
tric wrote: post #27

Why is there so much focus on running GPT models on Mac OS? Is there something special about Apple's new chip, or Mac OS?

> Why is there so much focus on running GPT models on Mac OS? Because a MacBook with 96GB of RAM is cheaper than a GPU with anything close to that.

So the question is how much ram do you need? You and another person mentioned 96gb, the person below says he can run it with 24gb. What's the proper amount that is the best amount of ram for now? Of course 128gb/max is the best, but what's a great amount to have now. I never bought an m1, thinking of buying one now ;-)

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#83

Post by lhl »

tyfon wrote: post #73
lhl wrote: post #67

Earlier quoted context omitted.

Sounds like you've had some more success w/ w/ raw LLaMA - would def be interested in how you're prompting it. BTW, for those interested (looks like the markdown rendering is a bit messed up) but here are some notes I'm taking for some of the nuts and bolts for the local models I'm running: https://mostlyobvious.org/?link=%2FReference%2FSoftware%2FGe...

So I didn't see this before today, I will respond anyway and the siblings can also see. There is quite a big difference between the 65B model and especially the 13B and 7B models. But here is the bash script[0] I launch my "go to" AI, it's called Omnius :) As written in the previous comment, it is a modified version of the examples/chat-13b.sh that is included in the llama.cpp github. [0] https://pastebin.com/SeKE3Ua…

Ah thanks a lot, I tried out the llama.cpp examples before the k-shot chat prompts, this is definitely much better!

I have a 5950X as well, but sadly, token generation is a bit too slow for me now. (I've had turbo turned off for efficiency as well, but maybe I'll see if the extra cycles helps).

I'm giving 30B a try on my GPU now with https://github.com/oobabooga/text-generation-webui/wiki/LLaM... and if it's not good then will give layer offloading with 65B a try and see if I can get it running well.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#85

Post by FLT8 »

dchuk wrote: post #39

So if I have a 32GB RAM Macbook Pro, and the instructions say this: "Vicuna-13B This conversion command needs around 60 GB of CPU RAM." Does this mean I simply cannot run that model at all? Or will it rip into HD swap or something to make the model weights and just take forever?

Vicuna-13B loads and idles at ~26GB RAM usage on a M1Max/64GB. When answering questions, that grows to around 75GB, and yes, you can feel it (and the machine) slow down significantly when it starts hitting swap. I think realistically you'd be wanting to stick to the 7B model on a 32G machine (even if you could get the weight deltas to apply correctly).

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#86

Post by jpp4e »

nickthegreek wrote: post #34
tric wrote: post #27

Why is there so much focus on running GPT models on Mac OS? Is there something special about Apple's new chip, or Mac OS?

I can run the 30b 4bit model on my m2 air that has 24gb of ram.

Hi nickthegreek!

Could you tell me how you did that? Did you use FastChat or something else? Which model to download? What command to run?

Thank you!!!

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#87

Post by matwood »

jondwillis wrote: post #75
matwood wrote: post #70

Earlier quoted context omitted.

I have despised Python ever since the 2=>3 transition for the reasons you say. Tools like pyenv help, but it's still a mess. It makes me sad that all the popular ML tooling ends up built in Python.

I wonder how much the space has been encumbered by Python’s relative weaknesses. As a bit of an outsider, I kind of assume there’s some hidden advantage of Python for AI/ML that I just don’t “get.”

I think Python is easy to learn/use for programming adjacent fields like data science. It's seen as easy for non-traditional programmers.

Re: State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

#88

Post by nickthegreek »

jpp4e wrote: post #86
nickthegreek wrote: post #34

Earlier quoted context omitted.

I can run the 30b 4bit model on my m2 air that has 24gb of ram.

Hi nickthegreek! Could you tell me how you did that? Did you use FastChat or something else? Which model to download? What command to run? Thank you!!!

https://huggingface.co/Pi3141/alpaca-lora-30B-ggml

I believe I’m using alpaca.cpp with a command:

./chat -m

Post reply on HN