Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alpaca-LoRA with Docker

github.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Alpaca-LoRA with Docker

#11

Post by nico »

syntaxing wrote: post #8
nico wrote: post #6

The demo on HuggingFace with the pre trained mode doesn’t seem that good. Although better than Bard (btw, Bard sucks compared to ChatGPT and can’t even do translations - which I would have expected out of the box from Google)

It’s worth noting this is the 7B model (nonquantized). You can get this running on pretty much any GPU with 8GB VRAM and above. You can run the 13B model but that would take two GPU or reducing FP16 to FP8 (I haven’t tried it myself). A single connection for chatgpt is rumored to require 8X A100.

It makes me wonder if this trend will kill NVIDIA.

At this pace we might not even need GPUs anymore.

Re: Alpaca-LoRA with Docker

#12

Post by b33j0r »

teekert wrote: post #9

That name is so unfortunate. Nobody searched “Lora” before picking it. Bit of a blunder if you ask me.

They even capitalize the R like LoRa, but I don’t think we’ll be running this model on an ESP32 to much profit.

Perhaps someone will release a llama I can run at home… how about “llama-homekit”? ;)

Re: Alpaca-LoRA with Docker

#13

Post by jnwatson »

nico wrote: post #11
syntaxing wrote: post #8

Earlier quoted context omitted.

It’s worth noting this is the 7B model (nonquantized). You can get this running on pretty much any GPU with 8GB VRAM and above. You can run the 13B model but that would take two GPU or reducing FP16 to FP8 (I haven’t tried it myself). A single connection for chatgpt is rumored to require 8X A100.

It makes me wonder if this trend will kill NVIDIA. At this pace we might not even need GPUs anymore.

The race for bigger NNs will never stop.

Re: Alpaca-LoRA with Docker

#14

Post by HnUser12 »

> Tell me about you

>I am a 25-year-old woman from the United States. I have a bachelor's degree in computer science and am currently pursuing a master's degree in data science. I am passionate about technology and am always looking for new ways to use it to make the world a better place. Outside of work, I enjoy spending time with my family and friends, reading, and traveling.

Well, I was starting to get tired of "as a AI language model" disclaimer. Out of curiosity, is this model meant to be a 25 year old personal assistant?

Re: Alpaca-LoRA with Docker

#16

Post by zapdrive »

Sorry this is moving too fast for me. So if I understand correctly, LoRa kind of does what Alpaca does but using different data.

So what is Alpaca-Lora? I know you get Alpaca by retraining Llama using Stanford Alpaca 52k instruction-following data? So if I am guessing right, you get Aplaca-Lora by retraining Alpaca using Lora's data?

Re: Alpaca-LoRA with Docker

#18

Post by return_to_monke »

zapdrive wrote: post #16

Sorry this is moving too fast for me. So if I understand correctly, LoRa kind of does what Alpaca does but using different data. So what is Alpaca-Lora? I know you get Alpaca by retraining Llama using Stanford Alpaca 52k instruction-following data? So if I am guessing right, you get Aplaca-Lora by retraining Alpaca using Lora's data?

I think your first statement is incorrect. Lora seems to be a method to fine-tune and optimize the weights of models like Alpaca. It is not a different dataset.

This reduces model sizes and therefore also compute costs.

See the abstract of https://arxiv.org/pdf/2106.09685.pdf

Re: Alpaca-LoRA with Docker

#19

Post by kkielhofner »

Ok, this is the base for actually self-hosted production use of these things now (if you don't care about licensing...). I've said in previous HN comments we've been a Dockerfile using an Nvidia base image away from this for a while now (just never got around to it myself).

I love the .ccp, Apple Silicon, etc projects but IMO for the time being Nvidia is still king when it comes to multi-user production use of these models with competitive response time, parameter count/size, etc.

Of course as others pointed out the quality of these models still leaves a lot to be desired but this is a good start for the inevitable actually open models, finetuned variants, etc that are being released on what seems like a daily basis at this point.

I'm walking through it (fun weekend project!) but my dual RTX 4090 dev workstation will almost certainly scream with these (even though VRAM isn't "great"). Over time with better and better models (with compatible licenses) the OpenAI lead will get smaller and smaller.

Re: Alpaca-LoRA with Docker

#20

Post by cuuupid »

kkielhofner wrote: post #19

Ok, this is the base for actually self-hosted production use of these things now (if you don't care about licensing...). I've said in previous HN comments we've been a Dockerfile using an Nvidia base image away from this for a while now (just never got around to it myself). I love the .ccp, Apple Silicon, etc projects but IMO for the time being Nvidia is still king when it comes to multi-user production use of these…

I’m hitting ChatGPT or faster speeds on my 3090. Have it running the image with a reverse SSH tunnel to an EC2 instance that’s ferrying requests from the web. It only took 4 hours of an afternoon, and based off the trending Databricks article on HN we’re probably only days away from a commercially licensed model.
Post reply on HN