Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alpaca-LoRA with Docker

github.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Alpaca-LoRA with Docker

#21

Post by zamalek »

syntaxing wrote: post #8
nico wrote: post #6

The demo on HuggingFace with the pre trained mode doesn’t seem that good. Although better than Bard (btw, Bard sucks compared to ChatGPT and can’t even do translations - which I would have expected out of the box from Google)

It’s worth noting this is the 7B model (nonquantized). You can get this running on pretty much any GPU with 8GB VRAM and above. You can run the 13B model but that would take two GPU or reducing FP16 to FP8 (I haven’t tried it myself). A single connection for chatgpt is rumored to require 8X A100.

Quantizing it to 8-bit basically eliminates its ability to write code.

Re: Alpaca-LoRA with Docker

#22

Post by jonny_eh »

HnUser12 wrote: post #14

> Tell me about you >I am a 25-year-old woman from the United States. I have a bachelor's degree in computer science and am currently pursuing a master's degree in data science. I am passionate about technology and am always looking for new ways to use it to make the world a better place. Outside of work, I enjoy spending time with my family and friends, reading, and traveling. Well, I was starting to get tired of "a…

No, it's just random "plausible" response. Re-roll the response and you'll get something different.

Think of the prompt as "pretend you're some random person, tell me some details"

Re: Alpaca-LoRA with Docker

#24

Post by kkielhofner »

cuuupid wrote: post #20
kkielhofner wrote: post #19

Ok, this is the base for actually self-hosted production use of these things now (if you don't care about licensing...). I've said in previous HN comments we've been a Dockerfile using an Nvidia base image away from this for a while now (just never got around to it myself). I love the .ccp, Apple Silicon, etc projects but IMO for the time being Nvidia is still king when it comes to multi-user production use of these…

I’m hitting ChatGPT or faster speeds on my 3090. Have it running the image with a reverse SSH tunnel to an EC2 instance that’s ferrying requests from the web. It only took 4 hours of an afternoon, and based off the trending Databricks article on HN we’re probably only days away from a commercially licensed model.

Bit of a tangent, have you tried CloudFlare tunnels for what you're doing? Literally one liner to install cloudflared and boom service is on the internet with Cloudflare in front. I've even used it in cases where my host was behind multiple layers of NAT - just works. If you're concerned with speed and performance I guarantee it will blow away your current approach (while giving you all of the other Cloudflare stuff). Of course if you hate CF (fair enough) disregard :).

I use this for an optimized hosted Whisper implementation I've been working on. It hits 120x realtime with large v2 on a 4090 and uses WebRTC to stream the audio in realtime with datachannels for ASR responses. Hopefully a "Show HN" soon once I get some legal stuff out of the way :). I mention it because AFAIK it's many multiples faster than the OpenAI hosted Whisper (especially for "realtime" speech).

I expect we'll see these kinds of innovations and more come to self-hosted approaches generally and the open source community will pull a web hosting, etc Microsoft vs Linux/LAMP/etc 1990s/early 2000s situation on OpenAI where open source wins in the end. The fact that MS is so heavily invested in OpenAI is just history repeating itself.

Yep, saw the Databricks article! I don't try to make specific time predictions but you're probably not far off :).

Re: Alpaca-LoRA with Docker

#26

Post by Havoc »

nico wrote: post #11
syntaxing wrote: post #8

Earlier quoted context omitted.

It’s worth noting this is the 7B model (nonquantized). You can get this running on pretty much any GPU with 8GB VRAM and above. You can run the 13B model but that would take two GPU or reducing FP16 to FP8 (I haven’t tried it myself). A single connection for chatgpt is rumored to require 8X A100.

It makes me wonder if this trend will kill NVIDIA. At this pace we might not even need GPUs anymore.

Quite the opposite. This entire trend is a godsend for them. If anything it opens more markets in lower powered tiers

Re: Alpaca-LoRA with Docker

#27

Post by Taek »

zamalek wrote: post #21
syntaxing wrote: post #8

Earlier quoted context omitted.

It’s worth noting this is the 7B model (nonquantized). You can get this running on pretty much any GPU with 8GB VRAM and above. You can run the 13B model but that would take two GPU or reducing FP16 to FP8 (I haven’t tried it myself). A single connection for chatgpt is rumored to require 8X A100.

Quantizing it to 8-bit basically eliminates its ability to write code.

?

All the research I've seen says quantization has basically negligible performance impact. My experience working with 65B 4bit has been great

Re: Alpaca-LoRA with Docker

#28

Post by syntaxing »

Taek wrote: post #27
zamalek wrote: post #21

Earlier quoted context omitted.

Quantizing it to 8-bit basically eliminates its ability to write code.

? All the research I've seen says quantization has basically negligible performance impact. My experience working with 65B 4bit has been great

I agree with OP. Only the nonquantized models has given me good results too. I only have used the 7B and 13B. I don't have enough computing power to run 65B.

Re: Alpaca-LoRA with Docker

#29

Post by DANmode »

nico wrote: post #6

The demo on HuggingFace with the pre trained mode doesn’t seem that good. Although better than Bard (btw, Bard sucks compared to ChatGPT and can’t even do translations - which I would have expected out of the box from Google)

Have you tried OpenAssistant?

Re: Alpaca-LoRA with Docker

#30

Post by NLPaep »

MacsHeadroom wrote: post #5
sp332 wrote: post #3

This says "We provide an Instruct model of similar quality to text-davinci-003", but two paragraphs later says the output is comparable to Stanford's Alpaca. Those seem like very different claims.

"We performed a blind pairwise comparison between text-davinci-003 and Alpaca 7B, and we found that these two models have very similar performance: Alpaca wins 90 versus 89 comparisons against text-davinci-003." https://crfm.stanford.edu/2023/03/13/alpaca.html

Except when ChatGPT wins, it wins hard
Post reply on HN