Ask HN: Who is using small OS LLMs in production? • Hacker News Board

Re: Ask HN: Who is using small OS LLMs in production?

#31

Post by binarymax » Wed, Aug 02, 2023, 1:56 PM UTC

I see comments here about running Llama on 4090's, which is fine for local development and testing - but getting into production is a significant leap and a significant cost.

The thing that I keep running into in my SLA plans is concurrency. Yes, you can have a Llama 2 model running on an A100 somewhere - but that will support 1 concurrent prompt. Anything at a higher concurrency needs another GPU, or your end users will be waiting a while. Want to rent an 8 GPU machine in the cloud for inference? Be prepared to pay a lot of money for it.

Re: Ask HN: Who is using small OS LLMs in production?

#32

Post by smallerfish » Wed, Aug 02, 2023, 1:57 PM UTC

metiscus wrote: post #13

LewisVerstappen wrote: post #7

Earlier quoted context omitted.

Can you elaborate on the pricing difference?

LLAMA 2 appears to require no cost because it can be run locally. The license does mention that if you are using facilitating than 700 million users that you have to negotiate for a different license or something like that but for most peoples uses, it would seem that LLAMA 2 is basically "free".

Right, but can you scale GPUs for cheaper than OpenAI charges to use their APIs? 3.5 is _cheap_, and perfectly good for many use cases.

Re: Ask HN: Who is using small OS LLMs in production?

#33

Post by ramesh31 » Wed, Aug 02, 2023, 2:01 PM UTC

jeanloolz wrote: post #27

ramesh31 wrote: post #23

Running llama-2-7b-chat at 8 bit quantization, and completions are essentially at GPT-3.5 levels (and instant) on a single RTX4090 using 15gb VRAM. I don't think most people realize just how small and efficient these models are going to become.

7B or 70B?

>7B or 70B?

7B 8bit GGML running on a single 4090 with llama.cpp. It's hard to overstate the massive jump in capability between llama 1 and 2.

Re: Ask HN: Who is using small OS LLMs in production?

#34

Post by [deleted] » Wed, Aug 02, 2023, 2:03 PM UTC

pavlov wrote: post #30

TradingPlaces wrote: post #17

Facebook is working very hard to make the main dividing line in generative AI not company vs. company but commercial vs free. Starting from way behind, they are trying to make that irrelevant.

“Way behind” seems harsh when they have one of the best models available.

[deleted]

Re: Ask HN: Who is using small OS LLMs in production?

#35

Post by rimeice » Wed, Aug 02, 2023, 2:06 PM UTC

alrlroipsp wrote: post #24

LewisVerstappen wrote: post #8

Earlier quoted context omitted.

Can you elaborate on the pricing difference?

Free vs monthly cost. What is there to elaborate on?

It’s gona have to be hosted and run from somewhere…

Re: Ask HN: Who is using small OS LLMs in production?

#36

Post by huac » Wed, Aug 02, 2023, 2:08 PM UTC

binarymax wrote: post #31

I see comments here about running Llama on 4090's, which is fine for local development and testing - but getting into production is a significant leap and a significant cost. The thing that I keep running into in my SLA plans is concurrency. Yes, you can have a Llama 2 model running on an A100 somewhere - but that will support 1 concurrent prompt. Anything at a higher concurrency needs another GPU, or your end users…

you need an inference server. I am doing ~400 tokens/sec on 7B with a 4090 with multiple concurrent (streaming!) requests.

it's reasonably straightforward for me to host this and serve public requests, but would likely just be a base model -- not sure if hosting (eg) 13B chat can serve peoples' use cases

Re: Ask HN: Who is using small OS LLMs in production?

#37

Post by rimeice » Wed, Aug 02, 2023, 2:09 PM UTC

ramesh31 wrote: post #33

jeanloolz wrote: post #27

Earlier quoted context omitted.

7B or 70B?

>7B or 70B? 7B 8bit GGML running on a single 4090 with llama.cpp. It's hard to overstate the massive jump in capability between llama 1 and 2.

Are you hosting that somewhere? If so, how much does that cost and do you have concurrent users?

Re: Ask HN: Who is using small OS LLMs in production?

#38

Post by [deleted] » Wed, Aug 02, 2023, 2:10 PM UTC

smallerfish wrote: post #32

metiscus wrote: post #13

Earlier quoted context omitted.

LLAMA 2 appears to require no cost because it can be run locally. The license does mention that if you are using facilitating than 700 million users that you have to negotiate for a different license or something like that but for most peoples uses, it would seem that LLAMA 2 is basically "free".

Right, but can you scale GPUs for cheaper than OpenAI charges to use their APIs? 3.5 is _cheap_, and perfectly good for many use cases.

[deleted]

Re: Ask HN: Who is using small OS LLMs in production?

#39

Post by binarymax » Wed, Aug 02, 2023, 2:11 PM UTC

huac wrote: post #36

binarymax wrote: post #31

I see comments here about running Llama on 4090's, which is fine for local development and testing - but getting into production is a significant leap and a significant cost. The thing that I keep running into in my SLA plans is concurrency. Yes, you can have a Llama 2 model running on an A100 somewhere - but that will support 1 concurrent prompt. Anything at a higher concurrency needs another GPU, or your end users…

you need an inference server. I am doing ~400 tokens/sec on 7B with a 4090 with multiple concurrent (streaming!) requests. it's reasonably straightforward for me to host this and serve public requests, but would likely just be a base model -- not sure if hosting (eg) 13B chat can serve peoples' use cases

But is the 7B model any good and actually production worthy for things like RAG?

Re: Ask HN: Who is using small OS LLMs in production?

#40

Post by treprinum » Wed, Aug 02, 2023, 2:11 PM UTC

I am using one of the uncensored versions of LLaMA 2 to allow chatbot roleplays without constant moralizing and replying to every other request with "I am just an AI, I don't have any opinion, emotion, feelings, don't like anything" etc.