Ask HN: Who is using small OS LLMs in production? • Hacker News Board

Re: Ask HN: Who is using small OS LLMs in production?

#41

Post by derealized » Wed, Aug 02, 2023, 2:11 PM UTC

To me, the simple models might not cross the boundary where LLMs start to be useful versus, say, a fixed menu with choices in a helpdesk app.

It's a paradox because, to really feel human like and not make huge mistakes, we need these huge LLMs and they are expensive... and the alternative is not-so-smart traditional code.

So what I'm trying to say is that I think the small LLMs might not be that useful before they cross some arbitrary quality threshold (which they may never do.. considering more parameters => better model, in general).

Re: Ask HN: Who is using small OS LLMs in production?

#42

Post by treprinum » Wed, Aug 02, 2023, 2:12 PM UTC

binarymax wrote: post #31

I see comments here about running Llama on 4090's, which is fine for local development and testing - but getting into production is a significant leap and a significant cost. The thing that I keep running into in my SLA plans is concurrency. Yes, you can have a Llama 2 model running on an A100 somewhere - but that will support 1 concurrent prompt. Anything at a higher concurrency needs another GPU, or your end users…

LLaMA 2 7B 8-bit can run pretty well on 64 core EPYCs which are cheaper than GPU instances. Moreover, you can periodically batch multiple users and not just run a single inference for a single user.

Re: Ask HN: Who is using small OS LLMs in production?

#43

Post by MilStdJunkie » Wed, Aug 02, 2023, 2:13 PM UTC

fogx wrote: post #16

npsomaratna wrote: post #6

Data security and privacy. Our clients (in aviation, finance, etc.) need this due to legal and regulatory reasons. Also, the new Llama 2 models are very powerful. In my testing, Llama 2 70b is comparable to GPT-3.5 in capability. (Shameless plug: here's our website: https://www.amw.ai/ )

have you considered azure's GPT, or is that not private enough?

There's some . . entrepreneurs . . who have been promising NIST/ITAR-compliant LLM frameworks on Azure, but when you ask around, they have not done all the legwork (AG/AGS). They're working off Azure Public, with "waivers" that they won't show anyone. Also, the history of their leadership is . . questionable. It all feels just a little hinky. Until that's cleared up, I advise anyone fooling with LLMs to do it on-prem, at least for the moment. One thing I'm worried about: doing LLMs with something like GovCloud is going to be absolutely bananas in terms of price-per-compute.

Re: Ask HN: Who is using small OS LLMs in production?

#44

Post by lumost » Wed, Aug 02, 2023, 2:13 PM UTC

Exploring a few options in my off-time. Main motivation for an OS LLM is to get it to do things which GPT-3.5/4 are somewhat promising at - but not good enough for applications.

Re: Ask HN: Who is using small OS LLMs in production?

#45

Post by phillipcarter » Wed, Aug 02, 2023, 2:13 PM UTC

tycoon177 wrote: post #4

Demotooodo wrote: post #2

Price. Data privacy. Controlled latency. Plenty of reasons to not send arbitrary data to a third party service.

There's also the availability factor. OpenAI has been known to go down on occasion and without warning. If a product relies on an LLM, I wouldn't feel great about the observed uptime of OpenAI APIs.

FWIW, OpenAI's availability seems to have gotten significantly better since May when we launched with them. I monitor our availability Service Level Objective and we keep needing to increase the success rate because they keep improving things.

This doesn't take away from high availability being a legitimate need to host your own LLM, though.

Re: Ask HN: Who is using small OS LLMs in production?

#46

Post by TradingPlaces » Wed, Aug 02, 2023, 2:14 PM UTC

pavlov wrote: post #30

TradingPlaces wrote: post #17

Facebook is working very hard to make the main dividing line in generative AI not company vs. company but commercial vs free. Starting from way behind, they are trying to make that irrelevant.

“Way behind” seems harsh when they have one of the best models available.

Not that impressed with llama 2 70b so far tbh. It’s a GPT3-level bullshit machine imo. But huge advantages in running privately and at the edge, so that’s going to be the dividing line imo. Commercial v free. Small v big. H100s in the cloud v edge

Re: Ask HN: Who is using small OS LLMs in production?

#47

Post by treprinum » Wed, Aug 02, 2023, 2:14 PM UTC

binarymax wrote: post #39

huac wrote: post #36

Earlier quoted context omitted.

you need an inference server. I am doing ~400 tokens/sec on 7B with a 4090 with multiple concurrent (streaming!) requests. it's reasonably straightforward for me to host this and serve public requests, but would likely just be a base model -- not sure if hosting (eg) 13B chat can serve peoples' use cases

But is the 7B model any good and actually production worthy for things like RAG?

Not as good as GPT-4 of course but nor far from 3.5 if you just need to reword whatever returned by the retrieval. It's like losing 20 IQ points which might be still better than most support interactions I had.

Re: Ask HN: Who is using small OS LLMs in production?

#48

Post by phillipcarter » Wed, Aug 02, 2023, 2:16 PM UTC

alrlroipsp wrote: post #24

LewisVerstappen wrote: post #8

Earlier quoted context omitted.

Can you elaborate on the pricing difference?

Free vs monthly cost. What is there to elaborate on?

Hosting your own LLM is anything but free. Aside from the constant operational expense with people monitoring and fixing issues, you need to provision enough resources and run your own inference server, which is both nontrivial and likely to perform far worse than OpenAI. There's legitimate reasons to host an LLM yourself, but it's not a "make this cheaper" button.

Re: Ask HN: Who is using small OS LLMs in production?

#49

Post by ramesh31 » Wed, Aug 02, 2023, 2:19 PM UTC

rimeice wrote: post #37

ramesh31 wrote: post #33

Earlier quoted context omitted.

>7B or 70B? 7B 8bit GGML running on a single 4090 with llama.cpp. It's hard to overstate the massive jump in capability between llama 1 and 2.

Are you hosting that somewhere? If so, how much does that cost and do you have concurrent users?

>Are you hosting that somewhere?

Tensordock. RTX4090 instances are ~$0.50/hr and can handle 3/4 concurrent users each.

Re: Ask HN: Who is using small OS LLMs in production?

#50

Post by huac » Wed, Aug 02, 2023, 2:21 PM UTC

binarymax wrote: post #39

huac wrote: post #36

Earlier quoted context omitted.

you need an inference server. I am doing ~400 tokens/sec on 7B with a 4090 with multiple concurrent (streaming!) requests. it's reasonably straightforward for me to host this and serve public requests, but would likely just be a base model -- not sure if hosting (eg) 13B chat can serve peoples' use cases

But is the 7B model any good and actually production worthy for things like RAG?

I'm writing a blog post with some more reasoning but my view is that it can be useful for certain simpler tasks (eg unstructured -> structured, basic summarization) and not more complex things (eg generation).

The tricky thing is that finetuning makes a big difference, and while it should be possible to hotswap LoRA adapters (at some cost to performance), I haven't figured that out yet.