Ask HN: Who is using small OS LLMs in production? • Hacker News Board

Re: Ask HN: Who is using small OS LLMs in production?

#21

Post by metiscus » Wed, Aug 02, 2023, 1:39 PM UTC

moffkalast wrote: post #18

metiscus wrote: post #13

Earlier quoted context omitted.

LLAMA 2 appears to require no cost because it can be run locally. The license does mention that if you are using facilitating than 700 million users that you have to negotiate for a different license or something like that but for most peoples uses, it would seem that LLAMA 2 is basically "free".

Well it's free as in free hops. You still gotta buy/rent the brewery to make the free beer.

Exactly. It isn't free as in FSF definitional freedom, you still can't do certain things with it etc but it is unencumbered by external costs unless you exceed a certain usage threshold.

Re: Ask HN: Who is using small OS LLMs in production?

#22

Post by microtonal » Wed, Aug 02, 2023, 1:40 PM UTC

LewisVerstappen wrote: post #7

boredumb wrote: post #5

I'm integrating llama 2 7b with an application I'm building out currently and one of the biggest reasons was privacy, followed closely by price and lastly by getting it to work locally in a few minutes. I built a now abandoned project using the GPT API and it was fine and not terribly expensive for my use case but customers didn't like the pay for usage model and the alternative was do weird UX to limit people abusin…

Can you elaborate on the pricing difference?

The long-term price difference can be hard to estimate. Suppose that your application is heavily dependent on GPT-4. OpenAI can double your API prices the next term. Or they can decide to stop supporting the model that you wrote all your custom prompts for. Or they could decide to disable your account because they feel like it.

Unless it's for some fringe feature, building your business on OpenAI is probably a considerable (financial) risk in the future.

Re: Ask HN: Who is using small OS LLMs in production?

#23

Post by ramesh31 » Wed, Aug 02, 2023, 1:40 PM UTC

Running llama-2-7b-chat at 8 bit quantization, and completions are essentially at GPT-3.5 levels (and instant) on a single RTX4090 using 15gb VRAM. I don't think most people realize just how small and efficient these models are going to become.

Re: Ask HN: Who is using small OS LLMs in production?

#24

Post by alrlroipsp » Wed, Aug 02, 2023, 1:40 PM UTC

LewisVerstappen wrote: post #8

Demotooodo wrote: post #2

Price. Data privacy. Controlled latency. Plenty of reasons to not send arbitrary data to a third party service.

Can you elaborate on the pricing difference?

Free vs monthly cost. What is there to elaborate on?

Re: Ask HN: Who is using small OS LLMs in production?

#25

Post by moffkalast » Wed, Aug 02, 2023, 1:42 PM UTC

ewuhic wrote: post #11

Tangential question - how well does Llama 2 do on coding tasks on less-mainstream languages like Rust?

Well I'm not too familiar with Rust so I can't gauge correctness, but I do have the Llama 2 13B NewHope fine tune loaded (which is afaik tuned for python coding), so I gave it and 3.5-turbo the same random post request question.

3.5's result: https://chat.openai.com/share/9e1aafd3-631c-4c13-80f6-f99c88...

NewHope's result: https://i.imgur.com/dfACQC3.png

If you have any ideas for a more comprehensive test let me know and I'll try to run it. Giving it some existing code to fix up or change is usually more of a typical use case for me anyway.

Re: Ask HN: Who is using small OS LLMs in production?

#26

Post by lolinder » Wed, Aug 02, 2023, 1:46 PM UTC

tycoon177 wrote: post #4

Demotooodo wrote: post #2

Price. Data privacy. Controlled latency. Plenty of reasons to not send arbitrary data to a third party service.

There's also the availability factor. OpenAI has been known to go down on occasion and without warning. If a product relies on an LLM, I wouldn't feel great about the observed uptime of OpenAI APIs.

Another side of availability is that they'll make changes to the model without warning, which alters the results of the prompts you already have written. Developing against their API is developing against a moving target.

Re: Ask HN: Who is using small OS LLMs in production?

#27

Post by jeanloolz » Wed, Aug 02, 2023, 1:49 PM UTC

ramesh31 wrote: post #23

Running llama-2-7b-chat at 8 bit quantization, and completions are essentially at GPT-3.5 levels (and instant) on a single RTX4090 using 15gb VRAM. I don't think most people realize just how small and efficient these models are going to become.

7B or 70B?

Re: Ask HN: Who is using small OS LLMs in production?

#28

Post by npsomaratna » Wed, Aug 02, 2023, 1:51 PM UTC

fogx wrote: post #16

npsomaratna wrote: post #6

Data security and privacy. Our clients (in aviation, finance, etc.) need this due to legal and regulatory reasons. Also, the new Llama 2 models are very powerful. In my testing, Llama 2 70b is comparable to GPT-3.5 in capability. (Shameless plug: here's our website: https://www.amw.ai/ )

have you considered azure's GPT, or is that not private enough?

We have. This is acceptable for some clients, but not for others. Both groups, however, prefer maintaining complete control over their data, given the chance.

Edit: plus, my personal view is that local LLMs are the future. They've already caught up to GPT-3.5 (based on my testing); and they continue to evolve rapidly. Makes sense to focus our limited resources on riding that wave.

OpenAI won't go away, but neither will they remain the first choice (or only choice!) for most use-cases.

Re: Ask HN: Who is using small OS LLMs in production?

#29

Post by boredumb » Wed, Aug 02, 2023, 1:53 PM UTC

LewisVerstappen wrote: post #14

boredumb wrote: post #12

Earlier quoted context omitted.

OpenAI costs money and llama 2 i'm able to run on my GPU so at least for development purposes at the moment it's ""free"" for me to experiment with - mind you this is a side project with zero funding outside of myself, ymmv if you have access to funding to a point of making OpenAI tokens totally disposable.

Ah okay, yeah I was curious how much you'd pay running it on a cloud server in a production-type scenario. Thanks

I have a vultr account I just logged into and checked, I could rent enough GPU memory to run llama 2 for ~180$ a month from my very quick look - so if I make more than 27000 requests with a 4k token payload I will break even - otherwise I could be using openAIs API and be making out better.

EDIT** im sorry that was gpt3.5 - with gpt4 it would be 1000 requests before I broke even at 180$

Re: Ask HN: Who is using small OS LLMs in production?

#30

Post by pavlov » Wed, Aug 02, 2023, 1:54 PM UTC

TradingPlaces wrote: post #17

Facebook is working very hard to make the main dividing line in generative AI not company vs. company but commercial vs free. Starting from way behind, they are trying to make that irrelevant.

“Way behind” seems harsh when they have one of the best models available.