It is not clear whether relatively small open source LLMs in production or not, for instance replit code 3b, llama 2 7b, codegen,... What are the motivations for using those models over prompted GPT api ?
Ask HN: Who is using small OS LLMs in production?
1–10 of 59 posts
Re: Ask HN: Who is using small OS LLMs in production?
#2Price.
Data privacy.
Controlled latency.
Plenty of reasons to not send arbitrary data to a third party service.
Re: Ask HN: Who is using small OS LLMs in production?
#3We have several customers who aren't using OpenAI API / Anthropic API because of privacy reasons. We are spinning up infrastructure and making the features who rely on those APIs to also wok with OS LLMs too.
Re: Ask HN: Who is using small OS LLMs in production?
#4Price. Data privacy. Controlled latency. Plenty of reasons to not send arbitrary data to a third party service.
There's also the availability factor. OpenAI has been known to go down on occasion and without warning. If a product relies on an LLM, I wouldn't feel great about the observed uptime of OpenAI APIs.
Re: Ask HN: Who is using small OS LLMs in production?
#5I'm integrating llama 2 7b with an application I'm building out currently and one of the biggest reasons was privacy, followed closely by price and lastly by getting it to work locally in a few minutes.
I built a now abandoned project using the GPT API and it was fine and not terribly expensive for my use case but customers didn't like the pay for usage model and the alternative was do weird UX to limit people abusing the prompts into something I couldn't afford bootstrapping as a side project.
Re: Ask HN: Who is using small OS LLMs in production?
#6Data security and privacy. Our clients (in aviation, finance, etc.) need this due to legal and regulatory reasons. Also, the new Llama 2 models are very powerful. In my testing, Llama 2 70b is comparable to GPT-3.5 in capability.
(Shameless plug: here's our website: https://www.amw.ai/)
Re: Ask HN: Who is using small OS LLMs in production?
#7I'm integrating llama 2 7b with an application I'm building out currently and one of the biggest reasons was privacy, followed closely by price and lastly by getting it to work locally in a few minutes. I built a now abandoned project using the GPT API and it was fine and not terribly expensive for my use case but customers didn't like the pay for usage model and the alternative was do weird UX to limit people abusin…
Can you elaborate on the pricing difference?
Re: Ask HN: Who is using small OS LLMs in production?
#8Price. Data privacy. Controlled latency. Plenty of reasons to not send arbitrary data to a third party service.
Can you elaborate on the pricing difference?
Re: Ask HN: Who is using small OS LLMs in production?
#9My friend trained his own gpt2 because it is faster and cheaper to tune it.
Re: Ask HN: Who is using small OS LLMs in production?
#10Data security and privacy. Our clients (in aviation, finance, etc.) need this due to legal and regulatory reasons. Also, the new Llama 2 models are very powerful. In my testing, Llama 2 70b is comparable to GPT-3.5 in capability. (Shameless plug: here's our website: https://www.amw.ai/ )
Looks great my friend