Tangential question - how well does Llama 2 do on coding tasks on less-mainstream languages like Rust?
Ask HN: Who is using small OS LLMs in production?
11–20 of 59 posts
Re: Ask HN: Who is using small OS LLMs in production?
#12I'm integrating llama 2 7b with an application I'm building out currently and one of the biggest reasons was privacy, followed closely by price and lastly by getting it to work locally in a few minutes. I built a now abandoned project using the GPT API and it was fine and not terribly expensive for my use case but customers didn't like the pay for usage model and the alternative was do weird UX to limit people abusin…
Can you elaborate on the pricing difference?
OpenAI costs money and llama 2 i'm able to run on my GPU so at least for development purposes at the moment it's ""free"" for me to experiment with - mind you this is a side project with zero funding outside of myself, ymmv if you have access to funding to a point of making OpenAI tokens totally disposable.
Re: Ask HN: Who is using small OS LLMs in production?
#13I'm integrating llama 2 7b with an application I'm building out currently and one of the biggest reasons was privacy, followed closely by price and lastly by getting it to work locally in a few minutes. I built a now abandoned project using the GPT API and it was fine and not terribly expensive for my use case but customers didn't like the pay for usage model and the alternative was do weird UX to limit people abusin…
Can you elaborate on the pricing difference?
LLAMA 2 appears to require no cost because it can be run locally. The license does mention that if you are using facilitating than 700 million users that you have to negotiate for a different license or something like that but for most peoples uses, it would seem that LLAMA 2 is basically "free".
Re: Ask HN: Who is using small OS LLMs in production?
#14Earlier quoted context omitted.
Can you elaborate on the pricing difference?
OpenAI costs money and llama 2 i'm able to run on my GPU so at least for development purposes at the moment it's ""free"" for me to experiment with - mind you this is a side project with zero funding outside of myself, ymmv if you have access to funding to a point of making OpenAI tokens totally disposable.
Ah okay, yeah I was curious how much you'd pay running it on a cloud server in a production-type scenario. Thanks
Re: Ask HN: Who is using small OS LLMs in production?
#15Earlier quoted context omitted.
Can you elaborate on the pricing difference?
LLAMA 2 appears to require no cost because it can be run locally. The license does mention that if you are using facilitating than 700 million users that you have to negotiate for a different license or something like that but for most peoples uses, it would seem that LLAMA 2 is basically "free".
Thanks for elaborating. Yeah I was curious how much you'd pay running it on a cloud server in a production-type scenario. Thanks
Re: Ask HN: Who is using small OS LLMs in production?
#16Data security and privacy. Our clients (in aviation, finance, etc.) need this due to legal and regulatory reasons. Also, the new Llama 2 models are very powerful. In my testing, Llama 2 70b is comparable to GPT-3.5 in capability. (Shameless plug: here's our website: https://www.amw.ai/ )
have you considered azure's GPT, or is that not private enough?
Re: Ask HN: Who is using small OS LLMs in production?
#17Facebook is working very hard to make the main dividing line in generative AI not company vs. company but commercial vs free. Starting from way behind, they are trying to make that irrelevant.
Re: Ask HN: Who is using small OS LLMs in production?
#18Earlier quoted context omitted.
Can you elaborate on the pricing difference?
LLAMA 2 appears to require no cost because it can be run locally. The license does mention that if you are using facilitating than 700 million users that you have to negotiate for a different license or something like that but for most peoples uses, it would seem that LLAMA 2 is basically "free".
Well it's free as in free hops. You still gotta buy/rent the brewery to make the free beer.
Re: Ask HN: Who is using small OS LLMs in production?
#19Customizing logists
Re: Ask HN: Who is using small OS LLMs in production?
#20Data security and privacy. Our clients (in aviation, finance, etc.) need this due to legal and regulatory reasons. Also, the new Llama 2 models are very powerful. In my testing, Llama 2 70b is comparable to GPT-3.5 in capability. (Shameless plug: here's our website: https://www.amw.ai/ )
have you considered azure's GPT, or is that not private enough?
This assumes you trust Microsoft.