Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who is using small OS LLMs in production?

news.ycombinator.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Ask HN: Who is using small OS LLMs in production?

#51

Post by fauigerzigerk »

microtonal wrote: post #22
LewisVerstappen wrote: post #7

Earlier quoted context omitted.

Can you elaborate on the pricing difference?

The long-term price difference can be hard to estimate. Suppose that your application is heavily dependent on GPT-4. OpenAI can double your API prices the next term. Or they can decide to stop supporting the model that you wrote all your custom prompts for. Or they could decide to disable your account because they feel like it. Unless it's for some fringe feature, building your business on OpenAI is probably a consid…

These are definitely very significant risks. But some of those risks are hard to avoid unless you train your own model, which can be prohibitively expensive.

Say you're building on top of llama and Facebook decides not to update it any more or change the licensing terms (again). Say you're building on some other "open source" model and that project dies.

At least you can keep running the existing model rather than getting locked out over night. That's definitely much better. At least it's survivable. But you would still have to find alternatives and review/scrap all your custom prompts.

Re: Ask HN: Who is using small OS LLMs in production?

#52

Post by andy99 »

npsomaratna wrote: post #6

Data security and privacy. Our clients (in aviation, finance, etc.) need this due to legal and regulatory reasons. Also, the new Llama 2 models are very powerful. In my testing, Llama 2 70b is comparable to GPT-3.5 in capability. (Shameless plug: here's our website: https://www.amw.ai/ )

I'm in the same boat. We have customers that need to run models in an environment that have access to the public internet, even if they did trust OpenAI et al.

More importantly for me, I don't want to be beholden to a model provider and have to take what they give me. I'd rather host my own model if an API was an option, because then I have control over it and can hack it as I want. I don't want to be just a wrapper on GPT which is sort of what you're stuck with if you just want to use their APIs.

Re: Ask HN: Who is using small OS LLMs in production?

#53

Post by phillipcarter »

Although we haven't gone down the path of deploying a fine-tuned model on our own infrastructure, we do see that as an eventual reality. Our current feature is disabled for any customer who signs a BAA with us because we can't get a DPA signed with OpenAI, and not for lack of trying. Maybe that resolves itself over time, but the most reliable option available is to fine-tune a model and run it ourselves. It's also likely a more expensive and challenging one, though, hence we're not doing it yet.

Re: Ask HN: Who is using small OS LLMs in production?

#54

Post by kuchenbecker »

phillipcarter wrote: post #48
alrlroipsp wrote: post #24

Earlier quoted context omitted.

Free vs monthly cost. What is there to elaborate on?

Hosting your own LLM is anything but free. Aside from the constant operational expense with people monitoring and fixing issues, you need to provision enough resources and run your own inference server, which is both nontrivial and likely to perform far worse than OpenAI. There's legitimate reasons to host an LLM yourself, but it's not a "make this cheaper" button.

There may be a tipping point where you're burning XXM/year in API costs and the maintenance cost of rolling your own can be justified.

In the short term I agree, and one thing to consider is how rapidly the space is evolving and whether your team can even keep up with the latest advancements.

However, there will come a time when the bill comes due after launch and it will be very tempting to hire people to reduce the CapEx on the API.

Re: Ask HN: Who is using small OS LLMs in production?

#55

Post by npsomaratna »

andy99 wrote: post #52
npsomaratna wrote: post #6

Data security and privacy. Our clients (in aviation, finance, etc.) need this due to legal and regulatory reasons. Also, the new Llama 2 models are very powerful. In my testing, Llama 2 70b is comparable to GPT-3.5 in capability. (Shameless plug: here's our website: https://www.amw.ai/ )

I'm in the same boat. We have customers that need to run models in an environment that have access to the public internet, even if they did trust OpenAI et al. More importantly for me, I don't want to be beholden to a model provider and have to take what they give me. I'd rather host my own model if an API was an option, because then I have control over it and can hack it as I want. I don't want to be just a wrapper…

Agreed. Worse, you don't want to be in a position where any single company can end your business at the flip of a switch.

Re: Ask HN: Who is using small OS LLMs in production?

#56

Post by kristianp »

moffkalast wrote: post #25
ewuhic wrote: post #11

Tangential question - how well does Llama 2 do on coding tasks on less-mainstream languages like Rust?

Well I'm not too familiar with Rust so I can't gauge correctness, but I do have the Llama 2 13B NewHope fine tune loaded (which is afaik tuned for python coding), so I gave it and 3.5-turbo the same random post request question. 3.5's result: https://chat.openai.com/share/9e1aafd3-631c-4c13-80f6-f99c88... NewHope's result: https://i.imgur.com/dfACQC3.png If you have any ideas for a more comprehensive test let me know…

Interestingly, they have withdrawn their model because they discovered that test data leaked into the training data. Quantised versions are still available on huggingface from others though.

https://github.com/SLAM-group/newhope

Re: Ask HN: Who is using small OS LLMs in production?

#57

Post by moffkalast »

kristianp wrote: post #56
moffkalast wrote: post #25

Earlier quoted context omitted.

Well I'm not too familiar with Rust so I can't gauge correctness, but I do have the Llama 2 13B NewHope fine tune loaded (which is afaik tuned for python coding), so I gave it and 3.5-turbo the same random post request question. 3.5's result: https://chat.openai.com/share/9e1aafd3-631c-4c13-80f6-f99c88... NewHope's result: https://i.imgur.com/dfACQC3.png If you have any ideas for a more comprehensive test let me know…

Interestingly, they have withdrawn their model because they discovered that test data leaked into the training data. Quantised versions are still available on huggingface from others though. https://github.com/SLAM-group/newhope

Yeah it seemed suspiciously high for HumanEval and it only ranks 14th for JS and 7th for Python on other benchmarks now: https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul...

WizardCoder is a bit of a problem since it's not llama 1/2 based but is its own 15B model and as such the support for it in anything practical is near nonexistent. WizardLM v1.2 looks like it may be worth testing out.

All of the LLama 2 fine tunes I've tried out so far have weird issues though. Saying unrelated things at times, ignoring parts of the conversation and such. Could be fine tuning or prompt template goofs or Llama 1 may actually be a more self consistent base model overall.

Re: Ask HN: Who is using small OS LLMs in production?

#59

Post by euazOn »

npsomaratna wrote: post #6

Data security and privacy. Our clients (in aviation, finance, etc.) need this due to legal and regulatory reasons. Also, the new Llama 2 models are very powerful. In my testing, Llama 2 70b is comparable to GPT-3.5 in capability. (Shameless plug: here's our website: https://www.amw.ai/ )

I am also interested in your open-source-first approach, and am a bit confused from the landing page. Could you explain what exactly your product does? Is it an LLM-enhanced document parser?
Post reply on HN