Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: If you use an LLM, which one do you use?

news.ycombinator.com

Post reply on HN
Print view View original

1–8 of 8 posts

Ask HN: If you use an LLM, which one do you use?

#1

Post by zopper »

It seems to me that most startups use one of the LLMs from OpenAI (ChatGPT / GPT-4), other closed-source LLMs like Bard or Claude are not that popular and open-source LLMs are only really used by researchers or hobbyists who run them locally.

Curious to hear from others who are building something on top of an LLM, which one do you use? Has anyone fine-tuned and deployed their own LLM or do you just rely on ChatGPT?

Re: Ask HN: If you use an LLM, which one do you use?

#2

Post by mackopes »

I recently started using the ChatGPT via OpenAI api. It seems to perform reasonably well on all tasks I needed it for so far. But bear in mind I use it for hobby projects so far.

I have not tried the open source LLMs so far as there's the additional hassle renting a server and deploying it there. Running it locally on a consumer GPU does not cut it yet as it is too slow. So to iterate faster, I prefer just using ChatGPT.

Re: Ask HN: If you use an LLM, which one do you use?

#4

Post by tlack »

I am experimenting with building software using the ReAct tool prompting pattern, using Llama derivative models like Manticore13B, Airoborous, etc. I script it all together using Microsoft Guidance with Llama.cpp and AutoGPTQ. Works pretty well for simple tasks and I know the costs are roughly fixed. Obviously their capabilities are far less than OpenAI's products but when you have tens of thousands of conversations to have the costs of ChatGPT become a distraction. Haven't tried finetuning yet.

Re: Ask HN: If you use an LLM, which one do you use?

#6

Post by zopper »

greenido wrote: post #3

We are using both openAI and Google's Bard. There are some aspects to each of them - but it's still mostly 'WIP'. The main challenge is how to 'tune' these LLMs with our own data as another layer that can improve the overall performance (=quality of answers).

Why did you decide to use both OpenAI and Bard?

Re: Ask HN: If you use an LLM, which one do you use?

#7

Post by zopper »

dserban wrote: post #5

For a side project I'm using a fine-tuned vicuna-13b. I'm using to generate search queries from natural language, and it outperforms all other open-source models at deep intent recognition.

Do you run Vicuna locally or have you had any success in deploying it somewhere? I have been thinking of using a fine-tuned model but haven't found anything cost effective where I cab deploy it. You either run the machine where it's served consistently and the cost is in the hundreds or you accept extremely slow starts.

Re: Ask HN: If you use an LLM, which one do you use?

#8

Post by dserban »

zopper wrote: post #7
dserban wrote: post #5

For a side project I'm using a fine-tuned vicuna-13b. I'm using to generate search queries from natural language, and it outperforms all other open-source models at deep intent recognition.

Do you run Vicuna locally or have you had any success in deploying it somewhere? I have been thinking of using a fine-tuned model but haven't found anything cost effective where I cab deploy it. You either run the machine where it's served consistently and the cost is in the hundreds or you accept extremely slow starts.

I have a monster of a Linux cluster at home, which is busy running Cassandra and Spark for my other side projects. I'm piggybacking on that while this particular project is still in development.

At the rate LLMs are evolving, a reasonably priced cloud offering will probably exist for me to leverage for production, by the time I need it.

Post reply on HN