It’s been ok. I’m running llama2 7b and it’s … fine. The results I get from gpt4 aren’t much better. This for general tasks.
Mostly I think I need to use LLMs more effectively
Ask HN: People who switched from GPT to their own models. How was it?
41–50 of 87 posts
Re: Ask HN: People who switched from GPT to their own models. How was it?
#42Anyone has a tutorial how to achieve it to own a self-hosted model?
The other answers are recommending paths which give you #1. less control and #2. projects with smaller eco-systems.
If you want a truly general purpose front-end for LLMs, the only good solution right now is oobabooga: https://github.com/oobabooga/text-generation-webui
All other alternatives have only small fractions of the features that oobabooga supports. All other alternatives only support a fraction of the LLM backends that oobabooga supports, etc.
Re: Ask HN: People who switched from GPT to their own models. How was it?
#43I've been using [continue]( https://continue.dev/ ) alongside Ollama. My go-to llm has been [deepseek-coder 7b]( https://ollama.com/library/deepseek-coder ). The setup feels as good as ChatGPT 4, local first, and overall, I enjoy it.
No markdown here sadly.
Re: Ask HN: People who switched from GPT to their own models. How was it?
#44Obviously talking my own book here, but we've helped dozens of customers make the transition from prompted GPT-4 or GPT-3.5 to their own fine-tuned models at OpenPipe.
The most common reaction I get is "wow, I didn't expect that to work so well with so little effort". For most tasks, a fine-tuned Mistral 7B will consistently outperform GPT-3.5 at a fraction of the cost, and for some use cases will even match or outperform GPT-4 (particularly for narrower tasks like classification, information extraction, summarization -- but a lot of folks have that kind of task). Some aggregate stats are in our blog: https://openpipe.ai/blog/mistral-7b-fine-tune-optimized
Re: Ask HN: People who switched from GPT to their own models. How was it?
#45I prefer to use local models when running data extraction or processing over 10k or more records. Hosted services would be slow and brittle at this point.
Mistral 7B fine-tunes (OpenChat is my favorite) just chug through the data and get the job done.
Details: using vLLM to run the models. Using ChatGPT-4 to condense information for complex prompts (that the local models will execute).
I think, the situation will just keep on getting better with each month.
Re: Ask HN: People who switched from GPT to their own models. How was it?
#46I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.
Maybe the key to a good universal LLM is having multiple fine tuned models for various domains. The user thinks he's querying a single model but really there's some mechanism that selecting the best model for his query out of say like 300 different possibilities.
This also helps distributes traffic as a side effect.
I guess the problem is how the conversation would flow. If the user changes topics from say art to quantum physics then asks a question about quantum physics and art then I'm not sure what the algorithm should do.
Re: Ask HN: People who switched from GPT to their own models. How was it?
#47A couple of months ago I attended a presentation of an on-prem LLM. An audience member asked, if it was using OpenAI in any way.
The presenter, somewhat overeagerly, "Why not ask our new AI?" and went on to type: "Are you an independent model or do you use OpenAI?"
To chat bot answered in flourish language that sure it was using ChatGPT as a backend. Which it was not and which was kind of the whole point of the presentation.
Re: Ask HN: People who switched from GPT to their own models. How was it?
#48I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.
> GPT 4 isn't that great, in general same here, it doesn't adhere to explicit instructions, maybe one or two simple instructions are ok but not more complex ones
YMMV, it depends very heavily on what you're asking it to do.
Re: Ask HN: People who switched from GPT to their own models. How was it?
#49Obviously talking my own book here, but we've helped dozens of customers make the transition from prompted GPT-4 or GPT-3.5 to their own fine-tuned models at OpenPipe. The most common reaction I get is "wow, I didn't expect that to work so well with so little effort". For most tasks, a fine-tuned Mistral 7B will consistently outperform GPT-3.5 at a fraction of the cost, and for some use cases will even match or outpe…
Have you started migrating anyone over to Mistral Large yet?
Re: Ask HN: People who switched from GPT to their own models. How was it?
#50Obviously talking my own book here, but we've helped dozens of customers make the transition from prompted GPT-4 or GPT-3.5 to their own fine-tuned models at OpenPipe. The most common reaction I get is "wow, I didn't expect that to work so well with so little effort". For most tasks, a fine-tuned Mistral 7B will consistently outperform GPT-3.5 at a fraction of the cost, and for some use cases will even match or outpe…
Have you started migrating anyone over to Mistral Large yet?
I don't think they've released a fine-tuning API, but we'll definitely support it once they do!