Anyone has a tutorial how to achieve it to own a self-hosted model?
You can try going get some pre-trained (sometimes, fine-tuned) models on HuggingFace, following their instructions. Good luck!
Ask HN: People who switched from GPT to their own models. How was it?
11–20 of 87 posts
Re: Ask HN: People who switched from GPT to their own models. How was it?
#12Anyone has a tutorial how to achieve it to own a self-hosted model?
Reddit community r/LocalLlama has great info
Re: Ask HN: People who switched from GPT to their own models. How was it?
#13I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.
Could you provide more details on this matter? Specifically, I'm interested in knowing which base model you've utilized and the approach you've taken to fine-tune it. Your insights would be greatly appreciated and highly beneficial.
For narrow stuff you can do better job than base gpt4/mistral/etc model. You fine tune it with your very custom data, stuff that got didn’t seem to be trained on, it will generalize it well.
Re: Ask HN: People who switched from GPT to their own models. How was it?
#14Anyone has a tutorial how to achieve it to own a self-hosted model?
I use lmstudio and continue.dev. Deepseek model usually but i try out other models every now and then.
Re: Ask HN: People who switched from GPT to their own models. How was it?
#15My 2024 prediction is we will see far more people moving off of openai once they encounter its cost and latency compared to (less proven/scaled) competitors. It’s often a speed versus quality tradeoff, and I’ve seen multiple providers 3x faster than OpenAI with far more than 1/3 the quality
Re: Ask HN: People who switched from GPT to their own models. How was it?
#16Running Mistral-Instruct-0.1 for call/email summarization, Mixtral for contract mining & OpenChat to augment agentic chatbot equipped with RAG tools(Instruct again). Experience has been great, INT8 tradeoffs are acceptable until hardware FP8(FP4 anyone?) becomes more widely & cheaply available. On-prem costs have been absorbed already for few boxes of A100s & legacy V100s running millions of such interactions.
Hey Akash would you mind elaborating about what you mean by contract mining?
Re: Ask HN: People who switched from GPT to their own models. How was it?
#17I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.
Which base model did you use and what methodologies you used?
Likely mistral
Re: Ask HN: People who switched from GPT to their own models. How was it?
#18I've been using [continue](https://continue.dev/) alongside Ollama. My go-to llm has been [deepseek-coder 7b](https://ollama.com/library/deepseek-coder). The setup feels as good as ChatGPT 4, local first, and overall, I enjoy it.
Re: Ask HN: People who switched from GPT to their own models. How was it?
#19Earlier quoted context omitted.
Could you provide more details on this matter? Specifically, I'm interested in knowing which base model you've utilized and the approach you've taken to fine-tune it. Your insights would be greatly appreciated and highly beneficial.
For narrow stuff you can do better job than base gpt4/mistral/etc model. You fine tune it with your very custom data, stuff that got didn’t seem to be trained on, it will generalize it well.
Have you done this? How did you do it?
I've been looking forward to someone providing a detailed guide on how to "fine tune it with your custom data" for ages!
Re: Ask HN: People who switched from GPT to their own models. How was it?
#20Running Mistral-Instruct-0.1 for call/email summarization, Mixtral for contract mining & OpenChat to augment agentic chatbot equipped with RAG tools(Instruct again). Experience has been great, INT8 tradeoffs are acceptable until hardware FP8(FP4 anyone?) becomes more widely & cheaply available. On-prem costs have been absorbed already for few boxes of A100s & legacy V100s running millions of such interactions.
What are the trade-offs with INT8? I thought even the INT4 loss of accuracy was small and the INT8 loss almost unmeasurable.