Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: People who switched from GPT to their own models. How was it?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: Ask HN: People who switched from GPT to their own models. How was it?

#71

Post by iAkashPaul »

alecco wrote: post #68
iAkashPaul wrote: post #5

Running Mistral-Instruct-0.1 for call/email summarization, Mixtral for contract mining & OpenChat to augment agentic chatbot equipped with RAG tools(Instruct again). Experience has been great, INT8 tradeoffs are acceptable until hardware FP8(FP4 anyone?) becomes more widely & cheaply available. On-prem costs have been absorbed already for few boxes of A100s & legacy V100s running millions of such interactions.

> until hardware FP8(FP4 anyone?) becomes more widely & cheaply available Nvidia 40xx Tensor Cores support FP8 (but not the fancy async stuff on H100s). For some reason it's not used for models (AFAIK). Some people report throttling issues in some cases, though.

I think PyTorch recently added FP8 & even Intel's Neural-Compressor has experimental support. But yeah most HF LLM models are currently just INT8 or FP4/NF4 loaded thanks to BitsandBytes by Tim Dettmers.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#72

Post by visarga »

exploringBytes wrote: post #60
visarga wrote: post #36

Earlier quoted context omitted.

> GPT 4 isn't that great, in general same here, it doesn't adhere to explicit instructions, maybe one or two simple instructions are ok but not more complex ones

More complex instructions should be split into multiple prompts, solving the challenge step by step

Yes, but sometimes the two parts are also correlated so it's better if they are done at the same time.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#73

Post by MaxMatti »

visarga wrote: post #36
geoduck14 wrote: post #3

I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.

> GPT 4 isn't that great, in general same here, it doesn't adhere to explicit instructions, maybe one or two simple instructions are ok but not more complex ones

What model are you using for more complex prompts?

Re: Ask HN: People who switched from GPT to their own models. How was it?

#74

Post by gettodachoppa »

jianfgo wrote: post #8

Anyone has a tutorial how to achieve it to own a self-hosted model?

https://docs.sillytavern.app/usage/local-llm-guide/how-to-us...

Follow the guide all the way until you get to "Loading our model in Oobabooga". Then ignore the rest. You can do inference in Ooba under the Notebook tab.

(You can also ignore the "enabling HTTP API" parts, but it's quite handy, it's an OpenAI-compatible API which means you can use any OpenAI-compatible web UI)

Re: Ask HN: People who switched from GPT to their own models. How was it?

#75

Post by apetresc »

ninetyninenine wrote: post #46
geoduck14 wrote: post #3

I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.

Maybe the key to a good universal LLM is having multiple fine tuned models for various domains. The user thinks he's querying a single model but really there's some mechanism that selecting the best model for his query out of say like 300 different possibilities. This also helps distributes traffic as a side effect. I guess the problem is how the conversation would flow. If the user changes topics from say art to qua…

> This also helps distributes traffic as a side effect.

I'm not sure it's "distributing" traffic so much as amplifying it.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#76

Post by ashu1461 »

kcorbitt wrote: post #44

Obviously talking my own book here, but we've helped dozens of customers make the transition from prompted GPT-4 or GPT-3.5 to their own fine-tuned models at OpenPipe. The most common reaction I get is "wow, I didn't expect that to work so well with so little effort". For most tasks, a fine-tuned Mistral 7B will consistently outperform GPT-3.5 at a fraction of the cost, and for some use cases will even match or outpe…

Were their any tasks like intent / entity detection etc ? - I guess classification / information extraction covers that, but still anything more specific ?

Re: Ask HN: People who switched from GPT to their own models. How was it?

#77

Post by ninetyninenine »

apetresc wrote: post #75
ninetyninenine wrote: post #46

Earlier quoted context omitted.

Maybe the key to a good universal LLM is having multiple fine tuned models for various domains. The user thinks he's querying a single model but really there's some mechanism that selecting the best model for his query out of say like 300 different possibilities. This also helps distributes traffic as a side effect. I guess the problem is how the conversation would flow. If the user changes topics from say art to qua…

> This also helps distributes traffic as a side effect. I'm not sure it's "distributing" traffic so much as amplifying it.

Two users. One user is talking about physics, the other about art. Two different models are utilized.

Load is divided across 2 models. Load balancing is a feature for free and division is across subjects. Of course this is assuming each model owns it's own set of gpus.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#80

Post by felurx »

ninetyninenine wrote: post #77
apetresc wrote: post #75

Earlier quoted context omitted.

> This also helps distributes traffic as a side effect. I'm not sure it's "distributing" traffic so much as amplifying it.

Two users. One user is talking about physics, the other about art. Two different models are utilized. Load is divided across 2 models. Load balancing is a feature for free and division is across subjects. Of course this is assuming each model owns it's own set of gpus.

You could use the same GPUs to run two instances of one model.
Post reply on HN