Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: People who switched from GPT to their own models. How was it?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 87 posts

Re: Ask HN: People who switched from GPT to their own models. How was it?

#51

Post by BillyTheMage »

ninetyninenine wrote: post #46
geoduck14 wrote: post #3

I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.

Maybe the key to a good universal LLM is having multiple fine tuned models for various domains. The user thinks he's querying a single model but really there's some mechanism that selecting the best model for his query out of say like 300 different possibilities. This also helps distributes traffic as a side effect. I guess the problem is how the conversation would flow. If the user changes topics from say art to qua…

The user could talk to an "expert opinion aggregator" model which in turn makes a bunch of queries to specialized models.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#52

Post by 0x008 »

ninetyninenine wrote: post #46
geoduck14 wrote: post #3

I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.

Maybe the key to a good universal LLM is having multiple fine tuned models for various domains. The user thinks he's querying a single model but really there's some mechanism that selecting the best model for his query out of say like 300 different possibilities. This also helps distributes traffic as a side effect. I guess the problem is how the conversation would flow. If the user changes topics from say art to qua…

That is actually the same idea as the (now) popular "Mixture of Experts" approach.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#53

Post by 0x008 »

manca wrote: post #31
codybontecou wrote: post #18

I've been using [continue]( https://continue.dev/ ) alongside Ollama. My go-to llm has been [deepseek-coder 7b]( https://ollama.com/library/deepseek-coder ). The setup feels as good as ChatGPT 4, local first, and overall, I enjoy it.

I've tried code-llama with Ollama, along with Continue.dev and found it to be pretty good. The only downside is that I couldn't "productively" run the 70B version, even on my MBP with M3 Max with 36GB of RAM (which interestingly should be enough to hold quantized model weights). It was simply painfully slow. 34B one works good enough for most of my use-cases, so I am happy.

I tried to use codellama 34B and I think it is pretty bad. For Example I asked it to convert a comment into a docstring and it would hallucinate a whole function around it.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#54

Post by 0x008 »

Mixed results. I think llama2 in general is pretty bad, especially at anything else than english. I've had very good results with Mixtral for Chat.

Of course all of them feel like a Frankenstein compared to actual ChatGPT. They feel similar and work just as well until, sometimes, they put out complete and utter garbage or artifacts and you wonder if they skimped on fine-tuning.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#55

Post by ilaksh »

kcorbitt wrote: post #44

Obviously talking my own book here, but we've helped dozens of customers make the transition from prompted GPT-4 or GPT-3.5 to their own fine-tuned models at OpenPipe. The most common reaction I get is "wow, I didn't expect that to work so well with so little effort". For most tasks, a fine-tuned Mistral 7B will consistently outperform GPT-3.5 at a fraction of the cost, and for some use cases will even match or outpe…

And those dozens are all doing narrow tasks like summarization right?

Re: Ask HN: People who switched from GPT to their own models. How was it?

#56

Post by notso411 »

kcorbitt wrote: post #44

Obviously talking my own book here, but we've helped dozens of customers make the transition from prompted GPT-4 or GPT-3.5 to their own fine-tuned models at OpenPipe. The most common reaction I get is "wow, I didn't expect that to work so well with so little effort". For most tasks, a fine-tuned Mistral 7B will consistently outperform GPT-3.5 at a fraction of the cost, and for some use cases will even match or outpe…

[dead]

Re: Ask HN: People who switched from GPT to their own models. How was it?

#57

Post by washadjeffmad »

arbitrandomuser wrote: post #27
simonw wrote: post #19

Earlier quoted context omitted.

Have you done this? How did you do it? I've been looking forward to someone providing a detailed guide on how to "fine tune it with your custom data" for ages!

this is imo the secret sauce that gives people an edge and not a lot of people will want to reveal

You're not wrong. There's been a lot of drama over licensing and releasing datasets, and a lot of the LLM scene are just pitchmen and promoters with no better grasp over what they're doing than "trust me, it's better".

Like with "prompt engineering", a lot of people are just hiding how much of the heavy lifting is from base models and a fluke of the merge. The past few "secret" set leaks were low/no delta diffs to common releases.

I said it a year ago, but if we want to wowed, make this a job for MLIS holders and references librarians. Without thorough, thoughtful curation, these things are just toys in the wrong hands.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#58

Post by notso411 »

osigurdson wrote: post #21
MaxLeiter wrote: post #15

My 2024 prediction is we will see far more people moving off of openai once they encounter its cost and latency compared to (less proven/scaled) competitors. It’s often a speed versus quality tradeoff, and I’ve seen multiple providers 3x faster than OpenAI with far more than 1/3 the quality

I greatly prefer to use ChatGPT-4 instead of 3.5 despite the slowness. Really a good feature for them to have would be to easily re-run a prompt on 4. However, the glitchiness of the service is kind of annoying.

[dead]

Re: Ask HN: People who switched from GPT to their own models. How was it?

#59

Post by rspoerri »

Im using mixtral 8x7b (q5) for my use cases, such as scripting, searching for ideas and or definitions that i allways need to factcheck.

Currently i use lmstudio on my m2 with 96gb ram. But i‘m looking into switchin to ollama or another oss solution.

Re: Ask HN: People who switched from GPT to their own models. How was it?

#60

Post by exploringBytes »

visarga wrote: post #36
geoduck14 wrote: post #3

I fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.

> GPT 4 isn't that great, in general same here, it doesn't adhere to explicit instructions, maybe one or two simple instructions are ok but not more complex ones

More complex instructions should be split into multiple prompts, solving the challenge step by step
Post reply on HN