Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My finetuned models beat OpenAI's GPT-4

mlops.systems

Post reply on HN
Print view View original

11–20 of 98 posts

Re: My finetuned models beat OpenAI's GPT-4

#11

Post by wuschel »

pandatigox wrote: post #9
gillesjacobs wrote: post #3

This is entirely unsurprising and in-line with the finding that even small specialized models do better in information extraction and text classification. So no wonder finetuned large LMs do good too. Personally, my PhD did fine grained ACE-like event and sentiment extraction and "small" specialized finetuned transformers outperformed prompting LLMs like BERT and Roberta-large. Would love to see an inclusion of small…

Your thesis sounds interesting! Do you have a link to it by any chance?

Seconded! Any URI to your PhD?

Re: My finetuned models beat OpenAI's GPT-4

#12

Post by botro »

Thanks for sharing this, It's well written and informative. I noticed you used 'temperature=1' in the GPT test for the example in the post. Is this best practice for a task requiring structured output? Have you tested other temperature settings? My casual understanding was that a temperature of 0 is best for these types of workloads while higher temperatures would be more effective for more 'creative' workloads.

Re: My finetuned models beat OpenAI's GPT-4

#13

Post by resource_waste »

strickvl wrote: post #7
courseofaction wrote: post #5

Really interesting. Could the potentially controversial content of the target news article have an effect on ChatGPT's ability to summarize it?

I think not. Normally if you get those kinds of errors you wouldn’t get any output at all. In the blog I show that all 724 of the test cases got proper JSON output etc for the queries so I don’t think this was an issue. I think these kinds of topics would have been well covered in the training data, and probably the OSS models would have used similar data so I don’t even think there’s a disparity to be found between…

>Normally if you get those kinds of errors you wouldn’t get any output at all

I am not sure. I disagree. If there is a pro-chatGPT user, I'm probably it.

Ive often seen it give significantly less effort to answer the question.

Re: My finetuned models beat OpenAI's GPT-4

#14

Post by scosman »

scosman wrote: post #2

And that’s the point of fine tuning models. Still good to see someone walk through their fine tuning process, with a mix of hosted and local options.

On that note: is there a good service for “here’s my dataset”, please fine tune these 9 models and give me evaluation stats?

Re: My finetuned models beat OpenAI's GPT-4

#15

Post by strickvl »

botro wrote: post #12

Thanks for sharing this, It's well written and informative. I noticed you used 'temperature=1' in the GPT test for the example in the post. Is this best practice for a task requiring structured output? Have you tested other temperature settings? My casual understanding was that a temperature of 0 is best for these types of workloads while higher temperatures would be more effective for more 'creative' workloads.

I followed whatever the guidance was for a specific model. Some of the LLM finetuning providers did indeed set the temperature to 0 and I followed that, but others suggested 1. I could probably iterate a bit to see what is best for each model, and I might well do that for the one that I choose as the one I’ll be doubling down on in subsequent iterations / finetunes. Thanks for the suggestion!

Re: My finetuned models beat OpenAI's GPT-4

#16

Post by strickvl »

scosman wrote: post #14
scosman wrote: post #2

And that’s the point of fine tuning models. Still good to see someone walk through their fine tuning process, with a mix of hosted and local options.

On that note: is there a good service for “here’s my dataset”, please fine tune these 9 models and give me evaluation stats?

OpenpPipe - https://openpipe.ai/ - is probably the service that most closely resembles what you’re asking for, but I found the evals weren’t really what I wanted — i.e. following my custom evaluation criteria — so you probably will end up having to do that yourself anyway. But for the finetuning, they’re all somewhat the same. Predibase and OpenPipe are two good options for that. Predibase has more base models for you to finetune, but it’s a bit more unwieldy to work with. I wrote about that in a previous post here -- https://mlops.systems/posts/2024-06-17-one-click-finetuning.....

Re: My finetuned models beat OpenAI's GPT-4

#20

Post by strickvl »

resource_waste wrote: post #13
strickvl wrote: post #7

Earlier quoted context omitted.

I think not. Normally if you get those kinds of errors you wouldn’t get any output at all. In the blog I show that all 724 of the test cases got proper JSON output etc for the queries so I don’t think this was an issue. I think these kinds of topics would have been well covered in the training data, and probably the OSS models would have used similar data so I don’t even think there’s a disparity to be found between…

>Normally if you get those kinds of errors you wouldn’t get any output at all I am not sure. I disagree. If there is a pro-chatGPT user, I'm probably it. Ive often seen it give significantly less effort to answer the question.

Interesting. I can maybe try finetuning one or two of the so-called 'uncensored' open models and see if that makes a difference. A bit harder to switch out the dataset completely, as that's really what I'm interested in :) I think the general point that finetuning a model for some custom task works is fairly uncontroversial, but if OpenAI's poor performance was on account of these kinds of guardrails it'd be yet another reason someone might want to finetune their own models I guess.
Post reply on HN