.
My finetuned models beat OpenAI's GPT-4
41–50 of 98 posts
Re: My finetuned models beat OpenAI's GPT-4
#42I took a look at a random row to try to find why mistakes were happening.
Why is this one labelled with start_date: 2011-02-07?
> Afghan, Coalition Forces Clear Northern Kandahar ISAF Joint Command - Afghanistan 2011-02-D-081 For Immediate Release KABUL, Afghanistan (Feb. 12) – Afghan and coalition forces set out to provide security and assist the local population during a clearing operation in a remote village in Shah Wali Kot district, Kandahar province, Feb. 8. District Chief of Police Bacha Khan, and his policemen; Afghan commandos from 2nd Company, 3rd Commando Kandak, along with U.S. service members from Special Operations Task Force – South, searched the village throughout the day and detained 20 suspected insurgents. Also found were 80 pounds (36 kilograms) of homemade explosives and various improvised explosive device-making materials. Leading a squad during the operation was Afghan commando Sgt. Hafiz Rahman, who said this operation has shown him progress. “The people are respecting us,” Rahman said. “They ask us if we want tea, or ‘do we want bread?’ They are thankful for the security.” Children during the operation brought commandos blankets in the evening and offered them food throughout the day.
Trying to find the source, I'm also not seeing any indication of Feb 7.
https://www.dvidshub.net/news/65238/afghan-police-commandos-...
---------------
And why is this labelled as Mar 6, GPT-4o and I personally find Mar 7 to be logical.
ISAF Joint Command Morning Operational Update, March 8, 2011 ISAF Joint Command - Afghanistan 2011-03-S-022 For Immediate Release KABUL, Afghanistan (March 8, 2011) Afghan and coalition forces targeted a Taliban district chief, killed one insurgent and detained several others during an operation in Burkah district, Baghlan province, yesterday. The Taliban district chief maintains ties to Taliban senior leadership throughout Kunduz, Baghlan, and Takhar provinces. He is involved in purchasing weapons and IEDs. Intelligence reports led the security force to the targeted compound in the city, where Afghan forces called for all occupants to exit the buildings peacefully before conducting a search. During that time, an armed individual threatened the security force and the force returned fire, killing him. Several suspected insurgents were detained after initial questioning at the scene.
But despite that the "finetuned" model also gets Mar 6. How does the finetuned model get Mar 6?
Re: My finetuned models beat OpenAI's GPT-4
#43(Disclaimer: I'm the founder of OpenPipe, one of the fine-tuning services OP tried and ultimately the one that produced the highest performing model, it appears.)
Data extraction is a use case that fine-tuned models are fantastic at, so I'm not surprised that OP got good results. That said, I've also found it's pretty easy to beat GPT-4 across many task types if you have a way of getting strong training data. We published some research[1] a week ago where we found that across 4 example tasks spanning creative summarization, question answering, data extraction and classification a fine-tuned Llama 3 8B was able to outperform GPT-4 on 3 of them. The key was to create a repeatable way of generating high-quality training data, which is also addressed in the post.
Re: My finetuned models beat OpenAI's GPT-4
#44Did you release the dataset and the code for testing? It would be interesting to check how 3.5 Sonnet performs on this task.
The dataset is there:
https://huggingface.co/datasets/strickvl/isafpressreleases_t...
but when looking for rows where GPT-4o was deemed inaccurate then to me it seems the label was wrong or at least it wasn't possible to infer that certain label from the input text. But finetuned model was able to predict it.
Which makes me wonder whether the finetuned models are poisoned with eval data...
See this one:
> ISAF Joint Command Morning Operational Update, March 8, 2011 ISAF Joint Command - Afghanistan 2011-03-S-022 For Immediate Release KABUL, Afghanistan (March 8, 2011) Afghan and coalition forces targeted a Taliban district chief, killed one insurgent and detained several others during an operation in Burkah district, Baghlan province, yesterday. The Taliban district chief maintains ties to Taliban senior leadership throughout Kunduz, Baghlan, and Takhar provinces. He is involved in purchasing weapons and IEDs. Intelligence reports led the security force to the targeted compound in the city, where Afghan forces called for all occupants to exit the buildings peacefully before conducting a search. During that time, an armed individual threatened the security force and the force returned fire, killing him. Several suspected insurgents were detained after initial questioning at the scene.
It claims "Yesterday" on March 8, so you would assume March 7 is correct start_date, but it's labelled Mar 6, and finetuned models get it "right", while GPT says Mar 7.
Re: My finetuned models beat OpenAI's GPT-4
#45(Disclaimer: I'm the founder of OpenPipe, one of the fine-tuning services OP tried and ultimately the one that produced the highest performing model, it appears.) Data extraction is a use case that fine-tuned models are fantastic at, so I'm not surprised that OP got good results. That said, I've also found it's pretty easy to beat GPT-4 across many task types if you have a way of getting strong training data. We publ…
[dead]
Re: My finetuned models beat OpenAI's GPT-4
#46Thanks for sharing this, It's well written and informative. I noticed you used 'temperature=1' in the GPT test for the example in the post. Is this best practice for a task requiring structured output? Have you tested other temperature settings? My casual understanding was that a temperature of 0 is best for these types of workloads while higher temperatures would be more effective for more 'creative' workloads.
I followed whatever the guidance was for a specific model. Some of the LLM finetuning providers did indeed set the temperature to 0 and I followed that, but others suggested 1. I could probably iterate a bit to see what is best for each model, and I might well do that for the one that I choose as the one I’ll be doubling down on in subsequent iterations / finetunes. Thanks for the suggestion!
You never use 1 for stuff like this. 1 is for poetry and creative writing. You need to redo this with temp=0 imo.
Re: My finetuned models beat OpenAI's GPT-4
#47Earlier quoted context omitted.
On that note: is there a good service for “here’s my dataset”, please fine tune these 9 models and give me evaluation stats?
OpenpPipe - https://openpipe.ai/ - is probably the service that most closely resembles what you’re asking for, but I found the evals weren’t really what I wanted — i.e. following my custom evaluation criteria — so you probably will end up having to do that yourself anyway. But for the finetuning, they’re all somewhat the same. Predibase and OpenPipe are two good options for that. Predibase has more base models for yo…
(Disclaimer: founder of OpenPipe). Thanks for the shout-out. Note that we're actively working on improved evaluations that will let you add more specific criteria as well as more evaluation types, like comparing field values to that of a golden dataset. This is definitely something that customers are asking for!
Re: My finetuned models beat OpenAI's GPT-4
#48Did you release the dataset and the code for testing? It would be interesting to check how 3.5 Sonnet performs on this task.
The dataset is there: https://huggingface.co/datasets/strickvl/isafpressreleases_t... but when looking for rows where GPT-4o was deemed inaccurate then to me it seems the label was wrong or at least it wasn't possible to infer that certain label from the input text. But finetuned model was able to predict it. Which makes me wonder whether the finetuned models are poisoned with eval data... See this one: > ISAF Joint…
I was wondering if there was some info in the bizarrely formatted date, but I think 022 is just the issue number: https://www.dvidshub.net/news/66703/correction-isaf-joint-co...
Re: My finetuned models beat OpenAI's GPT-4
#49At the risk of sounding like an old head;
Seems to me then, priority one should be "free and open source all the models as hard as possible, so that EVERYONE can fine-tune."
(This being a subset of the idea of, free / open source is generally preferable for both freedom and quality)
Re: My finetuned models beat OpenAI's GPT-4
#50Earlier quoted context omitted.
rovr beat me to it below. Here are more links: https://jacobsgill.es/phdobtained (fun fact: because my thesis contains published papers, I am in breach of a few journal's copyright by uploading my own thesis pdf, but fuck'em). LLM approaches were evaluated on my own time and but published (I left research after obtaining my PhD).
Thank you for the link! And congratulations on obtaining your PhD I have skimmed through it and it's truly amazing how good annotation of the dataset can lead to impressive results. I apologise in advance if the question seems ignorant: The blog post talked about fine-tuning models online. Given that BERT models can run comfortably on even iPhone hardware, were you able to finetune your models locally or did you have…
Thanks! The fine-tunes where done in 2019-21 on a 4xV100 server with hyperparameter search, so thousands of individual fine-tuned models were trained in the end. I used weights and biased for experiment dashboarding the hyperparam search, but the hardware was our own GPU server (no cloud service used).
I doubt you can fine-tune BERT-large on a phone. A quantized, inference optimised pipeline can be leaps and bounds more efficient and is not comparable with the huggingface training pipelines on full models I did at the time. For non-adapter based training you're going to need GPUs ideally.