For anyone interested, we wrote a paper on a similar topic: https://www.nature.com/articles/s41467-024-45563-x
My finetuned models beat OpenAI's GPT-4
21–30 of 98 posts
Re: My finetuned models beat OpenAI's GPT-4
#22And that’s the point of fine tuning models. Still good to see someone walk through their fine tuning process, with a mix of hosted and local options.
On that note: is there a good service for “here’s my dataset”, please fine tune these 9 models and give me evaluation stats?
Together.AI is a good starting point. Even though I'm not sure what fine-tuning method they're using, the results are REALLY good.
Re: My finetuned models beat OpenAI's GPT-4
#23Thanks for putting all this work and sharing it in such a detail! Data extraction/structuring data is the only serious application of LLMs I have actually engaged in for real work and found useful. I had to extract data from experience sampling reports which I could not share online, thus chatgpt etc was out of question. There were sentences describing onsets and offsets of events and descriptions of what went on. I ran models through llama.cpp to turn these into csv format with 4 columns (onset, offset, description, plus one for whether a specific condition was met in that event or not which had to interpreted through the description). Giving some examples of how I want it all structured in the prompt, was enough for many different models to do it right. Mixtral 8x7b was my favourite because it ran the fastest in that quality level on my laptop.
I am pretty sure that a finetuned smaller model would be better and faster for this task. It would be great to start finetuning and sharing such smaller models: they do not really have to be really better than commercial LLMs that run online, as long as they are not at least worse. They are already much faster and cheaper, which is a big advantage for this purpose. There is already need for these tasks to be offline when one cannot share the data with openai and the like. Higher speed and lower cost also allow for more experimentation with more specific finetuning and prompts, with less care about token lengths of prompts and cost. This is an application where smaller, locally run, finetunable models can shine.
Re: My finetuned models beat OpenAI's GPT-4
#24Thanks for putting all this work and sharing it in such a detail! Data extraction/structuring data is the only serious application of LLMs I have actually engaged in for real work and found useful. I had to extract data from experience sampling reports which I could not share online, thus chatgpt etc was out of question. There were sentences describing onsets and offsets of events and descriptions of what went on. I…
Thanks! Yes one 'next step' that I'd like to do (probably around the work on deployment / inference that I'm turning to now) will be to see just how small I can get the model. Spacy have been pushing this kind of workflow (models in the order of tens of MB) for years and it's nice that there's a bit more attention to it. As you say, ideally I'd want lots of these tiny models that were super specialists at what they do, small in size and speedy in inference time. As I hinted towards the end of the post, however, keeping all that updated starts to get unwieldy at a certain point if you don't set it all up in the right way.
Re: My finetuned models beat OpenAI's GPT-4
#25Thanks for sharing this, It's well written and informative. I noticed you used 'temperature=1' in the GPT test for the example in the post. Is this best practice for a task requiring structured output? Have you tested other temperature settings? My casual understanding was that a temperature of 0 is best for these types of workloads while higher temperatures would be more effective for more 'creative' workloads.
I followed whatever the guidance was for a specific model. Some of the LLM finetuning providers did indeed set the temperature to 0 and I followed that, but others suggested 1. I could probably iterate a bit to see what is best for each model, and I might well do that for the one that I choose as the one I’ll be doubling down on in subsequent iterations / finetunes. Thanks for the suggestion!
GPT models shouldn't be used at temp 1 unless you only care about creative writing. They get much worse at factual stuff and code than with lower temperatures. And yes, 3.5 Turbo is less affected by this, which might be the reason why the models performed for you in reverse.
Re: My finetuned models beat OpenAI's GPT-4
#26This is entirely unsurprising and in-line with the finding that even small specialized models do better in information extraction and text classification. So no wonder finetuned large LMs do good too. Personally, my PhD did fine grained ACE-like event and sentiment extraction and "small" specialized finetuned transformers outperformed prompting LLMs like BERT and Roberta-large. Would love to see an inclusion of small…
Your thesis sounds interesting! Do you have a link to it by any chance?
Re: My finetuned models beat OpenAI's GPT-4
#27Earlier quoted context omitted.
Your thesis sounds interesting! Do you have a link to it by any chance?
Seconded! Any URI to your PhD?
Re: My finetuned models beat OpenAI's GPT-4
#28Did you release the dataset and the code for testing? It would be interesting to check how 3.5 Sonnet performs on this task.
Re: My finetuned models beat OpenAI's GPT-4
#29What is a good fine-tuning script for Mistral and LLaMA3 on an A100?
Unsloth is a great tool, super fast.