31–40 of 98 posts
And that’s the point of fine tuning models. Still good to see someone walk through their fine tuning process, with a mix of hosted and local options.
They use a much simpler model, fine tune it, and manage to beat a way more advanced model
This is entirely unsurprising and in-line with the finding that even small specialized models do better in information extraction and text classification. So no wonder finetuned large LMs do good too. Personally, my PhD did fine grained ACE-like event and sentiment extraction and "small" specialized finetuned transformers outperformed prompting LLMs like BERT and Roberta-large. Would love to see an inclusion of small…
Your thesis sounds interesting! Do you have a link to it by any chance?
LLM approaches were evaluated on my own time and but published (I left research after obtaining my PhD).
1) beat at what? 2) do they beat Claude 3.5 Sonnet?
So very misleading title
Really interesting. Could the potentially controversial content of the target news article have an effect on ChatGPT's ability to summarize it?
404 Content moderation response in 4% of articles. This is just financial news text.
It is a prime reason we are considering open models.
Thanks for putting all this work and sharing it in such a detail! Data extraction/structuring data is the only serious application of LLMs I have actually engaged in for real work and found useful. I had to extract data from experience sampling reports which I could not share online, thus chatgpt etc was out of question. There were sentences describing onsets and offsets of events and descriptions of what went on. I…
I fully agree. I realized this early on when experimenting with GPT-3 for web data extraction. After posting the first prototype on Reddit and HN, we started seeing a lot of demand for automating rule-based web scraping stacks (lots of maintenance, hard to scale). This eventually led to the creation of our startup (https://kadoa.com) focused on automating this "boring and hard" problem.
It comes down to such relatively unexciting use cases where AI adds the most value.
AI won't eliminate our jobs, but it will automate tedious, repetitive work such as web scraping, form filling, and data entry.
Thanks for sharing this, It's well written and informative. I noticed you used 'temperature=1' in the GPT test for the example in the post. Is this best practice for a task requiring structured output? Have you tested other temperature settings? My casual understanding was that a temperature of 0 is best for these types of workloads while higher temperatures would be more effective for more 'creative' workloads.
I followed whatever the guidance was for a specific model. Some of the LLM finetuning providers did indeed set the temperature to 0 and I followed that, but others suggested 1. I could probably iterate a bit to see what is best for each model, and I might well do that for the one that I choose as the one I’ll be doubling down on in subsequent iterations / finetunes. Thanks for the suggestion!
I would say this actually invalidates the whole thing.
2. It would be nice to try again with 0 temperature, as I do a lot of structured data extraction. In my experience 0 temperature should always be used, and it can make a huge difference. Temperature of 1 essentially means that it will start to pick tokens with lower probability of being accurate...
Earlier quoted context omitted.
Your thesis sounds interesting! Do you have a link to it by any chance?
rovr beat me to it below. Here are more links: https://jacobsgill.es/phdobtained (fun fact: because my thesis contains published papers, I am in breach of a few journal's copyright by uploading my own thesis pdf, but fuck'em). LLM approaches were evaluated on my own time and but published (I left research after obtaining my PhD).
I have skimmed through it and it's truly amazing how good annotation of the dataset can lead to impressive results.
I apologise in advance if the question seems ignorant: The blog post talked about fine-tuning models online. Given that BERT models can run comfortably on even iPhone hardware, were you able to finetune your models locally or did you have to do it online too? If so, are there any products that you recommend?
And that’s the point of fine tuning models. Still good to see someone walk through their fine tuning process, with a mix of hosted and local options.
As I understood the point was not that they fine tuned a model and it got better They use a much simpler model, fine tune it, and manage to beat a way more advanced model
In that sense it's not that surprising that on a pure text extraction task with little "thinking" required a 7B model does well and outperforms other models after fine tuning. In the "noshotsfired" label GPT-4 is even accused of overthinking it.
It is interesting how finetuned mistral-7b and llama3-7b outperform finetuned gpt3.5-turbo. I would tend to attribute that to those models being newer and "more advanced" despite their low parameter count, but maybe that's interpreting too much into a small score difference.