Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My finetuned models beat OpenAI's GPT-4

mlops.systems

Post reply on HN
Print view View original

51–60 of 98 posts

Re: My finetuned models beat OpenAI's GPT-4

#51

Post by mewpmewp2 »

wrsh07 wrote: post #48
mewpmewp2 wrote: post #44

Earlier quoted context omitted.

The dataset is there: https://huggingface.co/datasets/strickvl/isafpressreleases_t... but when looking for rows where GPT-4o was deemed inaccurate then to me it seems the label was wrong or at least it wasn't possible to infer that certain label from the input text. But finetuned model was able to predict it. Which makes me wonder whether the finetuned models are poisoned with eval data... See this one: > ISAF Joint…

I was wondering if there was some info in the bizarrely formatted date, but I think 022 is just the issue number: https://www.dvidshub.net/news/66703/correction-isaf-joint-co...

Also a lot of the time the dates are wrong seems to be due to only having those formats, which does make me wonder again how do fine tuned get this right unless they have been fine tuned using eval data...

Re: My finetuned models beat OpenAI's GPT-4

#52

Post by toisanji »

I'm most excited about getting a faster model. A model like GPT4 can be overkill because its too slow. What are the smallest fine tuned models that could beat a gpt4 model? Is it 7b or could a 3b model like phi3 do well for tasks like classification and summarization?

Re: My finetuned models beat OpenAI's GPT-4

#53

Post by klabb3 »

jrm4 wrote: post #49

At the risk of sounding like an old head; Seems to me then, priority one should be "free and open source all the models as hard as possible, so that EVERYONE can fine-tune." (This being a subset of the idea of, free / open source is generally preferable for both freedom and quality)

It seems to me this means whoever has hoarded and declared ownership of the most personal data will make the best products. Kinda like how some people liked their targeted ads because they’re more “relevant”, only now it’s not just ads but useful products. Another winner is of course platform owners like Apple and Microsoft who can scrape your data off their apps and products, even locally. This is a much bigger edge than being 3-6 months ahead in model quality.

I despise the centralization of this tech as well, and while it’s hopeful that smaller fine tuned models are better, they won’t win (or barely stand a chance) out of the virtue of openness and privacy alone. Best we can hope for is proliferation in the small-medium sized business service space - that OpenAI tokens are not worth the extra expense if open models are commoditized and effective. This was probably Zuck's plan all along – to prevent centralized gate keepers in tech that’s mainly benefiting his rivals. But the enemy of my enemy is my friend, so his actions may be the best he’s ever done for the public good.

Re: My finetuned models beat OpenAI's GPT-4

#54

Post by furyofantares »

hubraumhugo wrote: post #35
dimask wrote: post #23

Thanks for putting all this work and sharing it in such a detail! Data extraction/structuring data is the only serious application of LLMs I have actually engaged in for real work and found useful. I had to extract data from experience sampling reports which I could not share online, thus chatgpt etc was out of question. There were sentences describing onsets and offsets of events and descriptions of what went on. I…

> Data extraction/structuring data is the only serious application of LLMs I fully agree. I realized this early on when experimenting with GPT-3 for web data extraction. After posting the first prototype on Reddit and HN, we started seeing a lot of demand for automating rule-based web scraping stacks (lots of maintenance, hard to scale). This eventually led to the creation of our startup ( https://kadoa.com ) focused…

The way you cut that quote turns it into an assertion that doesn't exist in parent post.

They didn't make the (incorrect) statement that no other serious, useful application exists.

But that's how it reads when you cut off before "I have actually engaged in for real work and found useful"

Re: My finetuned models beat OpenAI's GPT-4

#55

Post by Mockapapella »

gillesjacobs wrote: post #32
pandatigox wrote: post #9

Earlier quoted context omitted.

Your thesis sounds interesting! Do you have a link to it by any chance?

rovr beat me to it below. Here are more links: https://jacobsgill.es/phdobtained (fun fact: because my thesis contains published papers, I am in breach of a few journal's copyright by uploading my own thesis pdf, but fuck'em). LLM approaches were evaluated on my own time and but published (I left research after obtaining my PhD).

This is really cool -- thanks for posting it! I'll have to skim through it at some point since a lot of my work is in classifications models and mirrors the results you've seen

Re: My finetuned models beat OpenAI's GPT-4

#56

Post by SpaceManNabs »

gillesjacobs wrote: post #32
pandatigox wrote: post #9

Earlier quoted context omitted.

Your thesis sounds interesting! Do you have a link to it by any chance?

rovr beat me to it below. Here are more links: https://jacobsgill.es/phdobtained (fun fact: because my thesis contains published papers, I am in breach of a few journal's copyright by uploading my own thesis pdf, but fuck'em). LLM approaches were evaluated on my own time and but published (I left research after obtaining my PhD).

> because my thesis contains published papers, ..., but f 'em

Excluding the part in the middle because I don't wanna repost potential issues for you. I just wanted to comment that that is terrible. People often talk about the siloed nature of research in industry, without considering that academia supports the draconian publishing system. I understand IP protection, but IP protection doesn't have to mean no access. This is such a huge issue in the bio- world (biostats, genetics, etc).

Re: My finetuned models beat OpenAI's GPT-4

#57

Post by renegade-otter »

gillesjacobs wrote: post #3

This is entirely unsurprising and in-line with the finding that even small specialized models do better in information extraction and text classification. So no wonder finetuned large LMs do good too. Personally, my PhD did fine grained ACE-like event and sentiment extraction and "small" specialized finetuned transformers outperformed prompting LLMs like BERT and Roberta-large. Would love to see an inclusion of small…

The caveat here is that if you don't know how to create good specialized models - you are just wasting everyone't time and money:

https://www.threads.net/@ethan_mollick/post/C46AfItO8RS?hl=e...

Re: My finetuned models beat OpenAI's GPT-4

#59

Post by colordrops »

kcorbitt wrote: post #43

(Disclaimer: I'm the founder of OpenPipe, one of the fine-tuning services OP tried and ultimately the one that produced the highest performing model, it appears.) Data extraction is a use case that fine-tuned models are fantastic at, so I'm not surprised that OP got good results. That said, I've also found it's pretty easy to beat GPT-4 across many task types if you have a way of getting strong training data. We publ…

Why isn't someone providing a "meta model" that uses an LLM to choose between various fine tuned models depending on the question to get overall better results than gpt4?

Re: My finetuned models beat OpenAI's GPT-4

#60

Post by furyofantares »

singularity2001 wrote: post #33
XiphiasX wrote: post #17

1) beat at what? 2) do they beat Claude 3.5 Sonnet?

Just in the task of structured data extraction So very misleading title

> So very misleading title

Eh, I can see that, but to me "finetuned model" pretty strongly implies some specific task

Post reply on HN