Live data from Hacker News

The case for the return of fine-tuning

welovesota.com

41–50 of 85 posts

Re: The case for the return of fine-tuning

#41
I ran a survey on Twitter over the past few days asking for successful case studies that produced economically valuable results from fine-tuning LLMs.

I ask a version of this every six months or so, and usually the results are quite disappointing.

This time I had more credible replies than I have had in the past.

Here's my thread with highlights: https://twitter.com/simonw/status/1979254349235925084

And in a thread viewer for people who aren't signed into Twitter: https://twitter-thread.com/t/1979254349235925084

Some of the most impressive:

Datadog got https://twitter.com/_brimtown/status/1979669362232463704 and https://docs.datadoghq.com/logs/explorer/search/

Vercel run custom fine-tuned models on v0 for Next.js generation: https://vercel.com/blog/v0-composite-model-family

Shopify have a fine-tuned vision LLM for analyzing product photos: https://shopify.engineering/leveraging-multimodal-llms

Re: The case for the return of fine-tuning

#42
post #41

I ran a survey on Twitter over the past few days asking for successful case studies that produced economically valuable results from fine-tuning LLMs. I ask a version of this every six months or so, and usually the results are quite disappointing. This time I had more credible replies than I have had in the past. Here's my thread with highlights: https://twitter.com/simonw/status/1979254349235925084 And in a thread v…

If people have ideas for use cases where fine-tuning can make a big difference, but don't have the time/resources to try it out yourself yet want to see if it'll work, feel free to share your ideas as I'm currently creating a bunch of examples of this and could use some inspiration, I only have 3 real/confirmed use cases as of right now.

Re: The case for the return of fine-tuning

#43
post #32
post #29

Earlier quoted context omitted.

If your customers can't fine tune, do it for them instead.

How can you hire enough people to scale that while making the economics work? Why would they join you rather than founding their own company?

> How can you hire enough people to scale that while making the economics work?

Once you (as in you the person) have the expertise, what you need all the people for exactly? To fine-tuning you need to figure out the architecture, how to train, how to infer, pick together the dataset and then run the training (optionally setup a pipeline so the customer can run the "add more data -> train" process themselves). What in this process you need to hire so many people for?

> Why would they join you rather than founding their own company?

Same as always, in any industry, not everyone wants to lead and not everyone wants to follow.

Re: The case for the return of fine-tuning

#44
post #25

Earlier quoted context omitted.

Will Anthropic/OpenAI really hire anyone who can fine-tune an LLM?

They will hire anyone who can produce a model better than GPT5, which is the bar for fine tuning Otherwise, you should just use gpt5 Preparing a few thousands training examples and pressing fine tune can improve the base LLM in a few situations, but it also can make the LLM worse at other tasks in hard to understand ways that only show up in production because you didn’t build evals that are good enough to catch them…

> They will hire anyone who can produce a model better than GPT5, which is the bar for fine tuning

Depends on what you want to achieve, of course, but I see fine-tuning at the current point in time primarily as a cost-saving measure: Transfer GPT5-levels of skill onto a smaller model, where inference is then faster/cheaper to run. This of course slows down your innovation cycle, which is why generally this is imo not advisable.

Re: The case for the return of fine-tuning

#45
post #41

I ran a survey on Twitter over the past few days asking for successful case studies that produced economically valuable results from fine-tuning LLMs. I ask a version of this every six months or so, and usually the results are quite disappointing. This time I had more credible replies than I have had in the past. Here's my thread with highlights: https://twitter.com/simonw/status/1979254349235925084 And in a thread v…

Finetuning is pretty much necessary for regression tasks. Also useful for classification since you can get the direct probabilities in case you want to do some thresholding.

Re: The case for the return of fine-tuning

#46
post #32

Earlier quoted context omitted.

How can you hire enough people to scale that while making the economics work? Why would they join you rather than founding their own company?

> How can you hire enough people to scale that while making the economics work? Once you (as in you the person) have the expertise, what you need all the people for exactly? To fine-tuning you need to figure out the architecture, how to train, how to infer, pick together the dataset and then run the training (optionally setup a pipeline so the customer can run the "add more data -> train" process themselves). What in…

llm.finetune(data) is a leaky abstraction

Read Andrej’s blog that I linked earlier in the thread if you want to understand why.

Re: The case for the return of fine-tuning

#47

A couple of examples I have seen recently which makes me agree with OP: - PaddleOCR, a 0.9B model that reaches SOTA accuracy across text, tables, formulas, charts & handwriting. [0] - A 3B and 8B model which performs HTML to json extraction at GPT-5 level accuracy at 40-80x less cost, and faster inference. [1] I think it makes sense to fine tune when you're optimizing for a specific task. [0] https://huggingface.co/p…

Have you used PaddleOCR? I'm surprised they're claiming SOTA without comparing against Amazon Textract or Azure doc intelligence (LayoutLM v3 under the hood, as far as I know). I've played around with doc recognition quite a bit, and as far as I can tell those two are best-in-class.

Amazon textract is not great at multi colum layouts in my experience. Docupanda or some azure models beat it. Just my 2 cents.

Re: The case for the return of fine-tuning

#48
I go back and forth on this. A year ago, I was optimistic and I have had 1 case where RL fine tuning a model made sense. But while there are pockets of that, there is a clash with existing industry skills. I work with a lot of machine learning engineers and data scientists and here’s what I observe.

- many, if not most MLEs that got started after LLMs do not generally know anything about machine learning. For lack of clearer industry titles, they are really AI developers or AI devops

- machine learning as a trade is moving toward the same fate as data engineering and analytics. Big companies only want people using platform tools. Some ai products, even in cloud platforms like azure, don’t even give you the evaluation metrics that would be required to properly build ml solutions. Few people seem to have an issue with it.

- fine tuning, especially RL, is packed with nuance and details… lots to monitor, a lot of training signals that need interpretation and data refinement. It’s a much bigger gap than training simpler ML models, which people are also not doing/learning very often.

- The limited number of good use cases means people are not learning those skills from more senior engineers.

- companies have gotten stingy with sme-time and labeling

What confidence do companies have in supporting these solutions in the future? How long will you be around and who will take up the mantle after you leave?

AutoML never really panned out, so I’m less confident that platforming RL will go any better. The unfortunate reality is that companies are almost always willing to pay more for inferior products because it scales. Industry “skills” are mostly experience with proprietary platform products. Sure they might list “pytorch” as a required skill, but 99% of the time, there isn’t hardly anyone at the company that has spent any meaningful time with it. Worse, you can’t use it, because it would be too hard to support.

Re: The case for the return of fine-tuning

#49
post #20

Return? Did it run away? I don't think anyone thought fine tuning was dead.

There were many comments claiming that from around the end of 2023 to shortly before ChatGPT 5 was launched.

The main claim was that new models were much better than anything you could get your hands on to fine tune.

IMO, intuitively that never made sense. But I never tested it either.

Re: The case for the return of fine-tuning

#50
post #41

I ran a survey on Twitter over the past few days asking for successful case studies that produced economically valuable results from fine-tuning LLMs. I ask a version of this every six months or so, and usually the results are quite disappointing. This time I had more credible replies than I have had in the past. Here's my thread with highlights: https://twitter.com/simonw/status/1979254349235925084 And in a thread v…

If people have ideas for use cases where fine-tuning can make a big difference, but don't have the time/resources to try it out yourself yet want to see if it'll work, feel free to share your ideas as I'm currently creating a bunch of examples of this and could use some inspiration, I only have 3 real/confirmed use cases as of right now.

Something that's in my personal backlog is fine-tuning of TrOCR for purse seine observer workbooks. The default TrOCR is expecting English words, and so the FAO species codes used in the workbook result in terrible accuracy. LLMs do poorly in this space because you'll commonly see repeats (e.g. 100 out of 120 samples all have the same species code) which then leads to hallucination.
Post reply on HN