Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extract-0: A specialized language model for document information extraction

arxiv.org

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Extract-0: A specialized language model for document information extraction

#31

Post by DetroitThrow »

disiplus wrote: post #30
bangaladore wrote: post #26

Earlier quoted context omitted.

> Of course, it is specialized to outperform general models on this specific task in this specific domain with this specific json format for output. My understanding is generally this is not considered an obvious result. In that high parameter generalist models largely outperform lower parameter specialists. The real issue is they tested on data in their training set. * * Incorrect-- Edit misread parent comment.

They did not test on the data that they tested, that's not what he wrote.

They synthetically generated 290k examples and kept 10k of them for testing.

It's worth pointing out that that's technically not testing on the training set, but looking at how similar examples are in the dataset, it's clear that severe overfitting would be unavoidable. That also makes the headline very misleading.

The weights may not be published since using it for document extraction on even the same format but with slightly different content or lengths would show how abysmal this finetune does outside of the synthetic data.

Re: Extract-0: A specialized language model for document information extraction

#32

Post by Aperocky »

fxwin wrote: post #28
m3kw9 wrote: post #24

Earlier quoted context omitted.

So they tested using training examples? Lmao

> held out

Actually in this case that's not exactly true:

> generation of 281,128 augmented examples

All example are already correlated because they are generated in the same way.

Re: Extract-0: A specialized language model for document information extraction

#33

Post by lemonlearnings »

verbify wrote: post #14
era37 wrote: post #8

LLMs are only going to improve by fragmenting them into specialized systems for low parameter high performance results. We’ve reached the point where models will get smaller and more compact

I thought "The Bitter Lesson" was that whole a specialised system will outperform in the short term, generalized systems with lots of data win in the long term. http://www.incompleteideas.net/IncIdeas/BitterLesson.html

I think there is a bitter lesson to the bitter lesson.

Sure you can throw more compute at it. But it cost a lot of money and you hit resource limits.

We have been doing an end run around the bitter lesson with prompt engineering. Also by using different models for vision vs. text. By getting (human coding) agents to "think" and run code.

The bitter lesson might be that you cant predict what the thing is that will be most optimal tomorrow and any player in the AI game can be innovated out of existence at any time.

Maybe anyone except TMSC.

Re: Extract-0: A specialized language model for document information extraction

#34

Post by lemonlearnings »

mvieira38 wrote: post #5
giancarlostoro wrote: post #2

So its a model designed exclusively for a purpose? Then the results should not be that surprising. It's still impressive don't get me wrong.

The results are interesting for showing the efficacy of small, fine-tuned models that can be run locally. AI providers as a business need their do-all models to be better than these if they want long-term revenue through the APIs, right?

The car analogy is Tesla needs to make tractors to compete.

Re: Extract-0: A specialized language model for document information extraction

#35

Post by gundmc »

mnkv wrote: post #23

> the generation of 281,128 augmented examples, from which 1,000 were held out as a benchmark test set. This model is trained on a custom dataset of 280k examples then tested on 1k very similar examples from the same dataset. Of course it is specialized to outperform general models on this specific task in this specific domain with this specific json format for output. This is a reasonable hobby project and interesti…

It's not novel research, but I think it drives home the point that many narrow applications of AI do not require the largest, latest (and most expensive) models. And in many of those cases, a small fine-tuned model is the most performant and cost-effective.

It is probably obvious to most who follow the space closely, but you'd be surprised how many engineers don't recognize this.

Re: Extract-0: A specialized language model for document information extraction

#36

Post by littlestymaar »

bangaladore wrote: post #26
mnkv wrote: post #23

> the generation of 281,128 augmented examples, from which 1,000 were held out as a benchmark test set. This model is trained on a custom dataset of 280k examples then tested on 1k very similar examples from the same dataset. Of course it is specialized to outperform general models on this specific task in this specific domain with this specific json format for output. This is a reasonable hobby project and interesti…

> Of course, it is specialized to outperform general models on this specific task in this specific domain with this specific json format for output. My understanding is generally this is not considered an obvious result. In that high parameter generalist models largely outperform lower parameter specialists. The real issue is they tested on data in their training set. * * Incorrect-- Edit misread parent comment.

> The real issue is they tested on data in their training set.

Hm, no.

They trained on a part of their synthetic set and tested on another part of the set. Or at least that's what they said they did:

> from which 1,000 were held out as a benchmark test set.

Emphasis mine.

Re: Extract-0: A specialized language model for document information extraction

#37

Post by manishsharan »

Jimmc414 wrote: post #25

The LoRA + GRPO training pipeline and the semantic similarity reward function over exact matching is actually interesting, but there is an evaluation issue if you want to accept the headline at face value. They trained on synthetic extractions like "extract equations from arXiv papers" and "extract regulatory information from FDA documents," then tested on more synthetic extractions from the same sources. Essentially…

I would also love to see how the model performs on management type presentations. I am referring to Gartner and McKinsey and BCG etc. presentations.

Re: Extract-0: A specialized language model for document information extraction

#38

Post by jokethrowaway »

mountainriver wrote: post #17

It's wild to me how many people still think that fine-tuning doesn't work. There was just a thread the other day with numerous people arguing that RL should only be done by the big labs. There is so much research that shows you can beat frontier models with very little investment. It's confusing that the industry at large hasn't caught up with that

the subject of this news likely doesn't generalise to random documents

You need some serious resources to do this properly, think about granite docling model by IBM.

For LLM: Finetuning makes sense for light style adjustments with large models (eg. customize a chat assistant to sound a certain way) or to teach some simple transformations (eg. a new output format). You get away with 100-1000 samples.

If you want to teach new behaviour you need a lot of data, likely too much to justify the investment for your average chatgpt wrapper AI company. The pragmatic choice is often to just prompt engineer and maybe split your task and combine multiple prompts

Re: Extract-0: A specialized language model for document information extraction

#39

Post by littlestymaar »

Aperocky wrote: post #32
fxwin wrote: post #28

Earlier quoted context omitted.

> held out

Actually in this case that's not exactly true: > generation of 281,128 augmented examples All example are already correlated because they are generated in the same way.

> All example are already correlated because they are generated in the same way.

All examples of “document information extraction” would be correlated no matter where they come from because they all would be “document information extraction” examples…

The real question is whether or not the examples are representative of the broad “document information extraction” use-case.

Re: Extract-0: A specialized language model for document information extraction

#40

Post by Garlef »

gundmc wrote: post #35
mnkv wrote: post #23

> the generation of 281,128 augmented examples, from which 1,000 were held out as a benchmark test set. This model is trained on a custom dataset of 280k examples then tested on 1k very similar examples from the same dataset. Of course it is specialized to outperform general models on this specific task in this specific domain with this specific json format for output. This is a reasonable hobby project and interesti…

It's not novel research, but I think it drives home the point that many narrow applications of AI do not require the largest, latest (and most expensive) models. And in many of those cases, a small fine-tuned model is the most performant and cost-effective. It is probably obvious to most who follow the space closely, but you'd be surprised how many engineers don't recognize this.

It's a matter of ROI: When is it worth it to build something specialized?
Post reply on HN