Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extract-0: A specialized language model for document information extraction

arxiv.org

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Extract-0: A specialized language model for document information extraction

#51

Post by _carltg »

littlestymaar wrote: post #36
bangaladore wrote: post #26

Earlier quoted context omitted.

> Of course, it is specialized to outperform general models on this specific task in this specific domain with this specific json format for output. My understanding is generally this is not considered an obvious result. In that high parameter generalist models largely outperform lower parameter specialists. The real issue is they tested on data in their training set. * * Incorrect-- Edit misread parent comment.

> The real issue is they tested on data in their training set. Hm, no. They trained on a part of their synthetic set and tested on another part of the set. Or at least that's what they said they did: > from which 1,000 were held out as a benchmark test set. Emphasis mine.

Yes, but due to it being derived from the same underlying source dataset, it is effectively evaluating on the training dataset, not an independent validation/ test dataset.

The difference is subtle but important. If we expect the model to truly outperform a general model, it should generalize to a completely independent set.

Re: Extract-0: A specialized language model for document information extraction

#52

Post by _carltg »

fxwin wrote: post #28
m3kw9 wrote: post #24

Earlier quoted context omitted.

So they tested using training examples? Lmao

> held out

The problem is the methodology they use to hold them out. For a truly independent validation set, they need to hold out the material before augmentation, not after. If you hold out after augmentation, then you leverage biases from the training regimen already and hence you artificially boost your model's performance. This is not sufficient to demonstrate your model is generalizing properly.

In analogy: instead of taking leaves off of different trees, they are taking leaves from different branches from the same tree.

Re: Extract-0: A specialized language model for document information extraction

#53

Post by selim-now »

ImJasonH wrote: post #41
Garlef wrote: post #40

Earlier quoted context omitted.

It's a matter of ROI: When is it worth it to build something specialized?

Is anybody working on making building specialized things easier and cheaper?

yes! check out https://distillabs.ai/ – follows a similar approach except the evaluation set is held out before the synthetic data generation, which I would argue makes it more robust (I'm affiliated)

Re: Extract-0: A specialized language model for document information extraction

#54

Post by selim-now »

_carltg wrote: post #52
fxwin wrote: post #28

Earlier quoted context omitted.

> held out

The problem is the methodology they use to hold them out. For a truly independent validation set, they need to hold out the material before augmentation, not after. If you hold out after augmentation, then you leverage biases from the training regimen already and hence you artificially boost your model's performance. This is not sufficient to demonstrate your model is generalizing properly. In analogy: instead of tak…

That would definitely make the evaluation more robust. My fear is that with LLMs at hand people became allergic to preparing good human-labelled evaluation sets and would always to some degree use an LLM as a crutch.

Re: Extract-0: A specialized language model for document information extraction

#55

Post by selim-now »

lenerdenator wrote: post #18
trjordan wrote: post #4

It really seems like all the next big leaps in AI are going to be fine-tuning fit-for-purpose models. Everything past GPT5 has been ... fine. It's better at chat (sort of, depending on your tone preferenc) and way better at coding/tool use. In our product (plan out a migration with AI), they've gotten worse, because they want to chat or code. I'd have expected the coding knowledge to generalize, but no! Especially Cl…

We'll need to find a way to make fine-tuning happen on consumer hardware. I hope we do that sooner rather than later. $196 is not awful, but still pretty high up on the cost side for hobbyists.

well, fine-tuning is possible on consumer hardware, the problem is that it would be slow and that you're limited in the size of the dataset you can use in the process.

In case you would want to follow the approach in this paper and synthetically augment a dataset – using an LLM for that (instead of a smaller model) just makes sense and then the entire process cannot be easily run on your local machine.

Re: Extract-0: A specialized language model for document information extraction

#56

Post by selim-now »

jokethrowaway wrote: post #38
mountainriver wrote: post #17

It's wild to me how many people still think that fine-tuning doesn't work. There was just a thread the other day with numerous people arguing that RL should only be done by the big labs. There is so much research that shows you can beat frontier models with very little investment. It's confusing that the industry at large hasn't caught up with that

the subject of this news likely doesn't generalise to random documents You need some serious resources to do this properly, think about granite docling model by IBM. For LLM: Finetuning makes sense for light style adjustments with large models (eg. customize a chat assistant to sound a certain way) or to teach some simple transformations (eg. a new output format). You get away with 100-1000 samples. If you want to te…

interesting, I would argue that fine-tuning makes sense especially in cases where you want to narrow down a small model to a single task – in this case you can get the most bang-per-parameter in a way, using a small model that performs very well in a very narrow space.

Re: Extract-0: A specialized language model for document information extraction

#57

Post by selim-now »

whakim wrote: post #22

Ok, but what was the cost of labor put into curation of the training dataset and performing the fine-tuning? Hasn’t the paper’s conclusion been repeatedly demonstrated - that it is possible to get really good task-specific performance out of fine-tuned smaller models? There just remains the massive caveat that closed-source models are pretty cheap and so the ROI isn’t there in a lot of cases.

If the cost of getting the model is $200, then the cost of the trade-off seems to be quite clear.

You are right that the labor is a factor, unless you use a platform like https://www.distillabs.ai/ then the process is automated. (I'm affiliated)

Re: Extract-0: A specialized language model for document information extraction

#58

Post by selim-now »

christkv wrote: post #11
era37 wrote: post #8

LLMs are only going to improve by fragmenting them into specialized systems for low parameter high performance results. We’ve reached the point where models will get smaller and more compact

I guess we are going to be using multiple small specialized models with a reasoning model and tooling.

isn't that the premise of the Nvidia paper? https://arxiv.org/pdf/2506.02153

Re: Extract-0: A specialized language model for document information extraction

#59

Post by jvanderbot »

jvanderbot wrote: post #19
verbify wrote: post #14

Earlier quoted context omitted.

I thought "The Bitter Lesson" was that whole a specialised system will outperform in the short term, generalized systems with lots of data win in the long term. http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Over time. But for a given instant, specialization will always win. That message is for researchers, who seek to have long term impact and it's bitter because it goes against their desire to provide long term impact from their own clever abstraction or insights. But it's informative for the engineers that need something right now, because it means taking the best general purpose tool and specializing it will outperfo…

Unbeknownst to me, parent edited to incorporate my comment. Move along.

Re: Extract-0: A specialized language model for document information extraction

#60

Post by fxwin »

_carltg wrote: post #52
fxwin wrote: post #28

Earlier quoted context omitted.

> held out

The problem is the methodology they use to hold them out. For a truly independent validation set, they need to hold out the material before augmentation, not after. If you hold out after augmentation, then you leverage biases from the training regimen already and hence you artificially boost your model's performance. This is not sufficient to demonstrate your model is generalizing properly. In analogy: instead of tak…

I would agree with that
Post reply on HN