Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extract-0: A specialized language model for document information extraction

arxiv.org

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Extract-0: A specialized language model for document information extraction

#41

Post by ImJasonH »

Garlef wrote: post #40
gundmc wrote: post #35

Earlier quoted context omitted.

It's not novel research, but I think it drives home the point that many narrow applications of AI do not require the largest, latest (and most expensive) models. And in many of those cases, a small fine-tuned model is the most performant and cost-effective. It is probably obvious to most who follow the space closely, but you'd be surprised how many engineers don't recognize this.

It's a matter of ROI: When is it worth it to build something specialized?

Is anybody working on making building specialized things easier and cheaper?

Re: Extract-0: A specialized language model for document information extraction

#42

Post by derac »

trjordan wrote: post #4

It really seems like all the next big leaps in AI are going to be fine-tuning fit-for-purpose models. Everything past GPT5 has been ... fine. It's better at chat (sort of, depending on your tone preferenc) and way better at coding/tool use. In our product (plan out a migration with AI), they've gotten worse, because they want to chat or code. I'd have expected the coding knowledge to generalize, but no! Especially Cl…

GPT 5 came out less than two months ago, lol.

Re: Extract-0: A specialized language model for document information extraction

#43

Post by sigbottle »

Garlef wrote: post #40
gundmc wrote: post #35

Earlier quoted context omitted.

It's not novel research, but I think it drives home the point that many narrow applications of AI do not require the largest, latest (and most expensive) models. And in many of those cases, a small fine-tuned model is the most performant and cost-effective. It is probably obvious to most who follow the space closely, but you'd be surprised how many engineers don't recognize this.

It's a matter of ROI: When is it worth it to build something specialized?

Well, one day it might be at the level of shell scripting. I don't think about "the tradeoffs of building a specialized shell script", I just do it because it's cheap and easy and solves a problem right then and there.

I don't know how you would even begin to make this kind of same observation for ML models, but seems possible. The 2010s weren't exactly building out "trivial" models, but compared to the architectures and optimizations out now, yeah those models are toy by comparison.

Re: Extract-0: A specialized language model for document information extraction

#44

Post by dcastm »

Hey OP, I found some issues with your code:

During SFT, it uses the full training dataset[1]:

df = pd.read_csv('data/extraction_training_data.csv')

And during the evaluation, it uses the middle part of the same dataset[2]:

df = pd.read_csv('data/extraction_training_data.csv')

df = df[100000:100000+NUM_TEST_SAMPLES]

Also, you split train/test/val by chunk and not by document[3]. Then, the model "has seen" the documents that you're using to evaluate it (even if you're not evaluating it on the same chunks).

[1]: https://github.com/herniqeu/extract0/blob/0f8696a6fb1b620658...

[2]: https://github.com/herniqeu/extract0/blob/0f8696a6fb1b620658...

[3]: https://github.com/herniqeu/extract0/blob/0f8696a6fb1b620658...

Re: Extract-0: A specialized language model for document information extraction

#45

Post by NitpickLawyer »

jrm4 wrote: post #15

Seems this gives us a clear bifurcation of what AI is about to do. Open-Source style small players will actually solve problems with AI. And the big money invested things are going to do stupid pointless bubbly things at best, or enshittify other good things at worst. Govern yourselves accordingly.

> stupid pointless bubbly things at best, or enshittify other good things at worst.

oAI just announced like 5bn revenue for half a year, with 13bn projected till end of year. Doesn't seem so pointless now, does it?

Re: Extract-0: A specialized language model for document information extraction

#46

Post by bangaladore »

littlestymaar wrote: post #36
bangaladore wrote: post #26

Earlier quoted context omitted.

> Of course, it is specialized to outperform general models on this specific task in this specific domain with this specific json format for output. My understanding is generally this is not considered an obvious result. In that high parameter generalist models largely outperform lower parameter specialists. The real issue is they tested on data in their training set. * * Incorrect-- Edit misread parent comment.

> The real issue is they tested on data in their training set. Hm, no. They trained on a part of their synthetic set and tested on another part of the set. Or at least that's what they said they did: > from which 1,000 were held out as a benchmark test set. Emphasis mine.

Thanks, rereading it makes it clear that you are correct.

Re: Extract-0: A specialized language model for document information extraction

#47

Post by bangaladore »

disiplus wrote: post #30
bangaladore wrote: post #26

Earlier quoted context omitted.

> Of course, it is specialized to outperform general models on this specific task in this specific domain with this specific json format for output. My understanding is generally this is not considered an obvious result. In that high parameter generalist models largely outperform lower parameter specialists. The real issue is they tested on data in their training set. * * Incorrect-- Edit misread parent comment.

They did not test on the data that they tested, that's not what he wrote.

Thanks, rereading it makes it clear that you are correct.

Re: Extract-0: A specialized language model for document information extraction

#48

Post by -_- »

ImJasonH wrote: post #41
Garlef wrote: post #40

Earlier quoted context omitted.

It's a matter of ROI: When is it worth it to build something specialized?

Is anybody working on making building specialized things easier and cheaper?

Yes! At https://RunRL.com we offer hosted RL fine-tuning, so all you need to provide is a dataset and reward function or environment.

Re: Extract-0: A specialized language model for document information extraction

#49

Post by jrm4 »

NitpickLawyer wrote: post #45
jrm4 wrote: post #15

Seems this gives us a clear bifurcation of what AI is about to do. Open-Source style small players will actually solve problems with AI. And the big money invested things are going to do stupid pointless bubbly things at best, or enshittify other good things at worst. Govern yourselves accordingly.

> stupid pointless bubbly things at best, or enshittify other good things at worst. oAI just announced like 5bn revenue for half a year, with 13bn projected till end of year. Doesn't seem so pointless now, does it?

I'm sorry, did you just defend some big money company purely on the basis of the money they can project in one year?

Did you miss when I said "bubble?" Sigh, y'all are not serious.

Re: Extract-0: A specialized language model for document information extraction

#50

Post by _carltg »

dcastm wrote: post #44

Hey OP, I found some issues with your code: During SFT, it uses the full training dataset[1]: df = pd.read_csv('data/extraction_training_data.csv') And during the evaluation, it uses the middle part of the same dataset[2]: df = pd.read_csv('data/extraction_training_data.csv') df = df[100000:100000+NUM_TEST_SAMPLES] Also, you split train/test/val by chunk and not by document[3]. Then, the model "has seen" the document…

Yes, this is the main concern I have with this result as well.

In other words, rather than plucking different leaves (augments) from the same branch or tree (source dataset), you should be evaluating it on an entirely different tree.

This paper in essence does not have a validation dataset, it only has a training dataset and evaluates on a subpopulation (even though that population was never trained on)

Post reply on HN