Extracting financial disclosure and police reports with OpenAI Structured Output

Post by thenaturalist » Tue, Oct 15, 2024, 12:12 PM UTC

is_true wrote: post #58

thenaturalist wrote: post #43

Earlier quoted context omitted.

How are you going to find (not even talking about correcting) hallucinated errors? If money is involved and the LLM produces hallucination errors, how do you handle monetary impacts of such errors? How does that approach scale financially?

We tried all models from openai and google to get data from images and all of them made "mistakes". The images are tables with 4 columns and 10 rows of numbers and metadata above that are in a couple of fields. We had thousands of images already loaded and when we tried to check those previously loaded images we found quite a few errors.

Post by msp26 » Tue, Oct 15, 2024, 12:23 PM UTC

druskacik wrote: post #55

Data extraction is definitely one of the most useful functions of LLM, however, in my experience a large model is necessary for a reliable extraction - I tested smaller, open-weights models and the performance was not sufficient. I wonder, did anyone try to fine-tune a model specifically for general formatted data extraction? My naive thinking is that this should be pretty doable - after all, it's basically just rest…

Post by druskacik » Tue, Oct 15, 2024, 12:36 PM UTC

msp26 wrote: post #63

druskacik wrote: post #55

Data extraction is definitely one of the most useful functions of LLM, however, in my experience a large model is necessary for a reliable extraction - I tested smaller, open-weights models and the performance was not sufficient. I wonder, did anyone try to fine-tune a model specifically for general formatted data extraction? My naive thinking is that this should be pretty doable - after all, it's basically just rest…

Have you looked into structured generation with a library like outlines? https://github.com/dottxt-ai/outlines

Post by alach11 » Tue, Oct 15, 2024, 12:47 PM UTC

TrainedMonkey wrote: post #51

rcarmo wrote: post #42

I’ve had pretty dismal results doing the same with spreadsheets—even with the data nicely tagged (and numbers directly adjacent to the labels) GPT-4o would completely make up figures to satisfy the JSON schema passed to it. YMMV.

I wonder if adversarial model which looks at user input & LLM output and predicts whether output is accurate + maybe output what is not accurate. This worked pretty well for image generation.

Post by derivagral » Tue, Oct 15, 2024, 1:00 PM UTC

infecto wrote: post #64

mmsc wrote: post #60

Adding to the list of "now try it with".... The SEC's EDGAR database (which is for SEC filings) is another nightmare ready to end. Extracting individual sections from a filing is, afaik, impossible pragmatically. I tried making two parsers: https://github.com/MegaManSec/SEC-Feed-Parser and https://github.com/MegaManSec/SEC-sec-incident-notifier but they're just hacks. Then just link it up to your automated investment…

Would you not want to read the XBRL from the filing? I thought those are now mandatory. This is one of those interesting areas where its hard to innovate because the data is already available from most/all data vendors and its cheap and accurate enough that nobody is going to reinvent those processes but also too expensive for an individual to purchase.