Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting financial disclosure and police reports with OpenAI Structured Output

gist.github.com

Post reply on HN
Print view View original

71–80 of 93 posts

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#71

Post by is_true »

infecto wrote: post #66
is_true wrote: post #58

Earlier quoted context omitted.

We tried all models from openai and google to get data from images and all of them made "mistakes". The images are tables with 4 columns and 10 rows of numbers and metadata above that are in a couple of fields. We had thousands of images already loaded and when we tried to check those previously loaded images we found quite a few errors.

Multimodal LLMs are not up for these tasks imo. It can describe an image but its not great on tables and numbers. Now on the other hand, using something like Textract to get the text representation of the table and then feeding that into a LLM was a massive success for us.

LLMs don't offer much value for our use case, almost all values are just numbers

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#72

Post by 1oooqooq »

if you're "parsing" structured or even semi structured data with a LLM.... sigh.

an true scotch engineer know tagged data goes into the other end. but I guess that doesn't align with openai target audience and business goals.

i guess that would be fine to clean the new training data... but then you risk extrapolating hallucinations

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#73

Post by rolandog »

ClearAndPresent wrote: post #50
thenaturalist wrote: post #43

Earlier quoted context omitted.

How are you going to find (not even talking about correcting) hallucinated errors? If money is involved and the LLM produces hallucination errors, how do you handle monetary impacts of such errors? How does that approach scale financially?

Indeed. I anticipate the next Post Office Scandal(1,2) attributed to LLMs. 1 https://en.wikipedia.org/wiki/British_Post_Office_scandal 2 https://www.postofficescandal.uk/

That's awful.

Reminds me of the Dutch childcare benefits scandal [0], where 26,000 families were unfairly labeled as having committed tax fraud (11,000 of which had been targeted via "risk profiling", as they had dual nationalities [1]). Bad policy + automation = disaster. The wikipedia article doesn't fully explain how some automated decisions were made (e.g. You had a typo in a form, therefore all previous benefits were clawed-back; if you owe more than €3.000,- then you're a fraudster and if you called to ask for clarification they wouldn't help you — you're officially labeled a fraudster, you see).

Edit: couldn't find a source for my last statement, but I remember hearing it in an episode of the great Dutch News podcast. I'll see if I can find it.

[0]: https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scand...

[1]: https://www.dutchnews.nl/2021/02/full-scale-parliamentary-in...

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#74

Post by infecto »

is_true wrote: post #71
infecto wrote: post #66

Earlier quoted context omitted.

Multimodal LLMs are not up for these tasks imo. It can describe an image but its not great on tables and numbers. Now on the other hand, using something like Textract to get the text representation of the table and then feeding that into a LLM was a massive success for us.

LLMs don't offer much value for our use case, almost all values are just numbers

Then you should be using something like Textract or other tooling in that space. Multimodal LLMs are no replacement.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#75

Post by infecto »

derivagral wrote: post #69
infecto wrote: post #64

Earlier quoted context omitted.

Would you not want to read the XBRL from the filing? I thought those are now mandatory. This is one of those interesting areas where its hard to innovate because the data is already available from most/all data vendors and its cheap and accurate enough that nobody is going to reinvent those processes but also too expensive for an individual to purchase.

My (admittedly aged) experience with XBRL is that each company was able to define its own fields/format within that spec, and that most didn't agree on common names for common fields. Parsing it wasn't fun.

I have spotty education on the matter but I believe they all conform to a FASB taxonomy so there is at least a list of possible tags in use. I do wonder if any of the big data vendors actually use this though.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#76

Post by is_true »

infecto wrote: post #74
is_true wrote: post #71

Earlier quoted context omitted.

LLMs don't offer much value for our use case, almost all values are just numbers

Then you should be using something like Textract or other tooling in that space. Multimodal LLMs are no replacement.

We use opencv + tesseract and easyocr

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#77

Post by chx »

How do you know the output has anything to do with the input? Hint: you don't. You are building a castle on quicksand. As always, the only thing LLMs are usable for:

https://hachyderm.io/@inthehands/112006855076082650

> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.

> Alas, that does not remotely resemble how people are pitching this technology.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#78

Post by danso »

1oooqooq wrote: post #72

if you're "parsing" structured or even semi structured data with a LLM.... sigh. an true scotch engineer know tagged data goes into the other end. but I guess that doesn't align with openai target audience and business goals. i guess that would be fine to clean the new training data... but then you risk extrapolating hallucinations

The financial disclosures example was meant to be a toy example; with the way U.S. House members file their disclosure reports now, everything should be in a relatively predictable PDF with underlying text [0], but that wasn't always the case [1]. I think this API would've been pretty helpful to orgs like OpenSecrets who in the past had to do record and enter this data manually.

(I wouldn't trust the API alone, but combine it with human readers/validators, i.e., let OpenAI do the data entry part, and have humans do the proofreading)

[0] https://disclosures-clerk.house.gov/public_disc/financial-pd...

[1] https://disclosures-clerk.house.gov/public_disc/financial-pd...

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#79

Post by morkalork »

synthc wrote: post #53

My first job (around 2010) was to extract events from financial news and police reports. We built this huge system with tons of regexes, custom parsers, word lists, ontologies etc. It was a huge effort to get somewhat acceptable accuracy. It is humbling to see that these days a 100 line Python script can do the same thing but better: AI has basically taken over my first job.

Well, your first job today would be writing that 100 line Python script then doing something 100x more interesting with the events than writing truck loads of regexs?

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#80

Post by danofsteel32 »

synthc wrote: post #53

My first job (around 2010) was to extract events from financial news and police reports. We built this huge system with tons of regexes, custom parsers, word lists, ontologies etc. It was a huge effort to get somewhat acceptable accuracy. It is humbling to see that these days a 100 line Python script can do the same thing but better: AI has basically taken over my first job.

I’ve had a similar experience extracting transactions from my PDF bank statements [1]. GPT-4o and GPT-4o-mini perform as well the janky regex parser I wrote a few years ago. The fact that they can zero shot the problem makes me think there’s a lot of bank statements in the training data.

[1] https://dandavis.dev/pnc-virtual-wallet-statement-parser.htm...

Post reply on HN