Is there an automated way to check results and reduce hallucinations? Would it help to do a second pass with another LLM as a sanity check to see if numbers match?
Extracting financial disclosure and police reports with OpenAI Structured Output
81–90 of 93 posts
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#82The SEC has a well defined API with EDGAR.
https://jdsemrau.substack.com/p/mem0-building-a-sec-10k-anal...
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#83Does anyone follow Vik's work? eg https://x.com/VikParuchuri/status/1846153661791011158
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#84What a sad state for humanity that we have to resort to this sort of OCR/scrapping instead of the original data being released in a machine readable format in the first place.
I'd like to see financial transactions and purchases abide by some json format standard, metadata and a list of items with full product name, quantity purchased, total unit volume/amount of product, price, and unit price.
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#85Data extraction is definitely one of the most useful functions of LLM, however, in my experience a large model is necessary for a reliable extraction - I tested smaller, open-weights models and the performance was not sufficient. I wonder, did anyone try to fine-tune a model specifically for general formatted data extraction? My naive thinking is that this should be pretty doable - after all, it's basically just rest…
Jina did something related for extracting content from raw HTML and wrote about the techniques they used here: https://jina.ai/news/reader-lm-small-language-models-for-cle... .. in my tests, the 1.5bn model works extremely well, though the open model is non commercial.
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#86We used GPT 4o for more or less the same stuff. Got a boatload of scanned bills we had to digitize, and GPT really nailed the task. Made a schema, and just fed the model all the bills. Worked better than any OCR we tried.
How are you going to find (not even talking about correcting) hallucinated errors? If money is involved and the LLM produces hallucination errors, how do you handle monetary impacts of such errors? How does that approach scale financially?
Not the OP, but if doing this at scale, I'd consider a quorum approach using several models and looking for a majority to agree (otherwise bump it for human review). You could also get two different approaches out of each model by using purely the model and external OCR + model and compare those too.
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#87Earlier quoted context omitted.
How are you going to find (not even talking about correcting) hallucinated errors? If money is involved and the LLM produces hallucination errors, how do you handle monetary impacts of such errors? How does that approach scale financially?
Not the OP, but if doing this at scale, I'd consider a quorum approach using several models and looking for a majority to agree (otherwise bump it for human review). You could also get two different approaches out of each model by using purely the model and external OCR + model and compare those too.
I’m working on a problem in this space, and that’s the approach I’m taking.
More detailed explanation: I have to OCR dense, handwritten data using technical codes. Luckily, the form designers included intermediate steps. The intermediate fields are amenable to Textract, so I can use a multimodal model to OCR the full table and then error check.
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#88My first job (around 2010) was to extract events from financial news and police reports. We built this huge system with tons of regexes, custom parsers, word lists, ontologies etc. It was a huge effort to get somewhat acceptable accuracy. It is humbling to see that these days a 100 line Python script can do the same thing but better: AI has basically taken over my first job.
Well, your first job today would be writing that 100 line Python script then doing something 100x more interesting with the events than writing truck loads of regexs?
No, his first job would be a more senior developer writing 100 line Python script instead of hiring an intern to write a truck load of RegExs. After that dev saved time just writing the script over mentoring/explaining/hiring the intern, that dev would then do the more interesting things with the events.
That is, his first job is now gone.
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#89Is there an automated way to check results and reduce hallucinations? Would it help to do a second pass with another LLM as a sanity check to see if numbers match?
This is what I am implementing a the moment (together with sampling for errors).
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#90Very cool and real application of LLMs. Although hallucinations is still something to be very wary of.