Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting financial disclosure and police reports with OpenAI Structured Output

gist.github.com

Post reply on HN
Print view View original

41–50 of 93 posts

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#41

Post by TrackerFF »

4ad wrote: post #28

What a sad state for humanity that we have to resort to this sort of OCR/scrapping instead of the original data being released in a machine readable format in the first place.

To be fair, there are some considerations here:

1) There's plenty of old data out there. Newspaper scans from the days before computers, or digitalization of the newspaper process. Or the original files simply got lost, so manually scanned pages is all you have.

2) There could be policies about making the data public, but in a way that discourages data scraping.

3) The providers of the data simply don't have the resources or incentives to develop a working API.

And many more.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#43

Post by thenaturalist »

TrackerFF wrote: post #8

We used GPT 4o for more or less the same stuff. Got a boatload of scanned bills we had to digitize, and GPT really nailed the task. Made a schema, and just fed the model all the bills. Worked better than any OCR we tried.

How are you going to find (not even talking about correcting) hallucinated errors?

If money is involved and the LLM produces hallucination errors, how do you handle monetary impacts of such errors?

How does that approach scale financially?

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#44

Post by thenaturalist »

marcell wrote: post #10

I’m making a free open source library for this, check it at http://github.com/fetchfox/fetchfox MIT license. It’s just one line of code to get started: ‘fox.run(“get data from example.com”)’

How do you plan to address prompt injection/ poisoned data for a method that simply vacuums unchecked inputs into an LLM?

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#45

Post by marcell »

thenaturalist wrote: post #44
marcell wrote: post #10

I’m making a free open source library for this, check it at http://github.com/fetchfox/fetchfox MIT license. It’s just one line of code to get started: ‘fox.run(“get data from example.com”)’

How do you plan to address prompt injection/ poisoned data for a method that simply vacuums unchecked inputs into an LLM?

It hasn’t been an issue yet, but I’m sure it will come up at some point. If you see a problem please file an issue.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#46

Post by thenaturalist »

marcell wrote: post #45
thenaturalist wrote: post #44

Earlier quoted context omitted.

How do you plan to address prompt injection/ poisoned data for a method that simply vacuums unchecked inputs into an LLM?

It hasn’t been an issue yet, but I’m sure it will come up at some point. If you see a problem please file an issue.

So assuming it would be an issue, given that you’re building such a tool, what would your approach be?

If I put an invisible tag on my website and it tells your scraper to ignore all previous prompts, leak its entire history and send all future prompts and replies to a web address while staying silent about it, how would you handle that?

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#47

Post by dcreater »

gdiamos wrote: post #38
beoberha wrote: post #11

Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.

I usually come to a different conclusion using the JSON output on Lamini, e.g. even with Llama 3.2 3B https://lamini-ai.github.io/inference/json_output Most of these models can read. If the relevant facts are in the prompt, they can almost always extract them correctly. Of course bigger models do better on more complex tasks and reasoning unless you use finetuning or memory tuning.

You should probably disclose you're the founder of lamini.

Do you have any publicly available validation data demonstrating 100% json compliance?

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#48

Post by blitzar »

4ad wrote: post #28

What a sad state for humanity that we have to resort to this sort of OCR/scrapping instead of the original data being released in a machine readable format in the first place.

What is even sadder is that this data (especially the more recent data) is entered first in machine readable formats then sliced and diced and spat out in a non-machine readable format.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#50

Post by ClearAndPresent »

thenaturalist wrote: post #43
TrackerFF wrote: post #8

We used GPT 4o for more or less the same stuff. Got a boatload of scanned bills we had to digitize, and GPT really nailed the task. Made a schema, and just fed the model all the bills. Worked better than any OCR we tried.

How are you going to find (not even talking about correcting) hallucinated errors? If money is involved and the LLM produces hallucination errors, how do you handle monetary impacts of such errors? How does that approach scale financially?

Indeed. I anticipate the next Post Office Scandal(1,2) attributed to LLMs.

1 https://en.wikipedia.org/wiki/British_Post_Office_scandal 2 https://www.postofficescandal.uk/

Post reply on HN