Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting financial disclosure and police reports with OpenAI Structured Output

gist.github.com

Post reply on HN
Print view View original

21–30 of 93 posts

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#21

Post by A4ET8a8uTh0 »

beoberha wrote: post #11

Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.

I did not reach the same conclusion so I would be curious if you could provide rationale/basis for your assessment in the link. I am playing with humble llama3 8b here and results for federal register type stuff ( without going into details ) was good for what I was expecting to be.. not great.

edit: Since you mentioned llama explicitly, could you talk a little about the data/source you are using for your resutls. You got me curious and I want to dig a little deeper.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#22

Post by 0tfoaij »

beoberha wrote: post #11

Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.

OpenAI stopped releasing information about their models after gpt-3, which was 175b, but the leaks and rumours that gpt-4 is an 8x220 billion parameter model are most certainly correct. 4o is likely a distilled 220b model. Other commercial offerings are going to be in the same ballpark. Comparing these to llama 3 8b is like comparing a bicycle or a car to a train or cruise ship when you need to transport a few dozen passengers at best. There are local models in the 70-240b range that are more than capable of competing with commercial offerings if you're willing to look at anything that isn't bleeding edge state of the art.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#23

Post by maleldil »

Tiberium wrote: post #20
thatcat wrote: post #14

Earlier quoted context omitted.

I mean, those aren't comparable models. I wonder how the 405b version compares.

You raise a valid point, but 4o is way smaller than 405B. And 4o mini that's described in the article is highly likely <30B (if we're talking dense models).

Is the size of OpenAI's models public, or is this guesswork?

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#24

Post by saturn8601 »

bagels wrote: post #7
matchagaucho wrote: post #4

Earlier quoted context omitted.

Similarly I've found old-school OCR is needed for more reliability.

Combining google's ocr with llm gives OCR superpowers. Tell the llm the text is from an ocr and ask it to correct it.

That sounds like it could be very dangerous when the LLM gets it wrong...

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#25

Post by kgeist »

beoberha wrote: post #11

Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.

In my tests, Llama 3.1 8b was way worse than Llama 2 13b or Solar 13b.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#26

Post by MarkMarine »

matchagaucho wrote: post #4
Zaheer wrote: post #2

Made a small project to help extract structure from documents (pdf,jpg,etc -> JSON or CSV): https://datasqueeze.ai/ There's 10 free pages to extract if anyone wants to give it a try. I've found that just sending a pdf to models doesn't extract it properly especially with longer documents. Have tried to incorporate all best practices into this tool. It's a pet project for now. Lmk if you find it helpful!

Similarly I've found old-school OCR is needed for more reliability.

I've been using this to OCR some photos I took of books and it's remarkable at it. My first pass was just a loop where I'd OCR, feed the text to the model and ask it to normalize into a schema but I found out just sending the image to the model and asking it to OCR and turn it into the shape of data I wanted was so much more accurate.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#27

Post by chaos_emergent »

ec109685 wrote: post #17
tpswa wrote: post #6

Cool work! Correct me if I'm wrong, but I believe to use the new OpenAI structured output that's more reliable, the response_format should be "json_schema" instead of "json_object". It's been a lot more robust for me.

I’ve been using jsonschema since forever with function calling. Does structured output just formalize things?

function calling provides a "hint" in the form of a JSON schema for an LLM to follow. the models are trained to follow provided schemas. If you have really complicated or deeply nested models, they can become less stable at generating schema-conformant JSON.

Structured outputs apply a context-free grammar to the prediction generation so that, for each token generation, only tokens that generate a perfectly conformant JSON schema are considered.

The benefit of doing this is predictability, but there's a trade-off in prediction stability; apparently structured output can constrain the model to generate in a way that takes it off the "happy path" of how it assumes text should be generated.

Happy to link you to some papers I've skimmed on it if you're interested!

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#29

Post by qwe----3 »

maleldil wrote: post #23
Tiberium wrote: post #20

Earlier quoted context omitted.

You raise a valid point, but 4o is way smaller than 405B. And 4o mini that's described in the article is highly likely <30B (if we're talking dense models).

Is the size of OpenAI's models public, or is this guesswork?

If your company has a lot of ex openai employees then you know ;)

And the public numbers are mostly right, the latest values are likely smaller now- they have been working on down sizing everything

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#30

Post by lolinder »

minimaxir wrote: post #12
beoberha wrote: post #11

Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.

The Berkeley Function-Calling Leaderboard tracks function calling/structured data performance from multiple models: https://gorilla.cs.berkeley.edu/leaderboard.html Llama isn't on there but a few finetunes of it (Hermes) are OSS.

Llama 3 70B is on there, ranked 20.
Post reply on HN