Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.
Extracting financial disclosure and police reports with OpenAI Structured Output
11–20 of 93 posts
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#12Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.
The Berkeley Function-Calling Leaderboard tracks function calling/structured data performance from multiple models: https://gorilla.cs.berkeley.edu/leaderboard.html
Llama isn't on there but a few finetunes of it (Hermes) are OSS.
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#13Made a small project to help extract structure from documents (pdf,jpg,etc -> JSON or CSV): https://datasqueeze.ai/ There's 10 free pages to extract if anyone wants to give it a try. I've found that just sending a pdf to models doesn't extract it properly especially with longer documents. Have tried to incorporate all best practices into this tool. It's a pet project for now. Lmk if you find it helpful!
Is this simply the OCR bits to feed to openai structured output?
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#14Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.
I mean, those aren't comparable models. I wonder how the 405b version compares.
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#15> Note that this example simply passes a PNG screenshot of the PDF to OpenAI's API — results may be different/more efficient if you send it the actual PDF. OpenAI's API only accepts images: https://platform.openai.com/docs/guides/vision To my knowledge, all the LLM services that take in PDF input do their own text extraction of the PDF before feeding it to an LLM.
or convert PDF to image and send that. We’ve done it for things that textract completely mangled, but sonnet has no problem. Especially tables built out of text characters from very old systems
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#16> Note that this example simply passes a PNG screenshot of the PDF to OpenAI's API — results may be different/more efficient if you send it the actual PDF. OpenAI's API only accepts images: https://platform.openai.com/docs/guides/vision To my knowledge, all the LLM services that take in PDF input do their own text extraction of the PDF before feeding it to an LLM.
I don’t think it does OCR. It’s able to use the structure of the PDF to guide the parsing.
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#17Cool work! Correct me if I'm wrong, but I believe to use the new OpenAI structured output that's more reliable, the response_format should be "json_schema" instead of "json_object". It's been a lot more robust for me.
I’ve been using jsonschema since forever with function calling. Does structured output just formalize things?
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#18Cool work! Correct me if I'm wrong, but I believe to use the new OpenAI structured output that's more reliable, the response_format should be "json_schema" instead of "json_object". It's been a lot more robust for me.
I’ve been using jsonschema since forever with function calling. Does structured output just formalize things?
Structured output uses "constrained decoding" under the hood. They convert the JSON schema to a context free grammar so that when the model samples tokens, invalid tokens are masked to have a probability of zero. It's much less likely to go off the rails.
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#19Cool work! Correct me if I'm wrong, but I believe to use the new OpenAI structured output that's more reliable, the response_format should be "json_schema" instead of "json_object". It's been a lot more robust for me.
I may be reading the documentation wrong [0], but I think if you specify `json_schema`, you actually have to provide a schema. I get this error when I do `response_format={"type": "json_schema"}`:
openai.BadRequestError: Error code: 400 - {'error': {'message': "Missing required parameter: 'response_format.json_schema'.", 'type': 'invalid_request_error', 'param': 'response_format.json_schema', 'code': 'missing_required_parameter'}}
[0] https://platform.openai.com/docs/guides/structured-outputs/s...
Re: Extracting financial disclosure and police reports with OpenAI Structured Output
#20Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.
I mean, those aren't comparable models. I wonder how the 405b version compares.
You raise a valid point, but 4o is way smaller than 405B. And 4o mini that's described in the article is highly likely <30B (if we're talking dense models).