Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting financial disclosure and police reports with OpenAI Structured Output

gist.github.com

Post reply on HN
Print view View original

31–40 of 93 posts

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#31

Post by Baeocystin »

0tfoaij wrote: post #22
beoberha wrote: post #11

Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.

OpenAI stopped releasing information about their models after gpt-3, which was 175b, but the leaks and rumours that gpt-4 is an 8x220 billion parameter model are most certainly correct. 4o is likely a distilled 220b model. Other commercial offerings are going to be in the same ballpark. Comparing these to llama 3 8b is like comparing a bicycle or a car to a train or cruise ship when you need to transport a few dozen…

Any pointers on where we can check the best local models per amount of VRAM available? I only have consumer level cards available, but I would think something that just fits in to a 24Gb card should noticably outperform something scaled for an 8Gb card, yes?

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#32

Post by [deleted] »

chaos_emergent wrote: post #27
ec109685 wrote: post #17

Earlier quoted context omitted.

I’ve been using jsonschema since forever with function calling. Does structured output just formalize things?

function calling provides a "hint" in the form of a JSON schema for an LLM to follow. the models are trained to follow provided schemas. If you have really complicated or deeply nested models, they can become less stable at generating schema-conformant JSON. Structured outputs apply a context-free grammar to the prediction generation so that, for each token generation, only tokens that generate a perfectly conformant…

[deleted]

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#33

Post by pmg0 »

chaos_emergent wrote: post #27
ec109685 wrote: post #17

Earlier quoted context omitted.

I’ve been using jsonschema since forever with function calling. Does structured output just formalize things?

function calling provides a "hint" in the form of a JSON schema for an LLM to follow. the models are trained to follow provided schemas. If you have really complicated or deeply nested models, they can become less stable at generating schema-conformant JSON. Structured outputs apply a context-free grammar to the prediction generation so that, for each token generation, only tokens that generate a perfectly conformant…

Could you share some of those papers? I had a great discussion with Marc Fischer from the LMQL team [0] on this topic while at ICML earlier this year. Their work recommended decoding to natural language templates with mad lib-style constraints to follow that “happy path” you refer to, instead of decoding to a (relatively more specific latent) JSON schema [1]. Since you provided a template and knew the targeted tokens for generation you could strip your structured content out of the message. This technique also allowed for beam search where you can optimize tokens which lead to the tokens contain your expected strings, avoiding some weird token concatenation process. Really cool stuff!

[0] https://lmql.ai/ [1] https://arxiv.org/abs/2311.04954

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#34

Post by fnord77 »

Baeocystin wrote: post #31
0tfoaij wrote: post #22

Earlier quoted context omitted.

OpenAI stopped releasing information about their models after gpt-3, which was 175b, but the leaks and rumours that gpt-4 is an 8x220 billion parameter model are most certainly correct. 4o is likely a distilled 220b model. Other commercial offerings are going to be in the same ballpark. Comparing these to llama 3 8b is like comparing a bicycle or a car to a train or cruise ship when you need to transport a few dozen…

Any pointers on where we can check the best local models per amount of VRAM available? I only have consumer level cards available, but I would think something that just fits in to a 24Gb card should noticably outperform something scaled for an 8Gb card, yes?

lm studio tells you what models fit in your available RAM, with or without quantization

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#36

Post by bagels »

saturn8601 wrote: post #24
bagels wrote: post #7

Earlier quoted context omitted.

Combining google's ocr with llm gives OCR superpowers. Tell the llm the text is from an ocr and ask it to correct it.

That sounds like it could be very dangerous when the LLM gets it wrong...

Depends what you're using it for. If you're relying on OCR, you've already got to accept some amount of error.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#37

Post by gloosx »

TrackerFF wrote: post #8

We used GPT 4o for more or less the same stuff. Got a boatload of scanned bills we had to digitize, and GPT really nailed the task. Made a schema, and just fed the model all the bills. Worked better than any OCR we tried.

Did you finally balance out lol? If you didn't, would you approach finding a mistake by going through each bill manually?

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#38

Post by gdiamos »

beoberha wrote: post #11

Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.

I usually come to a different conclusion using the JSON output on Lamini, e.g. even with Llama 3.2 3B

https://lamini-ai.github.io/inference/json_output

Most of these models can read. If the relevant facts are in the prompt, they can almost always extract them correctly.

Of course bigger models do better on more complex tasks and reasoning unless you use finetuning or memory tuning.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#39

Post by int_19h »

beoberha wrote: post #11

Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.

Your problem isn't that you're using a local model. It's that you're using an 8b model. The stuff you're comparing it to is two orders of magnitude larger.

Re: Extracting financial disclosure and police reports with OpenAI Structured Output

#40

Post by tpm »

beoberha wrote: post #11

Stuff like this shows how much better the commercial models are than local models. I’ve been playing around with fairly simple structured information extraction from news articles and fail to get any kind of consistent behavior from llama3.1:8b. Claude and chatGPT do exactly what I want without fail.

In my experience the Qwen2-VL models are great at this.
Post reply on HN