Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run structured extraction on documents/images locally with Ollama and Pydantic

github.com

Post reply on HN
Print view View original

11–20 of 32 posts

Re: Run structured extraction on documents/images locally with Ollama and Pydantic

#11

Post by fzysingularity »

jasonjmcghee wrote: post #7

I've used "structured output" (with supplied schema) on Google and openai, and function calling / tool use on those, anthropic and others- and afaict they are functionally the same (if you force a specific function / schema). Has someone had a different experience?

They’re slightly nuanced - every model provider has a slightly different Pydantic /JSON schema compatibility (i.e for handling Literals, Unions, nested subtypes etc).

So you end up hitting roadblocks for seemingly simple Pydantic schemas.

Re: Run structured extraction on documents/images locally with Ollama and Pydantic

#14

Post by andrewinardeer »

fzysingularity wrote: post #5
jbmsf wrote: post #3

Interesting. We're using a SAAS solution for document extraction right now. I don't know if it's in our interest to build out more but I do like the idea of keeping extraction local.

Cool, what types of documents do you currently handle? We could share some of our learnings/schemas here too.

Different commenter; Here I'm extracting data from commerical invoices, POs and bills of lading.

Re: Run structured extraction on documents/images locally with Ollama and Pydantic

#15

Post by jasonjmcghee »

fzysingularity wrote: post #11
jasonjmcghee wrote: post #7

I've used "structured output" (with supplied schema) on Google and openai, and function calling / tool use on those, anthropic and others- and afaict they are functionally the same (if you force a specific function / schema). Has someone had a different experience?

They’re slightly nuanced - every model provider has a slightly different Pydantic /JSON schema compatibility (i.e for handling Literals, Unions, nested subtypes etc). So you end up hitting roadblocks for seemingly simple Pydantic schemas.

I meant between "structured output" and "function calling". Afaict one is outputting according to a schema and the other is outputting according to a schema... which will be used as the parameters to a function.

But they seem to be considered disparate concepts. So I'm trying to understand if there's some additional nuance I'm missing.

Re: Run structured extraction on documents/images locally with Ollama and Pydantic

#16

Post by fzysingularity »

youknowwhentous wrote: post #13

This seems to work for videos as well. Pretty cool demo and very nice interface for the pydantic types.

Yes, good catch. We'll be adding several more schemas for videos in the next few weeks.

A few video schemas are already added to the main catalog: https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/c...

Re: Run structured extraction on documents/images locally with Ollama and Pydantic

#17

Post by fzysingularity »

andrewinardeer wrote: post #14
fzysingularity wrote: post #5

Earlier quoted context omitted.

Cool, what types of documents do you currently handle? We could share some of our learnings/schemas here too.

Different commenter; Here I'm extracting data from commerical invoices, POs and bills of lading.

Ah cool, care to share a few examples? We can probably add those schemas in the next few days if there's enough folks who could benefit from this. A basic invoice schema is already there: https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/s...

You can see some of the qualitative results on GPT4o, Gemini, Llama 3.2 11B, Phi-4 here: https://github.com/vlm-run/vlmrun-hub?tab=readme-ov-file#-qu...

Re: Run structured extraction on documents/images locally with Ollama and Pydantic

#19

Post by guntars »

jasonjmcghee wrote: post #15
fzysingularity wrote: post #11

Earlier quoted context omitted.

They’re slightly nuanced - every model provider has a slightly different Pydantic /JSON schema compatibility (i.e for handling Literals, Unions, nested subtypes etc). So you end up hitting roadblocks for seemingly simple Pydantic schemas.

I meant between "structured output" and "function calling". Afaict one is outputting according to a schema and the other is outputting according to a schema... which will be used as the parameters to a function. But they seem to be considered disparate concepts. So I'm trying to understand if there's some additional nuance I'm missing.

With function calls the model may or may not output something that matches the schema, with structured output the schema is enforced at the logit level.

Re: Run structured extraction on documents/images locally with Ollama and Pydantic

#20

Post by fzysingularity »

jasonjmcghee wrote: post #15
fzysingularity wrote: post #11

Earlier quoted context omitted.

They’re slightly nuanced - every model provider has a slightly different Pydantic /JSON schema compatibility (i.e for handling Literals, Unions, nested subtypes etc). So you end up hitting roadblocks for seemingly simple Pydantic schemas.

I meant between "structured output" and "function calling". Afaict one is outputting according to a schema and the other is outputting according to a schema... which will be used as the parameters to a function. But they seem to be considered disparate concepts. So I'm trying to understand if there's some additional nuance I'm missing.

Ah ok, I misunderstood. As far as I've seen, structured outputs is essentially "json-mode" with some constraints (i.e. guided decoding over a known schema) - so the model effectively emits valid JSON that conforms to the schema. In function calling, the model is asked to emit "code" that conforms to some function parameter spec. You could use json-mode for function-calling, but probably not the other way around.

I've generally found json-mode to be more useful than function-calling, even though the latter is what everyone fixates on because of it's obvious use in agents.

Post reply on HN