Run structured extraction on documents/images locally with Ollama and Pydantic
1–10 of 32 posts
Re: Run structured extraction on documents/images locally with Ollama and Pydantic
#2We put together an open-source collection of Pydantic schemas for a variety of document categories (W2 filings, invoices etc.), including instructions for how to get structured JSON responses from any visual input with the model of your choosing. Run everything locally.
Re: Run structured extraction on documents/images locally with Ollama and Pydantic
#3Interesting. We're using a SAAS solution for document extraction right now. I don't know if it's in our interest to build out more but I do like the idea of keeping extraction local.
Re: Run structured extraction on documents/images locally with Ollama and Pydantic
#4Interesting. We're using a SAAS solution for document extraction right now. I don't know if it's in our interest to build out more but I do like the idea of keeping extraction local.
Our customers insist we run everything on their docs locally.
Re: Run structured extraction on documents/images locally with Ollama and Pydantic
#5Interesting. We're using a SAAS solution for document extraction right now. I don't know if it's in our interest to build out more but I do like the idea of keeping extraction local.
Cool, what types of documents do you currently handle? We could share some of our learnings/schemas here too.
Re: Run structured extraction on documents/images locally with Ollama and Pydantic
#6Interesting. We're using a SAAS solution for document extraction right now. I don't know if it's in our interest to build out more but I do like the idea of keeping extraction local.
Our customers insist we run everything on their docs locally.
Absolutely, we’ve been hearing the same from our customers - which is why we thought it makes sense to open source a bunch of schemas so that they’re reusable and compatible across various inference providers (esp. Ollama/local ones).
Re: Run structured extraction on documents/images locally with Ollama and Pydantic
#7I've used "structured output" (with supplied schema) on Google and openai, and function calling / tool use on those, anthropic and others- and afaict they are functionally the same (if you force a specific function / schema). Has someone had a different experience?
Re: Run structured extraction on documents/images locally with Ollama and Pydantic
#8Have you folks tried finetuning models for data extraction from visual data?
Re: Run structured extraction on documents/images locally with Ollama and Pydantic
#9I'd really like to play with Qwen2.5-VL at some point, perhaps for reading data-sheets for microchips. Nicely for some applications, it's also very good at reporting position of what it finds, which many ML tools are pretty mediocre at. https://qwenlm.github.io/blog/qwen2.5-vl/
Not really this application, but QvQ for visual reasoning is also impressive. https://qwenlm.github.io/blog/qvq-72b-preview/
Meta has used Qwen as the basis for their Apollo research. https://arxiv.org/abs/2412.10360
Re: Run structured extraction on documents/images locally with Ollama and Pydantic
#10I'd really like to play with Qwen2.5-VL at some point, perhaps for reading data-sheets for microchips. Nicely for some applications, it's also very good at reporting position of what it finds, which many ML tools are pretty mediocre at. https://qwenlm.github.io/blog/qwen2.5-vl/ Not really this application, but QvQ for visual reasoning is also impressive. https://qwenlm.github.io/blog/qvq-72b-preview/ Meta has used Qw…
Is Qwen2.5-VL on Ollama? Could give it a try with a few of the schemas we have.
We’ve locally tested with Llama 3.2 11B Vision on Ollama: https://github.com/vlm-run/vlmrun-hub/blob/main/tests/benchm...
FWIW I think Ollama structured outputs API is quite buggy compared to the HF transformers variant.