This is awesome!
Replace OCR with Vision Language Models
31–40 of 130 posts
Re: Replace OCR with Vision Language Models
#32It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.)
I tried using a VLM to recognize handwritten text in genealogical sources, and it made up names and dates that sort of fit the vibe of the document when it couldn’t read the text! They sounded right for the ethnicity and time period but were entirely fake. There’s no way to ground the model using the source text when the model is your OCR.
Re: Replace OCR with Vision Language Models
#33Maybe I’m being greedy but is it possible to have a vLLM detect when a portion is an image? I want to convert some handwritten notes into markdown but some portion are diagrams. I want the vLLM to extract the diagrams to embed into the markdown output
Re: Replace OCR with Vision Language Models
#34It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…
This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.
Re: Replace OCR with Vision Language Models
#35Maybe I’m being greedy but is it possible to have a vLLM detect when a portion is an image? I want to convert some handwritten notes into markdown but some portion are diagrams. I want the vLLM to extract the diagrams to embed into the markdown output
We have successfully tested the model with vLLM and plan to release it across multiple inference server frameworks, including vLLM and OLAMA.
Re: Replace OCR with Vision Language Models
#36It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…
Thing is, the majority of OCR errors aren't character issues, but layout issues. Things like complex tables with cells being returned under the wrong header. And if the numbers in an income statement are one column off creates a pretty big risk.
Confidence intervals are a red herring. And only as good as the code interpreting them. If the OCR model gives you back 500 words all ranging from 0.70 to 0.95 confidence, what do you do? Reject the entire document if there's a single value below 0.90?
If so you'd be passing every single document to a human review, and might as well not run the OCR. But if you're not rejecting based on CI, then you're exposed to just as much risk as using an LLM.
Re: Replace OCR with Vision Language Models
#37We recently published an open source benchmark [1] specifically for evaluating VLM vs OCR. And generally the VLMs did much better than the traditional OCR models.
VLM highlights:
- Handwriting. Being contextually aware helps here. i.e. they read the document like a human would, interpreting the whole word/sentence instead of character by character
- Charts/Infographics. VLMs can actually interpret charts or flow diagrams into a text format. Including things like color coded lines.
Traditional OCR highlights:
- Standardized documents (e.x. US tax forms that they've been trained on)
- Dense text. Imagine textbooks and multi column research papers. This is the easiest OCR use case, but VLMS really struggle as the number of output tokens increase.
- Bounding boxes. There still isn't really a model that gives super precise bounding boxes. Supposedly Gemini and Qwen were trained for it, but they don't perform as well as traditional models.
There's still a ton of room for improvement, but especially with models like Gemini the accuracy/cost is really competitive.
Re: Replace OCR with Vision Language Models
#38What I want: take scan/photo of a document (including a full book), pass it to the language model, and then get out a Latex document that matches the original document exactly (minus the copier/camera glitches and angles). I feel like some kind of reinforcement learning model would be possible for this. It should be able to learn to generate Latex that reproduces the exact image, pixel for pixel (learning which pixels are just noise).
Re: Replace OCR with Vision Language Models
#39Looks cool!
May also be interested in Allen AI's OCR tool olmOCR they just released too [1][2]. They say "convert a million PDF pages for only $190 USD".
[1] https://github.com/allenai/olmocr [2] https://arxiv.org/abs/2502.18443
Re: Replace OCR with Vision Language Models
#40Looks cool! May also be interested in Allen AI's OCR tool olmOCR they just released too [1][2]. They say "convert a million PDF pages for only $190 USD". [1] https://github.com/allenai/olmocr [2] https://arxiv.org/abs/2502.18443
The issue with that promise is that anyone can convert pdfs, the question is whether the conversions are correct or whether you have
Income Expenses 200 100
On one document, and
Income Expenses 20 0100
On others.
There's no shortage of products that tried to solve this problem from scratch (or by piggybacking on other projects) and called it a day without worrying about the huge problem that is quality and parseability.
The most robust players just give you the coordinates of a glyph and you are on your own: Textract, PDFBox.