Replace OCR with Vision Language Models • Hacker News Board

Re: Replace OCR with Vision Language Models

#91

Post by UnlockedSecrets » Thu, Feb 27, 2025, 1:17 PM UTC

j_bum wrote: post #89

rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

This is naive, but can you ask the model to provide a confidence rating for sections of the document?

You can ask, and it will be made up not grounded in reality

Re: Replace OCR with Vision Language Models

#92

Post by raxxorraxor » Thu, Feb 27, 2025, 1:23 PM UTC

orliesaurus wrote: post #4

I think OCR tools are good at what they say on the box, recognizing characters on a piece of paper etc. If I understand this right, the advantage of using a vision language model is the added logic that you can say things like: "Clearly this is a string, but does it look like a timestamp or something else?"

This has always been part of the complete OCR package as far as I know. The raw result of an OCR constantly fails to differentiate 1 l I i | or other similar symbols/letters.

Maybe this necessary step can be improved and altered with a VLM. There is also the preprocessing where the image get its perspective corrected. Not sure how well a VLM performs here.

As you said, I think combining these techniques will be the most efficient way forward.

Re: Replace OCR with Vision Language Models

#93

Post by j_bum » Thu, Feb 27, 2025, 1:30 PM UTC

UnlockedSecrets wrote: post #91

j_bum wrote: post #89

Earlier quoted context omitted.

This is naive, but can you ask the model to provide a confidence rating for sections of the document?

You can ask, and it will be made up not grounded in reality

Sure, but I’m curious if it would serve to provide some self-regulation.

E.g., all of this “thinking” trend that’s happening. It would be interesting if the model does a first pass, scored its individual outputs, then reviews its scores and censors/flags scores that are low.

I know it’s all “made up”, but generally I have a lot of success asking the model to give 0-1 ratings on confidence for its answers, especially for new niche questions that are likely out of the training set.

Re: Replace OCR with Vision Language Models

#94

Post by sva_ » Thu, Feb 27, 2025, 1:35 PM UTC

BrannonKing wrote: post #38

What I want: take scan/photo of a document (including a full book), pass it to the language model, and then get out a Latex document that matches the original document exactly (minus the copier/camera glitches and angles). I feel like some kind of reinforcement learning model would be possible for this. It should be able to learn to generate Latex that reproduces the exact image, pixel for pixel (learning which pixel…

Did you try mathpix? Not sure about full pages, but it is pretty good at eqn

Re: Replace OCR with Vision Language Models

#95

Post by thatjoeoverthr » Thu, Feb 27, 2025, 1:35 PM UTC

j_bum wrote: post #89

rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

This is naive, but can you ask the model to provide a confidence rating for sections of the document?

More broadly, it’s not trained to have any self awareness and this is a factor in other “hallucinations”. If you ask, for example, to describe the “marathon crater”, it doesn’t recognize that there’s no such thing in its corpus, but will instead start by writing an answer (“sure! The marathon crater is..”) and freestyle from there. Same if you ask it why it did something, or details about itself, etc. You should access one directly (not through an app like chatGPT) and build a careful suite of tests to learn more. Really fascinating.

Re: Replace OCR with Vision Language Models

#96

Post by ptx » Thu, Feb 27, 2025, 1:45 PM UTC

egorfine wrote: post #81

I had a need to scan serial numbers from Apple's product boxes out of pictures taken by a clueless person on their phone. All OCR tools failed. Vision model did the trick so well it's not even funny to discuss anything further. "This is a picture of Apple product box. Find and return only the serial number of the product as found on a label. Return 'none' if no serial number can be found".

Did you check if all the numbers were correct?

Re: Replace OCR with Vision Language Models

#97

Post by egorfine » Thu, Feb 27, 2025, 2:04 PM UTC

ptx wrote: post #96

egorfine wrote: post #81

I had a need to scan serial numbers from Apple's product boxes out of pictures taken by a clueless person on their phone. All OCR tools failed. Vision model did the trick so well it's not even funny to discuss anything further. "This is a picture of Apple product box. Find and return only the serial number of the product as found on a label. Return 'none' if no serial number can be found".

Did you check if all the numbers were correct?

Of course. There was a little piece of code to query Apple for S/N data and it validated whether it was correct.

Re: Replace OCR with Vision Language Models

#98

Post by ttyprintk » Thu, Feb 27, 2025, 2:08 PM UTC

j_bum wrote: post #89

rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

This is naive, but can you ask the model to provide a confidence rating for sections of the document?

It’s not naive; tesseract does this.

Re: Replace OCR with Vision Language Models

#99

Post by ungerik » Thu, Feb 27, 2025, 2:08 PM UTC

constantinum wrote: post #59

rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

The primary issue with LLMs is hallucination, which can lead to incorrect data and flawed business decisions. For example, Llamaparse( https://docs.llamaindex.ai/en/stable/llama_cloud/llama_parse... ) uses LLMs for PDF text extraction but faces hallucination problems. See this issue for more details: https://github.com/run-llama/llama_parse/issues/420 . For those interested, try LLMWhisperer( https://unstract.com/llm…

Those images look exactly like what you get from every OCR tool out there if you use the XY information.

Re: Replace OCR with Vision Language Models

#100

Post by rafram » Thu, Feb 27, 2025, 2:16 PM UTC

j_bum wrote: post #93

UnlockedSecrets wrote: post #91

Earlier quoted context omitted.

You can ask, and it will be made up not grounded in reality

Sure, but I’m curious if it would serve to provide some self-regulation. E.g., all of this “thinking” trend that’s happening. It would be interesting if the model does a first pass, scored its individual outputs, then reviews its scores and censors/flags scores that are low. I know it’s all “made up”, but generally I have a lot of success asking the model to give 0-1 ratings on confidence for its answers, especially…

It doesn’t. Asking for confidence doesn’t prompt it to make multiple passes, and there’s no real concept of “passes” when you’re talking about non-reasoning models. The model takes in text and image tokens and spits out the text tokens that logically follow them. You can try asking it to think step by step, or you can use a reasoning model that essentially bakes that behavior into the training data, but I haven’t found that to be very useful for OCR tasks. If the encoded version of your image doesn’t resolve to text in the model’s latent space, it never will, no matter how much the model “reasons” (spits out intermediate text tokens) before giving a final answer.