VLM's can't replace ocr one to one.. most hosted multimodal models seem to have a classical OCR (tesseract-based) step in their inference loop
Replace OCR with Vision Language Models
111–120 of 130 posts
Re: Replace OCR with Vision Language Models
#112Wouldn’t VLM be susceptible to prompt injection?
Re: Replace OCR with Vision Language Models
#113Earlier quoted context omitted.
Tesseract doesn’t use an LLM. LLMs don’t know how confident they are; Tesseract’s model does.
Kind of. Tesseract's confidence is just a raw model probability output. You could easily use the entropy associated with each token coming out of an LLM to do the same thing.
True, but LLM token probability doesn't map nearly as cleanly to "how readable was the text".
Re: Replace OCR with Vision Language Models
#114It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…
Agree wholeheartedly. Modern OCR is astonishingly good, more importantly it's deterministically so. It's failure modes, when it's unable to read the text, are recognizably failures. Results for VLM accuracy & precision are not good. https://arxiv.org/html/2406.04470v1#S4
which solutions would you classify as "modern OCR"
are we talking tesseract or something?
Re: Replace OCR with Vision Language Models
#115Earlier quoted context omitted.
Kind of. Tesseract's confidence is just a raw model probability output. You could easily use the entropy associated with each token coming out of an LLM to do the same thing.
True, but LLM token probability doesn't map nearly as cleanly to "how readable was the text".
Why not though? Both kinds of models jumble around the data and spit out a probability distribution. Why is the tesseract distribution inherently more explainable (aside from the UI/UX problem of the uncertainty being per-token instead of per-character)?
Re: Replace OCR with Vision Language Models
#116Earlier quoted context omitted.
Thing is, the majority of OCR errors aren't character issues, but layout issues. Things like complex tables with cells being returned under the wrong header. And if the numbers in an income statement are one column off creates a pretty big risk. Confidence intervals are a red herring. And only as good as the code interpreting them. If the OCR model gives you back 500 words all ranging from 0.70 to 0.95 confidence, wh…
> But if you're not rejecting based on CI, then you're exposed to just as much risk as using an LLM. That's not true. LLMs and OCR have very different failure modes. With LLMs, there is unbounded potential for hallucination, and the entire document is at risk. For example: if something in the lower right-hand corner of the page takes the model to a sparsely sampled part of the latent space, it can end up deciding tha…
100% this, combining traditional OCR with VLMs that can work with bounding boxes so that you can correlate the two is the way to go.
Re: Replace OCR with Vision Language Models
#117It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…
I've been using gemini 2 flash to extract financial data, within my sample which is perhaps small (probably 1000 entries so far), I've had one single error only so like a 99.9% success rate. (There's slightly more errors if I ask it to add numbers but this isn't OCR and a bit more of a reach, although it is very good at this too regardless). Many hallucinations can be avoided by telling it to use null if there is no…
And by using two different systems (say Gem plus ChatGPT) you essentially reduce chances of hallucination to zero, no? You would need to be VERY unlucky to find to LLMs hallucinating the exact same response.
Re: Replace OCR with Vision Language Models
#118Earlier quoted context omitted.
This is naive, but can you ask the model to provide a confidence rating for sections of the document?
More broadly, it’s not trained to have any self awareness and this is a factor in other “hallucinations”. If you ask, for example, to describe the “marathon crater”, it doesn’t recognize that there’s no such thing in its corpus, but will instead start by writing an answer (“sure! The marathon crater is..”) and freestyle from there. Same if you ask it why it did something, or details about itself, etc. You should acce…
Yes, there’s research showing that models’ self-assessment of probabilities (when you ask them via prompting) don’t even match the same models’ actual probabilities, in cases where you can measure the probabilities directly (e.g. by looking at the logits): https://arxiv.org/abs/2305.13264
Re: Replace OCR with Vision Language Models
#119Earlier quoted context omitted.
Agree wholeheartedly. Modern OCR is astonishingly good, more importantly it's deterministically so. It's failure modes, when it's unable to read the text, are recognizably failures. Results for VLM accuracy & precision are not good. https://arxiv.org/html/2406.04470v1#S4
which solutions would you classify as "modern OCR" are we talking tesseract or something?
Probably something like Apple Vision Framework or Amazon Textract or Google's Cloud Vision.
Tesseract does well under ideal conditions, but the world is messy.
Re: Replace OCR with Vision Language Models
#120Earlier quoted context omitted.
which solutions would you classify as "modern OCR" are we talking tesseract or something?
Probably something like Apple Vision Framework or Amazon Textract or Google's Cloud Vision. Tesseract does well under ideal conditions, but the world is messy.
I was thinking ABBYY FineReader, but those, too.
Instead of using VLMs or any sort of generative AI,
they're build on good old-fashioned feature extraction and nearest neighbor classifiers such as the k-nearest neighbors algorithm.
It's possible to build a working prototype of this technique using basic ML algorithms.