Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace OCR with Vision Language Models

github.com

Post reply on HN
Print view View original

121–130 of 130 posts

Re: Replace OCR with Vision Language Models

#121

Post by the8472 »

rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

Shouldn't confidence be available at the sampler level and also be conditional on the vision input, not just the next-token prediction?

Re: Replace OCR with Vision Language Models

#122

Post by jjk166 »

rafram wrote: post #71
hashta wrote: post #47

Earlier quoted context omitted.

An effective way that usually increases accuracy is to use an ensemble of capable models that are trained independently (e.g., gemini, gpt-4o, qwen). If >x% of them have the same output, accept it, otherwise reject and manually review

There’s a very low chance that three separate models will come up with the same result. There are always going to be errors, small or large. Even if you find a way around that, running the process three times on every page is going to be prohibitively expensive, especially if you want to finetune.

The odds of them getting the same result for any given patch should be very high if it is the correct result and they aren't garbage. The only times where they are not getting the same result would be the times when at least one has made a mistake. The odds of 3 different models making the same mistake should be low (unless it's something genuinely ambiguous like 0 vs O in a random alphanumeric string).

Best 2 out of 3 should be far more reliable than any model on its own. You could even weight their responses for different types of results, like say model B is consistently better for serif fonts, maybe their confidence counts for 1.5 times as much as the confidence of models A and C.

Re: Replace OCR with Vision Language Models

#124

Post by sudoshred »

themanmaran wrote: post #36
rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

Thing is, the majority of OCR errors aren't character issues, but layout issues. Things like complex tables with cells being returned under the wrong header. And if the numbers in an income statement are one column off creates a pretty big risk. Confidence intervals are a red herring. And only as good as the code interpreting them. If the OCR model gives you back 500 words all ranging from 0.70 to 0.95 confidence, wh…

That’s assuming that confidence intervals are even independently comparable. Anecdotally major OCR services with specific languages have average confidence intervals that are wildly divergent from similar services with different languages for the same relative quality of result. Acting as if confidence interval is in any way absolute or otherwise able to reliably and consistently indicate the relative quality of results is a mischaracterization at best. In the worst case CI is as good as an RNG. The value of the CI is in the ability to tune usage of the results based on observations of the users and characteristics of the request, sometimes it is meaningful but not always. In this case “good” code essentially hardcodes handling for all the idiosyncrasies of the common usage and the OCR service.

Re: Replace OCR with Vision Language Models

#125

Post by anon291 »

_delirium wrote: post #118
thatjoeoverthr wrote: post #95

Earlier quoted context omitted.

More broadly, it’s not trained to have any self awareness and this is a factor in other “hallucinations”. If you ask, for example, to describe the “marathon crater”, it doesn’t recognize that there’s no such thing in its corpus, but will instead start by writing an answer (“sure! The marathon crater is..”) and freestyle from there. Same if you ask it why it did something, or details about itself, etc. You should acce…

Yes, there’s research showing that models’ self-assessment of probabilities (when you ask them via prompting) don’t even match the same models’ actual probabilities, in cases where you can measure the probabilities directly (e.g. by looking at the logits): https://arxiv.org/abs/2305.13264

Logits are not probabilities... at least not in the way you understand probability. Probabilities mathematically are anything that broadly behaves like a probability, whereas colloquially probabilities represent the likelihood or the preponderance of a particular phenomenon. Logits are not either of those.

Re: Replace OCR with Vision Language Models

#126

Post by _delirium »

anon291 wrote: post #125
_delirium wrote: post #118

Earlier quoted context omitted.

Yes, there’s research showing that models’ self-assessment of probabilities (when you ask them via prompting) don’t even match the same models’ actual probabilities, in cases where you can measure the probabilities directly (e.g. by looking at the logits): https://arxiv.org/abs/2305.13264

Logits are not probabilities... at least not in the way you understand probability. Probabilities mathematically are anything that broadly behaves like a probability, whereas colloquially probabilities represent the likelihood or the preponderance of a particular phenomenon. Logits are not either of those.

The probability of token generation is a function of the logits. Do you have an actual point related to the linked paper?

Re: Replace OCR with Vision Language Models

#127

Post by wantlotsofcurry »

I'll definitely be trying this out on my current side project!

Question: What tools/libs are people using to accurately detect square/rectangle objects in images?

I've used VNDetectRectangle [1] in Swift but it's not as accurate as I'd like it to be, even with preprocessing.

[1]: https://developer.apple.com/documentation/vision/vndetectrec...

Re: Replace OCR with Vision Language Models

#128

Post by anon291 »

_delirium wrote: post #126
anon291 wrote: post #125

Earlier quoted context omitted.

Logits are not probabilities... at least not in the way you understand probability. Probabilities mathematically are anything that broadly behaves like a probability, whereas colloquially probabilities represent the likelihood or the preponderance of a particular phenomenon. Logits are not either of those.

The probability of token generation is a function of the logits. Do you have an actual point related to the linked paper?

That is one way of sampling tokens. It is not the only way. Logits do not map neatly to belief, although it is convenient to behave as if they do

Re: Replace OCR with Vision Language Models

#129

Post by temp0826 »

wrs wrote: post #53
temp0826 wrote: post #52

I've been looking for a solution to translate a dictionary for me. It is a Shipibo-Conibo (indigenous Peruvian language) to Spanish dictionary- I'd like to translate the Spanish to English (and leave the Shipibo intact). Curious for any thoughts here. I have the dictionary as a PDF (already searchable so I don't think it would need to be re-OCR'd...though that's possible too, it's not clearest scan).

I wouldn’t be surprised to find that Claude/ChatGPT/etc. can just… do that. With the prompt you just gave. The output could be in Markdown, which is easily turned into a PDF. You would have to break up the input PDF into pages to avoid running out of output window.

I didn't consider that would actually work and am giving it a try now...but by its own estimate it's going to take several days to finish (I'm not paying for plus or whatever).

Re: Replace OCR with Vision Language Models

#130

Post by temp0826 »

zzleeper wrote: post #83
temp0826 wrote: post #52

I've been looking for a solution to translate a dictionary for me. It is a Shipibo-Conibo (indigenous Peruvian language) to Spanish dictionary- I'd like to translate the Spanish to English (and leave the Shipibo intact). Curious for any thoughts here. I have the dictionary as a PDF (already searchable so I don't think it would need to be re-OCR'd...though that's possible too, it's not clearest scan).

By any chance, would it be possible to share the PDF? I haven't heard shipibo language in a long while, and am quite curious about it.

Here you go-

https://archive.org/details/shipibodiccionario

Post reply on HN