Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace OCR with Vision Language Models

github.com

Post reply on HN
Print view View original

61–70 of 130 posts

Re: Replace OCR with Vision Language Models

#61

Post by cdolan »

acdha wrote: post #57
iLemming wrote: post #56

What's the fastest and accurate CLI OCR tool? My use case is simple - I want to be able to grab a piece of screen (Flameshot is great for that), and OCR it. I need this for note-taking during pair-programming over Zoom. Currently I'm using tesseract - it works, it's fast, but it also makes mistakes; it would be also great if it could discern tabular data and put them in ascii or markdown tables. I've tried docling, b…

Anything using the Apple Vision framework is fast and surprisingly accurate: https://github.com/bytefer/macos-vision-ocr

Cool to see, may use this locally for OCR in some cases. But I think the "handwriting" example is a little misleading. Thats a font, not a scan of hand written material

Re: Replace OCR with Vision Language Models

#62

Post by ANighRaisin »

iLemming wrote: post #56

What's the fastest and accurate CLI OCR tool? My use case is simple - I want to be able to grab a piece of screen (Flameshot is great for that), and OCR it. I need this for note-taking during pair-programming over Zoom. Currently I'm using tesseract - it works, it's fast, but it also makes mistakes; it would be also great if it could discern tabular data and put them in ascii or markdown tables. I've tried docling, b…

The AI OCR build into snipping tool in windows is better than tesseract, albeit more inconvenient than something like powertoys or Capture2Text, which use a quick shortcut.

Re: Replace OCR with Vision Language Models

#63

Post by Hackbraten »

constantinum wrote: post #59
rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

The primary issue with LLMs is hallucination, which can lead to incorrect data and flawed business decisions. For example, Llamaparse( https://docs.llamaindex.ai/en/stable/llama_cloud/llama_parse... ) uses LLMs for PDF text extraction but faces hallucination problems. See this issue for more details: https://github.com/run-llama/llama_parse/issues/420 . For those interested, try LLMWhisperer( https://unstract.com/llm…

> try LLMWhisperer(https://unstract.com/llmwhisperer/) for OCR. It avoids LLMs

The website you linked says it uses LLMs?

Re: Replace OCR with Vision Language Models

#64

Post by refulgentis »

EarlyOom wrote: post #34
rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.

That's not OCR.

It is an absolute miracle.

It is transmutating a picture into JSON.

I never thought this would be possible in my lifetime.

But that is different from what your interlocutor is discussing.

Re: Replace OCR with Vision Language Models

#65

Post by ritvikpandey21 »

hey -- wrote a blog post about this exact phenomena [1] (also posted on HN couple weeks back [2]). tldr: maintaining confidence from LLM nondeterministic outputs over millions of pages is a problem. especially in production environments like healthcare, finance, etc. we've noticed decently high hallucination rates, even in finetuned LLMs.

[1]: https://www.runpulse.com/blog/why-llms-suck-at-ocr [2]: https://news.ycombinator.com/item?id=42966958#42977527

Re: Replace OCR with Vision Language Models

#66

Post by 1024core »

refulgentis wrote: post #64
EarlyOom wrote: post #34

Earlier quoted context omitted.

This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.

That's not OCR. It is an absolute miracle. It is transmutating a picture into JSON. I never thought this would be possible in my lifetime. But that is different from what your interlocutor is discussing.

> I never thought this would be possible in my lifetime.

I used to work in Computer Vision and Image Processing. These days I utter this sentence on an almost daily basis. :-D

Re: Replace OCR with Vision Language Models

#67

Post by NoMoreNicksLeft »

BrannonKing wrote: post #38

What I want: take scan/photo of a document (including a full book), pass it to the language model, and then get out a Latex document that matches the original document exactly (minus the copier/camera glitches and angles). I feel like some kind of reinforcement learning model would be possible for this. It should be able to learn to generate Latex that reproduces the exact image, pixel for pixel (learning which pixel…

A big difficulty there is typeface detection, some of these were never digital fonts. But, even if it could detect them, you likely don't have those fonts on your computer to be able to put it back together as a digital typesetting for any but the most trivial fonts.

Re: Replace OCR with Vision Language Models

#68

Post by anon373839 »

themanmaran wrote: post #36
rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

Thing is, the majority of OCR errors aren't character issues, but layout issues. Things like complex tables with cells being returned under the wrong header. And if the numbers in an income statement are one column off creates a pretty big risk. Confidence intervals are a red herring. And only as good as the code interpreting them. If the OCR model gives you back 500 words all ranging from 0.70 to 0.95 confidence, wh…

> But if you're not rejecting based on CI, then you're exposed to just as much risk as using an LLM.

That's not true. LLMs and OCR have very different failure modes. With LLMs, there is unbounded potential for hallucination, and the entire document is at risk. For example: if something in the lower right-hand corner of the page takes the model to a sparsely sampled part of the latent space, it can end up deciding that it makes sense to rewrite the document title! Or anything else. LLMs also have a pernicious habit of "helpfully" completing partial sentences that appear at the beginning or end of a page of text.

With OCR, errors are localized and have a greater chance of being detected when read.

I think for a lot of cases, the best solution is to fine-tune a model like LayoutLM, which can classify the actual text tokens in a document (whether obtained from OCR or a native text layer) using visual and spatial information. Then, there are no hallucinations and you can use uncertainty information from both the OCR (if used) and the text classification. But it does mean that you have to do the work of annotating data and training a model, rather than prompt engineering...

Re: Replace OCR with Vision Language Models

#69

Post by rafram »

EarlyOom wrote: post #34
rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

This is the main focus of VLM Run and typed extraction more generally. If you provide proper type constraints (e.g. with Pydantic) you can dramatically reduce the surface area for hallucination. Then there's actually fine-tuning on your dataset (we're working on this) to push accuracy beyond what you get from an unspecialized frontier model.

Re type constraints: Not really. If one of the fields in my JSON is `name` but the model can’t read the name on the page, it will very happily make one up. Type constraints are good for making sure that your data is parseable, but they don’t do anything to fix the undetectable inaccuracy problem.

Fine-tuning does help, though.

Re: Replace OCR with Vision Language Models

#70

Post by rafram »

themanmaran wrote: post #36
rafram wrote: post #32

It’s an interesting idea, but still way too unreliable to use in production IMO. When a traditional OCR model can’t read the text, it’ll output gibberish with low confidence; when a VLM can’t read the text, it’ll output something confidently made up, and it has no way to report confidence. (You can ask it to, but the number will itself be made up.) I tried using a VLM to recognize handwritten text in genealogical sou…

Thing is, the majority of OCR errors aren't character issues, but layout issues. Things like complex tables with cells being returned under the wrong header. And if the numbers in an income statement are one column off creates a pretty big risk. Confidence intervals are a red herring. And only as good as the code interpreting them. If the OCR model gives you back 500 words all ranging from 0.70 to 0.95 confidence, wh…

Then use an LLM to extract layout information. Don’t trust it to read the text.

> If the OCR model gives you back 500 words all ranging from 0.70 to 0.95 confidence, what do you do? Reject the entire document if there's a single value below 0.90?

No, of course not. You have a human review the words/segments with low confidence.

Post reply on HN