Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

github.com

Post reply on HN
Print view View original

81–90 of 178 posts

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#81

Post by barrenko »

zerojames wrote: post #9
x-yl wrote: post #6

I'm curious if a multimodal model would be better at the OCR step than tesseract? Probably would increase the cost but I wonder if that would be offset by needing less post processing.

I have seen excellent performance with Florence-2 for OCR. I wrote https://blog.roboflow.com/florence-2-ocr/ that shows a few examples. Florence-2 is On a T4 in Colab, you can run inference in < 1s per image.

This is pretty cool, when checking how Microsoft models (then) stacked against Donut, I chose Donut, didn't know they published more models!

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#82

Post by aliosm »

bugglebeetle wrote: post #79
aliosm wrote: post #65

I'm working on Arabic OCR for a massive collection of books and pages (over 13 million pages so far). I've tried multiple open-source models and projects, including Tesseract, Surya, and a Nougat small model fine-tuned for Arabic. However, none of them matched the latency and accuracy of Google OCR. As a result, I developed a Python package called tahweel ( https://github.com/ieasybooks/tahweel ), which leverages Goo…

This has been my experience with Japanese texts as well. I have a number of fairly obscure Japanese books and magazines I’ve collected as part of a research interest. During the pandemic, I began digitizing them and found that nothing but Google OCR could extract the text correctly. I recently tried again with the libraries you mentioned, but they also performed worse than traditional tools.

Good to know :3

I'm currently planning to develop a tool to correct Arabic outputs for ASR and OCR. It will function like spell-correction but with a focus specifically on these two areas. Perhaps you could start something similar for Japanese? English (and Latin languages in general) perform at a different level across multiple tasks, to be honest...

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#83

Post by eigenvalue »

esafak wrote: post #77
eigenvalue wrote: post #73

Earlier quoted context omitted.

Not sure how you would do that without having the ground truth to compare to. It's also very hard to measure once you start messing with the formatting (like converting it to markdown or suppressing page numbers and repeated headers/footers). I think it would also vary a lot depending on the quality of the original scan and the format and content of the document. There's really no substitute from just trying it on yo…

You can correct the transcript to create the ground truth. Or print your own document, then run OCR on it. OCR evaluation has been a thing for decades. edit: Better than a single document, process a standard OCR dataset: https://paperswithcode.com/task/optical-character-recognitio...

Oh you meant for just a single benchmarked document. I thought you meant to report that for every document you process. I wouldn't want to mislead people by giving stats on a particular kind of scan/document, because it likely wouldn't carry over in general.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#84

Post by fred123 »

aliosm wrote: post #65

I'm working on Arabic OCR for a massive collection of books and pages (over 13 million pages so far). I've tried multiple open-source models and projects, including Tesseract, Surya, and a Nougat small model fine-tuned for Arabic. However, none of them matched the latency and accuracy of Google OCR. As a result, I developed a Python package called tahweel ( https://github.com/ieasybooks/tahweel ), which leverages Goo…

Azure Vison OCR is supposed to be the best commercial OCR model right now and it’s really cheap (same price as Google‘s)

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#85

Post by fred123 »

vikp wrote: post #75
aliosm wrote: post #65

I'm working on Arabic OCR for a massive collection of books and pages (over 13 million pages so far). I've tried multiple open-source models and projects, including Tesseract, Surya, and a Nougat small model fine-tuned for Arabic. However, none of them matched the latency and accuracy of Google OCR. As a result, I developed a Python package called tahweel ( https://github.com/ieasybooks/tahweel ), which leverages Goo…

Hi, I'm the author of surya ( https://github.com/VikParuchuri/surya ) - working on improving speed and accuracy now. Happy to collaborate if you have specific page types it's not working on. For modern/clean documents it benchmarks very similarly to Google Cloud, but working on supporting older documents better now.

It seems to struggle with German text a lot (umlauts etc)

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#86

Post by adrianmonk »

foota wrote: post #32

I wonder if you could feed back the results from an LLM into the OCR model to get it to make better decisions. E.g., if it's distinguishing a 1 from an I, the LLM could provide a probability distribution.

Or the other direction. Tesseract can give you confidence levels for the guesses it makes about a symbol (see https://tesseract-ocr.github.io/tessdoc/APIExample.html).

You can give the LLM this information and have it tell you which is right. Here's a demo.

=== input to ChatGPT 4o ===

I'm trying to read some text but one of the characters is unclear. I will use the notation [xy] to indicate that I think the character is either x or y. Please tell me which one you think is correct. Here is the text: "There was a lot of traffic on [I1]-80 this morning." In your response, please include a line like "correct character: x" so I can see the conclusion at a glance.

=== ChatGPT's response ===

The correct character in the text is likely "I".

Interstate highways in the United States are often denoted by "I-" followed by the highway number. "I-80" is a major interstate highway that runs across the United States, so it makes more sense in this context than "1-80," which would imply a different and less common road designation.

correct character: I

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#87

Post by myth_drannon »

zerojames wrote: post #9
x-yl wrote: post #6

I'm curious if a multimodal model would be better at the OCR step than tesseract? Probably would increase the cost but I wonder if that would be offset by needing less post processing.

I have seen excellent performance with Florence-2 for OCR. I wrote https://blog.roboflow.com/florence-2-ocr/ that shows a few examples. Florence-2 is On a T4 in Colab, you can run inference in < 1s per image.

I couldn't find any comparisons with Microsoft's TrOCR model. I guess they are for different purposes. But since you used Florence-2 for OCR, did you compare the two?

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#88

Post by KTibow »

kelsey98765431 wrote: post #19

Fantastic work is emerging in this field, and with the new release of the schnell model of the flux series we will have the downstream captioning datasets we need to produce a new SOTA vision model, which has been the last straggler in the various open llm augmentations. Most vision models are still based on ancient CLIP/BLIP captioning and even with something like LLAVA or the remarkable phi-llava, we are still held…

Is LLaVA-Phi better than Phi Vision?

edit: I think parent just doesn't know about Phi Vision, it appears to be a better model

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#90

Post by aliosm »

fred123 wrote: post #84
aliosm wrote: post #65

I'm working on Arabic OCR for a massive collection of books and pages (over 13 million pages so far). I've tried multiple open-source models and projects, including Tesseract, Surya, and a Nougat small model fine-tuned for Arabic. However, none of them matched the latency and accuracy of Google OCR. As a result, I developed a Python package called tahweel ( https://github.com/ieasybooks/tahweel ), which leverages Goo…

Azure Vison OCR is supposed to be the best commercial OCR model right now and it’s really cheap (same price as Google‘s)

Note that the tool is uploading/downloading to/from Google Drive through GCP Service Account credentials to perform OCR for free.
Post reply on HN