Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

github.com

Post reply on HN
Print view View original

151–160 of 178 posts

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#151

Post by wahnfrieden »

fred123 wrote: post #137
wahnfrieden wrote: post #114

Earlier quoted context omitted.

Yes but it's relatively shit The Vision API can't even read vertical Japanese text

Fair enough. There are some new OCR APIs in the next macOS release. I wonder if the model has been improved.

They're just a new Swift-only interface to the same underlying behaviors, no apparent improvement. I was hoping for more given the visionOS launch but alas

What I'm trying now is combining ML Kit v2 with Live Text - Apple's for the accurate paragraphs of text, and then custom indexing that against the ML Kit v2 output to add bounding rects and guessing corrections for missing/misidentified parts from ML Kit (using it only for bounding rects and expecting it will make mistakes on the text recognition)

I also investigated private APIs for extracting rects from Live Text. It looks possible, the APIs are there (it has methods or properties which give bounding rects as is obviously required for Live Text functionality), but I can't wrap my head around accessing them yet.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#152

Post by itsadok »

In your assess_output_quality function, you ask the LLM to give a score first, then an explanation. I haven't been following the latest research on LLMs, but I thought you usually want the explanation first, to get the model to "think out loud" before committing to the final answer. Otherwise, it might commit semi-radndomly to some score, and proceed to write whatever explanation it can come up with to justify that score.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#154

Post by constantinum »

kergonath wrote: post #47
eigenvalue wrote: post #5

Earlier quoted context omitted.

You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them. One of the benefits of this project is that it…

> You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them. I did more or less the same, trying to solve…

Was this by any chance Paddle OCR https://github.com/PaddlePaddle/PaddleOCR

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#155

Post by Sidneys1 »

mewpmewp2 wrote: post #148
Sidneys1 wrote: post #100

Earlier quoted context omitted.

I'd like to hear more about this! I keep coming back to trying to OCR my journals, but nothing I've tried so far works well (enough) on handwriting.

Have you tried uploading image of your handwriting to ChatGPT interface with ChatGPT 4o? And what the results were? And if not could you try and let us know what the results are.

Not with 4o, but I tried it with 4 (through Copilot) a while ago and the results were abysmal, even with very neatly printed handwriting.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#156

Post by wahnfrieden »

jszymborski wrote: post #142
troysk wrote: post #140

Earlier quoted context omitted.

How to segment the document without LLM? I prefer to do all of this in 1 step with an LLM with a good prompt and few shots. With so many passes with images, the costs/time will be high with ViT being slower.

Segmenting can likely be done on a really small resolution and with a CNN, making it real short. There are some heuristic ways of doing it but i doubt you'll be able to distinguish equations from text.

Apple APIs such as Live Text, subject identification, Vision. Run them on a server, too

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#157

Post by kevingadd »

ozim wrote: post #111
janalsncm wrote: post #26

Having tried this in the past, it can work pretty well 90% of the time. However, there are still some areas it will struggle. Imagine you are trying to read a lease contract. The two areas which the LLM may be useless are numbers and names (names of people or places/addresses). There’s no way for your LLM to accurately know what the rent should be, or to know about the name of a specific person.

Why does it have to be 100% accurate? If you get 90% of work done and you have to fix some numbers and names it still saves you time, isn't it?

Let's say you're OCRing a contract. Odds are good that almost every part of the contract is there for an important reason, though it may not matter to you. How many errors can you tolerate in the terms of a contract that governs i.e. your home, or the car you drive to work, or your health insurance coverage? Do you want to take a gamble on those terms that could - in the worst case - result in getting kicked out of your apartment or having to pay a massive medical bill yourself?

The important question is which parts are inaccurate. If it's messing up names and numbers but is 99.9% accurate for everything else, you can just go back and check all the names and numbers at the end. But if the whole thing is only 90% accurate, you now either recheck the whole document or you risk a 'must' turning into a 'may' in a critical place that undermines the whole document.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#158

Post by _boffin_ »

fred123 wrote: post #106
wahnfrieden wrote: post #99

Earlier quoted context omitted.

Sadly no bounding rects

You can get them through the Vision API (Swift/Objective-C/AppleScript)

You’re forgetting about Python and TypeScript/JavaScript. PyObjC and whatever it is for TypeScript.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#159

Post by fred123 »

wahnfrieden wrote: post #151
fred123 wrote: post #137

Earlier quoted context omitted.

Fair enough. There are some new OCR APIs in the next macOS release. I wonder if the model has been improved.

They're just a new Swift-only interface to the same underlying behaviors, no apparent improvement. I was hoping for more given the visionOS launch but alas What I'm trying now is combining ML Kit v2 with Live Text - Apple's for the accurate paragraphs of text, and then custom indexing that against the ML Kit v2 output to add bounding rects and guessing corrections for missing/misidentified parts from ML Kit (using it…

I feel like text detection is much better covered by the various ML models discussed elsewhere in the comments. Maybe you can combine those with Live Text. I found Tesseract pretty ok for text detection as well but I don’t know if any of the models are good for vertical text.

Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

#160

Post by scw »

Exciting concept! Note that the LLM corrected version does drop a full paragraph from the output at the bottom of the second page (starting with an asterisk and "My views regarding inflationary possibilities". I'm not sure if there is a simple way to mitigate this risk but would be nice to fall back on uncorrected text if the LLM can't produce valid results for some region of the document.
Post reply on HN