Cool stuff! I noticed that it threw away the footnote beginning with "My views regarding inflationary possibilities" in the example text, though.
Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
31–40 of 178 posts
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#32I wonder if you could feed back the results from an LLM into the OCR model to get it to make better decisions. E.g., if it's distinguishing a 1 from an I, the LLM could provide a probability distribution.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#33I'm curious if a multimodal model would be better at the OCR step than tesseract? Probably would increase the cost but I wonder if that would be offset by needing less post processing.
My experience is that at least the models which are price-competitive (~= open weight and small enough to run on a 3/4090 - MiniCPM-V, Phi-3-V, Kosmos-2.5) are not as good as Tesseract or EasyOCR. They're often more accurate on plain text where their language knowledge is useful but on symbols, numbers, and weird formatting they're at best even. Sometimes they go completely off the rails when they see a dashed line or handwriting or an image, things which the conventional OCR tools can ignore or at least recover from.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#34Have you tried using other OCR packages? I had to give up on Tesseract after every mode and model I tried read a quite plain image of "77" as "7" (and interestingly the javascript port reads it as "11"). Pic related: https://i.postimg.cc/W3QkkhCK/speed-roi-thresh.png
Most issues related to Tesseract will have to do with input DPI, often you need to crank that setting way up from its default.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#35I keep hoping someone at YouTube will do this for their autogenerated Closed Captioning. Nice work!
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#36This assumes that input text actually is well formed, right? If I scan a page containing bogus text / typos, this will actually correct those mistakes in the output, right?
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#37Having tried this in the past, it can work pretty well 90% of the time. However, there are still some areas it will struggle. Imagine you are trying to read a lease contract. The two areas which the LLM may be useless are numbers and names (names of people or places/addresses). There’s no way for your LLM to accurately know what the rent should be, or to know about the name of a specific person.
Agreed, this should not be used for anything mission critical unless you're going to sit there and carefully review the output by hand (although that is still going to be 100x faster than trying to manually correct the raw OCR output).
Where it's most useful to me personally is when I want to read some old book from the 1800s about the history of the Royal Navy [0] or something like that which is going to look really bad on my Kindle Oasis as a PDF, and the OCR version available from Archive.org is totally unreadable because there are 50 typos on each page. The ability to get a nice Markdown file that I can turn into an epub and read natively is really nice, and now cheap and fast.
[0] https://archive.org/details/royalnavyhistory02clowuoft/page/...
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#38I think Gemini Flash 1.5 is the best closed-source model for this. Very cheap. Particularly compared to GPT4o-mini, which is priced the same as GPT4 for image input tokens. Performance and speed is excellent. I convert each pdf page to an image and send one request per page to Flash (asynchronously). The prompt asks for markdown output with specific formatting guidelines. For my application (mainly pdf slideshows wit…
> I convert each pdf page to an image and send one request per page to Flash
Why convert? Flash 1.5 accepts whole PDFs just fine. It will also increase the models response accuracy.
Context: I have found Flash 1.5 is excellent and stable for this kind of use-case. Even at a non-EA price-point it's incredibly cheap, especially when utilizing Batch Prediction Jobs (50% discount!).
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#39Have you tried using other OCR packages? I had to give up on Tesseract after every mode and model I tried read a quite plain image of "77" as "7" (and interestingly the javascript port reads it as "11"). Pic related: https://i.postimg.cc/W3QkkhCK/speed-roi-thresh.png
Most issues related to Tesseract will have to do with input DPI, often you need to crank that setting way up from its default.
IIRC Tesseract is trained on 300 DPI
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#40Earlier quoted context omitted.
You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them. One of the benefits of this project is that it…
macOS Live Text is incredible. Mac only though
Yes, I imagine it's using the same OCR model as the iPhone, which is really incredibly good. In fact, it's so good that I made a little app for fun just to be able to use it for OCRing whole PDF books: