I think Gemini Flash 1.5 is the best closed-source model for this. Very cheap. Particularly compared to GPT4o-mini, which is priced the same as GPT4 for image input tokens. Performance and speed is excellent. I convert each pdf page to an image and send one request per page to Flash (asynchronously). The prompt asks for markdown output with specific formatting guidelines. For my application (mainly pdf slideshows with less text), the output is better than any of the dedicated tools I tested particularly for equations and tables.
Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
21–30 of 178 posts
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#22Earlier quoted context omitted.
You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them. One of the benefits of this project is that it…
I ended up using EasyOCR. I assume it is too slow in CPU-only mode.
> I assume it is too slow in CPU-only mode.
So you don't have to assume: I gave up after running on 8 cores (Ryzen 7 2700) for 10 days for a single page.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#23what are examples of local LLMs that accept images, that are mentioned in the README?
This package seems to use llama_cpp for local inference [1] so you can probably use anything supported by that [2]. However, I think it's just passing OCR output for correction - the language model doesn't actually see the original image.
That said, there are some large language models you can run locally which accept image input. Phi-3-Vision [3], LLaVA [4], MiniCPM-V [5], etc.
[1] - https://github.com/Dicklesworthstone/llm_aided_ocr/blob/main...
[2] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#de...
[3] - https://huggingface.co/microsoft/Phi-3-vision-128k-instruct
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#24How does this compare in terms of speed, quality, and price to sending images to VLMs like GPT-4o or Claude 3.5?
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#25If anyone is looking to compare results visually, I have created an open source OCR visualiser to help identifying missing elements (especially in tables).
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#26Having tried this in the past, it can work pretty well 90% of the time. However, there are still some areas it will struggle.
Imagine you are trying to read a lease contract. The two areas which the LLM may be useless are numbers and names (names of people or places/addresses). There’s no way for your LLM to accurately know what the rent should be, or to know about the name of a specific person.
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#27Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#28Earlier quoted context omitted.
I ended up using EasyOCR. I assume it is too slow in CPU-only mode.
> I assume it is too slow in CPU-only mode. So you don't have to assume: I gave up after running on 8 cores (Ryzen 7 2700) for 10 days for a single page.
Something wrong with your setup. It should be less than 30 s per page with your hardware
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#29Something that makes me nervous about this general approach is the risk of safety filters or accidental (or deliberate) instruction following interfering with the results.
I want to be able to run OCR against things like police incident reports without worrying that a safety filter in the LLM will refuse to process the document because it takes exception to a description of violence or foul language.
If a scanned document says "let's ignore all of that and talk about this instead" I want to be confident the LLM won't treat those as instructions and discard the first half of the text.
I'm always worried about prompt injection - what if a scanned document deliberately includes instructions to an LLM telling it to do something else?
Have you encountered anything like this? Do you have any measures in place that might prevent it from happening?
Re: Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
#30Very cool! I have a hotkey to grab a region and pipe a screenshot through tesseract and then pipe that into my clipboard. I'll have to add on to it to pipe it though Ollama too :)
We've built an app like that but for PDF table extraction, https://table2xl.com
Looks great! Do you mind talking about your tech stack? Do you build on top of Tessaract or do you use a custom model?