41–50 of 178 posts
I'm curious if a multimodal model would be better at the OCR step than tesseract? Probably would increase the cost but I wonder if that would be offset by needing less post processing.
I have seen excellent performance with Florence-2 for OCR. I wrote https://blog.roboflow.com/florence-2-ocr/ that shows a few examples. Florence-2 is On a T4 in Colab, you can run inference in < 1s per image.
Have you tried using other OCR packages? I had to give up on Tesseract after every mode and model I tried read a quite plain image of "77" as "7" (and interestingly the javascript port reads it as "11"). Pic related: https://i.postimg.cc/W3QkkhCK/speed-roi-thresh.png
You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them. One of the benefits of this project is that it…
For our use case PaddleOCR + LLM has been quite nice combo.
I keep hoping someone at YouTube will do this for their autogenerated Closed Captioning. Nice work!
https://github.com/Dicklesworthstone/llm_aided_transcription...
the font was perfectly fine, the screenshots were crispy PNGs.
A LLM can't really correct that. I appreciate that Tesseract exists, and it's mostly fine for non-serious things, but I wouldn't let it anywhere near critical data.
Something that makes me nervous about this general approach is the risk of safety filters or accidental (or deliberate) instruction following interfering with the results. I want to be able to run OCR against things like police incident reports without worrying that a safety filter in the LLM will refuse to process the document because it takes exception to a description of violence or foul language. If a scanned doc…
https://huggingface.co/Orenguteng/Llama-3.1-8B-Lexi-Uncensor...
This model will literally tell you how to make meth at home, so I wouldn't be worried about it refusing to correct police report text! Only issue is that you can't do the massive concurrency then like you can for the hosted APIs, so it's much much slower. You could also theoretically use a service like OpenRouter that hosts the same model, but I was getting tons of rate limiting errors with it so I removed it from my project code.
As for prompt injection attacks where the document tells the LLM to do something bad... if the LLM doesn't have access to tools, what's the worst that could really happen? I think that can mostly be avoided anyway with good prompt engineering that clearly delineates what is "quoted text" and what is part of the instructions/annotations, especially since these newer models are much better about following instructions.
As for what can be done to mitigate these issues, I think realistically the only thing is to take the entire final work product and submit it to a bigger/better model that has a super long context window (although this will of course cost a lot more, but only requires a single inference call) and in that prompt, you ask it to look for any indications that there was interference from safety filtering or injection attacks, things that obviously don't fit into the flow of the writing, etc.
Have you tried using other OCR packages? I had to give up on Tesseract after every mode and model I tried read a quite plain image of "77" as "7" (and interestingly the javascript port reads it as "11"). Pic related: https://i.postimg.cc/W3QkkhCK/speed-roi-thresh.png
You know, I’ve really looked hard at what’s out there and haven’t been able to find anything else that’s totally free/open, that runs well on CPU, and which has better quality output than Tesseract. I found a couple Chinese projects but had trouble getting them to work and the documentation wasn’t great. If you have any leads on others to try I’d love to hear about them. One of the benefits of this project is that it…
I did more or less the same, trying to solve the same problem. I ended up biting the bullet and using Amazon Textract. The OCR is much better than Tesseract, and the layout tool is quite reliable to get linear text out of 2-columns documents (which is critical for my use case).
I would be very happy to find something as reliable that would work on a workstation without relying on anyone’s cloud.
when I was working with Tesseract, a particular issue I had was its tendency to parse a leading "+" as "4" about half the time. e.g. "+40% ROI" would get parsed as "440% ROI". the font was perfectly fine, the screenshots were crispy PNGs. A LLM can't really correct that. I appreciate that Tesseract exists, and it's mostly fine for non-serious things, but I wouldn't let it anywhere near critical data.
Fix this text extracted with OCR:
Return on Investment (RO1) is a crucial metric used to evaluate the efficiency and profitability of an investment. If you have achieved an ROI of 440%, it indicates a strong performance and successful investment strategy. To calculate ROI, you subtract the initial cost of the investment from the final value of the investment, then divide that difference by the initial cost, and multiply by 100 to get a percentage. For example, if you invested $10000 and the value grew to 214,000, your ROI would be calculated as follows:
(+attached image with the text)
GPT-4O:
Here's the corrected text based on the extracted OCR:
*Original OCR Extracted Text:*
> Return on Investment (RO1) is a crucial metric used to evaluate the efficiency and profitability of an investment. If you have achieved an ROI of 440%, it indicates a strong performance and successful investment strategy. > To calculate ROI, you subtract the initial cost of the investment from the final value of the investment, then divide that difference by the initial cost, and multiply by 100 to get a percentage. For example, if you invested $10000 and the value grew to 214,000, your ROI would be calculated as follows:
*Corrected Text:*
> Return on Investment (ROI) is a crucial metric used to evaluate the efficiency and profitability of an investment. If you have achieved an ROI of *+40%*, it indicates a strong performance and successful investment strategy. > To calculate ROI, you subtract the initial cost of the investment from the final value of the investment, then divide that difference by the initial cost, and multiply by 100 to get a percentage. For example, if you invested *$10,000* and the value grew to *$14,000*, your ROI would be calculated as follows:
Changes made:
- Corrected "RO1" to "ROI"
- Corrected "440%" to "+40%"
- Corrected "$10000" to "$10,000"
- Corrected "214,000" to "$14,000"
This assumes that input text actually is well formed, right? If I scan a page containing bogus text / typos, this will actually correct those mistakes in the output, right?
Earlier quoted context omitted.
macOS Live Text is incredible. Mac only though
Yes, I imagine it's using the same OCR model as the iPhone, which is really incredibly good. In fact, it's so good that I made a little app for fun just to be able to use it for OCRing whole PDF books: https://apps.apple.com/us/app/super-pdf-ocr/id6479674248
From your experience, how does the OCR engine work with multiple-columns documents?