Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs

An OCR will always mix up characters so I don’t really see the issue here?

I don't want to jump to conclusions, but I don't feel confident using gpt4o/claude for OCR, as I often experience issues mentioned on this page https://github.com/Yuliang-Liu/MultimodalOCR

[edit] But it is not applicable to OCR specialised models like Florence-2

Nope. Most compression does not mix up characters the way JBIG2 does (see the article), and most OCR does not substitute plausible text in for text it fails to scan.

Let's say the text is "The laptop costs $1,000 (one thousand dollars)." but the image is blurry.

Normal compression will give you an image where "$1,000" is blurry. JBIG2 can give you an image where "$1,000" has been replaced by a perfectly-clear "$7,000."

Normal OCR will give you some nonsense like "The laptop costs $7,000 (one 1housand dollars)". The LLM can "fix this up" to something more plausible like "The laptop costs $2,000 (two thousand dollars)."

LLaVA is one LLM that takes both text and images as inputs - https://llava-vl.github.io/

Although LLaVA specifically it might not be great for OCR; IIRC it scales all input images to 336 x 336 - meaning it'll only spot details that are visible at that scale.

You can also search on HuggingFace for the tag "image-text-to-text" https://huggingface.co/models?pipeline_tag=image-text-to-tex... and find a variety of other models.

Fantastic work is emerging in this field, and with the new release of the schnell model of the flux series we will have the downstream captioning datasets we need to produce a new SOTA vision model, which has been the last straggler in the various open llm augmentations. Most vision models are still based on ancient CLIP/BLIP captioning and even with something like LLAVA or the remarkable phi-llava, we are still held back by the pretained vision components which have been needing love for some months now.

Tessy and LLM is a good pipe, it's likely what produced SCHNELL and will soon be the reverse of this configuration, used for testing and checking while the LLM does the bulk of transcription via vision modality adaption. The fun part of that is that multi lingual models will be able to read and translate, opening up new work for scholars searching through digitized works. Already I have had success in this area with no development at all, after we get our next SOTA vision models I am expecting a massive jump in quality. I expect english vision model adapters to show up using LLAVA architecture first, this may put some other latin script languages into the readable category depending on the adapted model, but we could see a leapfrog of scripts becoming readable all at once. LLAVA-PHI3 already seems to be able to transcribe tiny pieces of hebrew with relative consistency. It also has horrible hallucinations, so there is very much an unknown limiting factor here currently. I was planning some segmentation experiments but schnell knocked that out of my hands like a bar of soap in a prison shower, I will be waiting for a distilled captioning sota to come before I re-evaluate this area.

Exciting times!