Mistral OCR

Intriguing announcement, however the examples on the mistral.ai page seem rather "easy".

What about rare glyphs in different languages using handwriting from previous centuries?

I've been dealing with OCR issues and evaluating different approaches for past 5+ years at a national library that I work at.

Usual consensus is that widely used open source Tesseract is subpar to commercial models.

That might be so without fine tuning. However one can perform supplemental training and build your own Tesseract models that can outperform the base ones.

Case study of Kant's letter's from 18th century:

About 6 months ago, I tested OpenAi approach to OCR to some old 18th century letters that needed digitizing.

The results were rather good (90+% accuracy) with the usual hallucination here and there.

What was funny that OpenAI was using base Tesseract to generate the segmenting and initial OCR.

The actual OCRed content before last inference step was rather horrid because the Tesseract model that OpenAi was using was not appropriate for the particular image.

When I took OpenAi off the first step and moved to my own Tesseract models, I gained significantly in "raw" OCR accuracy at character level.

Then I performed normal LLM inference at the last step.

What was a bit shocking: My actual gains for the task (humanly readable text for general use) were not particularly significant.

That is LLMs are fantastic at "untangling" complete mess of tokens into something humanly readable.

For example:

P!3goattie -> prerogative (that is given the surrounding text is similarly garbled)

Isn' it just statistical word pattern prediction based on training data? These models likely don't "know" something anyway and cannot verify "truth" and facts. Reasoning attempts seem to me basically just like looping until the model finds a self-satisfying equilibrium state with different output.

In that way, LLMs are more human than, say, a database or a book containing agreed-upon factual information which can be directly queried on demand.

Imagine if there was just ONE human with human limitations on the entire planet who was taught everything for a long time - how reliable do you think they are with information retrieval? Even highly trained individuals (e.g. professors) can get stuff wrong on their specific topics at times. But this is not what we expect and demand from computers.

... unavoidable "human in the loop" - depends imo.

From the comments here, it certainly seems that for general OCR it's not up to snuff yet. Luckily, I don't have great ambitions.

I can see this working for me with just a little care upfront preprocessing now that I know where it falls over. It casually skips portions of the document, and misses certain lines consistently. Knowing that I can do a bit massaging, and feed it what I know it likes, and then reassemble.

I found in testing that it failed consistently at certain parts, but where it worked, it worked extremely well in contrast to other methods/services that I've been using.

It did better than any other solution out there. However, I can only validate by the logic of the text. It's a recipe book.

Forgive my absolute ignorance, I should probably run this through a chat bot before posting ... So I'm updating my post with answers now!

Q: Do LLMs specialise in "document level" recognition based on headings, paragraphs, columns tables etc? Ie: ignore words and characters for now and attempt to recognise a known document format.

A: Not most LLMs, but those with multimodal / vision capability could (eg DeepSeek Vision. ChatGPT 4). There are specialized models for this work like Tesseract, LayoutLM.

Q: How did OCR work "back in the day" before we had these LLMs? Are any of these methods useful now?

A: They used pattern recognition and feature extraction, rules and templates. Newer ML based OCR used SVM to isolate individual characters and HMM to predict the next character or word. Today's multimodal models process images and words, can handle context better than the older methods, and can recognise whole words or phrases instead of having to read each character perfectly. This is why they can produce better results but with hallucinations.

Q: Can LLMs rate their own confidence in each section, maybe outputting text with annotations that say "only 10% certain of this word", and pass the surrounding block through more filters, different LLMs, different methods to try to improve that confidence?

A: Short answer, "no". But you can try to estimate with post processing.

Or am I super naive, and all of those methods are already used by the big commercial OCR services like Textract etc?