Benchmarks look good. I tried this with a PDF that already has accurate PDF embedded just with new lines making pdftotext fail, and it was accurate for the text it found, but missed entire pages
Mistral OCR
391–400 of 450 posts
Re: Mistral OCR
#392Earlier quoted context omitted.
The tagline boasts that it is "introducing the world’s best document understanding API". So, holding them to their marketing seems fair
Isn't anyone who releases anything putting "the world's best blablabla" on their page nowadays? I've become entirely blind to it.
If they put it, and it's subpar, I write off the product.
Re: Mistral OCR
#393Earlier quoted context omitted.
> with LLM as a judge For anyone else interested, prompt is here [0]. The model used was gemini-2.0-flash-001. Benchmarks are hard, and I understand the appeal of having something that seems vaguely deterministic rather than having a human in the loop, but I have a very hard time accepting any LLM-judged benchmarks at face value. This is doubly true when we're talking about something like OCR which, as you say, is a…
We also ran an OCR benchmark with LLM as judge using structured outputs. You can check out the full methodology on the repo [1]. But the general idea is: - Every document has ground truth text, a JSON schema, and the ground truth JSON. - Run OCR on each document and pass the result to GPT-4o along with the JSON Schema - Compare the predicted JSON against the ground truth JSON for accuracy. In our benchmark, the groun…
Wouldn't that just bias itself to the shape of the text extracted from the OCR against the shape of the raw text alone? It doesn't seem like it would be a great benchmark for estimating semantic accuracy?
Re: Mistral OCR
#394[deleted]
Re: Mistral OCR
#395[dead]
Re: Mistral OCR
#396[deleted]
Re: Mistral OCR
#397[deleted]
Re: Mistral OCR
#398[dead]
Re: Mistral OCR
#399I had a need to scan serial numbers from Apple's product boxes out of pictures taken by a random person on their phone.
All OCR tools that I have tried have failed. Granted, I would get much better results if I used OpenCV to detect the label, rotate/correct it, normalize contrast, etc.
But... I have tried the then new vision model from OpenAI and it did the trick so well it's wasn't feasible to consider anything else at that point.
I have checked all S/N afterwards for being correct via third-party API - and all of theme were. Sure, sometimes I had to check versions with 0/o and i/l/1 substitutions but I believe these kind of mistakes are non-issues.
Re: Mistral OCR
#400I would like to see how it performs with massively warped and skewed scanned text images, basically a scanned image where the text lines are wavy as opposed as straight horizontal, where the letters are elongated. One where the line widths are different depending on the position on the scanned image. I once had to deal with such a task that somebody gave me with OCR software, Acrobat, and other tools could not decode…
I use gemini to solve textual CAPTCHAS with those kind of distortions and more: 60% of the time it works every time.