Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral OCR

mistral.ai

Post reply on HN
Print view View original

351–360 of 450 posts

Re: Mistral OCR

#351

Post by merb »

kiratp wrote: post #269
merb wrote: post #215

Earlier quoted context omitted.

Mistral is Europe based where invoices are more or less sent digitally in like 95% of all the cases anyway. Some are even digital invoices, which will at some point in the eu be mandatory. For orders there are proposals for that, too. And basically invoice data extraction is a different beast.

This isn't even close to true. Source: We have large EU customers.

So your eu customer will send you the invoice via letters ? Wow. There are some companies that still deal with printed invoices, but they are most often smaller companies that deal with health related things.

Re: Mistral OCR

#352

Post by merb »

codetrotter wrote: post #234
merb wrote: post #215

Earlier quoted context omitted.

Mistral is Europe based where invoices are more or less sent digitally in like 95% of all the cases anyway. Some are even digital invoices, which will at some point in the eu be mandatory. For orders there are proposals for that, too. And basically invoice data extraction is a different beast.

One use-case is digitising receipts from business related travels for expenses that employees paid for out of their own pocket and which they are submitting pictures to the business for reimbursement. Bus travels, meals including dinners and snacks, etc. for which the employee has receipts on paper.

Receipts are different. And they are harder to OCR. Thermo prints are most often aweful in quality. Most often you need to correct some stuff when dealing with them. I doubt that this tech changes that significantly.

Re: Mistral OCR

#354

Post by DeathArrow »

vikp wrote: post #289

I ran a partial benchmark against marker - https://github.com/VikParuchuri/marker . Across 375 samples with LLM as a judge, mistral scores 4.32, and marker 4.41 . Marker can inference between 20 and 120 pages per second on an H100. You can see the samples here - https://huggingface.co/datasets/datalab-to/marker_comparison... . The code for the benchmark is here - https://github.com/VikParuchuri/marker/tree/master/ben…

>Mistral OCR is an impressive model, but OCR is a hard problem, and there is a significant risk of hallucinations/missing text with LLMs.

To fight hallucinations, can't we use more LLMs and pick blocks where the majority of LLMs agree?

Re: Mistral OCR

#355

Post by themanmaran »

lolinder wrote: post #332
vikp wrote: post #289

I ran a partial benchmark against marker - https://github.com/VikParuchuri/marker . Across 375 samples with LLM as a judge, mistral scores 4.32, and marker 4.41 . Marker can inference between 20 and 120 pages per second on an H100. You can see the samples here - https://huggingface.co/datasets/datalab-to/marker_comparison... . The code for the benchmark is here - https://github.com/VikParuchuri/marker/tree/master/ben…

> with LLM as a judge For anyone else interested, prompt is here [0]. The model used was gemini-2.0-flash-001. Benchmarks are hard, and I understand the appeal of having something that seems vaguely deterministic rather than having a human in the loop, but I have a very hard time accepting any LLM-judged benchmarks at face value. This is doubly true when we're talking about something like OCR which, as you say, is a…

We also ran an OCR benchmark with LLM as judge using structured outputs. You can check out the full methodology on the repo [1]. But the general idea is:

- Every document has ground truth text, a JSON schema, and the ground truth JSON.

- Run OCR on each document and pass the result to GPT-4o along with the JSON Schema

- Compare the predicted JSON against the ground truth JSON for accuracy.

In our benchmark, the ground truth text => gpt-4o was 99.7%+ accuracy. Meaning whenever gpt-4o was given the correct text, it could extract the structured JSON values ~100% of the time. So if we pass in the OCR text from Mistral and it scores 70%, that means the inaccuracies are isolated to OCR errors.

https://github.com/getomni-ai/benchmark

Re: Mistral OCR

#356

Post by boxed »

DeathArrow wrote: post #354
vikp wrote: post #289

I ran a partial benchmark against marker - https://github.com/VikParuchuri/marker . Across 375 samples with LLM as a judge, mistral scores 4.32, and marker 4.41 . Marker can inference between 20 and 120 pages per second on an H100. You can see the samples here - https://huggingface.co/datasets/datalab-to/marker_comparison... . The code for the benchmark is here - https://github.com/VikParuchuri/marker/tree/master/ben…

>Mistral OCR is an impressive model, but OCR is a hard problem, and there is a significant risk of hallucinations/missing text with LLMs. To fight hallucinations, can't we use more LLMs and pick blocks where the majority of LLMs agree?

Why wouldn't hallucinations be agreed upon if they have roughly the same training data?

Re: Mistral OCR

#357

Post by cdolan »

themanmaran wrote: post #355
lolinder wrote: post #332

Earlier quoted context omitted.

> with LLM as a judge For anyone else interested, prompt is here [0]. The model used was gemini-2.0-flash-001. Benchmarks are hard, and I understand the appeal of having something that seems vaguely deterministic rather than having a human in the loop, but I have a very hard time accepting any LLM-judged benchmarks at face value. This is doubly true when we're talking about something like OCR which, as you say, is a…

We also ran an OCR benchmark with LLM as judge using structured outputs. You can check out the full methodology on the repo [1]. But the general idea is: - Every document has ground truth text, a JSON schema, and the ground truth JSON. - Run OCR on each document and pass the result to GPT-4o along with the JSON Schema - Compare the predicted JSON against the ground truth JSON for accuracy. In our benchmark, the groun…

were you guys able to finish running the benchmark with mistral and got a 70% score? Missed that

Edit - I see it on the Benchmark page now. Woof, low 70% scores in some areas!

https://getomni.ai/ocr-benchmark

Re: Mistral OCR

#358

Post by sadcrab »

kapitalx wrote: post #153
bambax wrote: post #136

Earlier quoted context omitted.

Where did you test it? At the end of the post they say: > Mistral OCR capabilities are free to try on le Chat but when asked, Le Chat responds: > can you do ocr? > I don't have the capability to perform Optical Character Recognition (OCR) directly. However, if you have an image with text that you need to extract, you can describe the text or provide details, and I can help you with any information or analysis related…

Yes I used the API. They have examples here: https://docs.mistral.ai/capabilities/document/ I used base64 encoding of the image of the pdf page. The output was an object that has the markdown, and coordinates for the images: [OCRPageObject(index=0, markdown='![img-0.jpeg](img-0.jpeg)', images=[OCRImageObject(id='img-0.jpeg', top_left_x=140, top_left_y=65, bottom_right_x=2136, bottom_right_y=1635, image_base64=None)],…

Any luck with this? I'm trying to process photos of paperwork (.pdf, .png) and got the same results as you.

Feels like something is missing in the docs, or the API itself.

https://imgur.com/a/1J9bkml

Re: Mistral OCR

#359

Post by hdjrudni »

Still terrible at handwriting.

I signed up for the API, cobbled together from their tutorial (https://docs.mistral.ai/capabilities/document/) -- why can't they give the full script instead of little bits?

Tried uploading a tiff, they rejected it. Tried upload JPG, they rejected it (even though they supposed support images?). Tried resaving as PDF. It took that, but the output was just bad. Then tried ChatGPT on the original .tiff (not using API), and it got it perfectly. Honestly I could barely make out the handwriting with my eyes but now that I see ChatGPT's version I think it's right.

Re: Mistral OCR

#360

Post by hackernewds »

raunakchowdhuri wrote: post #345

We ran some benchmarks comparing against Gemini Flash 2.0. You can find the full writeup here: https://reducto.ai/blog/lvm-ocr-accuracy-mistral-gemini A high level summary is that while this is an impressive model, it underperforms even current SOTA VLMs on document parsing and has a tendency to hallucinate with OCR, table structure, and drop content.

meanwhile, you're comparing it to the output of almost a trillion dollar company
Post reply on HN