Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral OCR

mistral.ai

Post reply on HN
Print view View original

411–420 of 450 posts

Re: Mistral OCR

#411

Post by mvac »

thelittleone wrote: post #294
mvac wrote: post #142

Great progress, but unfortunately, for our use case (converting medical textbooks from PDF to MD), the results are not as good as those by MinerU/PDF-Extract-Kit [1]. Also the collab link in the article is broken, found a functional one [2] in the docs. [1] https://github.com/opendatalab/MinerU [2] https://colab.research.google.com/github/mistralai/cookbook/...

Have you had a chance to compare results from MinerU vs LLM such a Gemini 2.0 or anthropic's native PDF tool?

Yes, i have. The problem with using just an LLM is that while it reads and understands text, but it cannot reproduce it accurately. Additionaly the textbooks I've mentioned have many diagrams and illustrations in them (e.g. books on anatomy or biochemistry). I don't really care about extracting text from them, I just need them extracted as images alongside the text, and no LLM does that.

Re: Mistral OCR

#412

Post by cdolan »

arthurcolle wrote: post #341
lolinder wrote: post #332

Earlier quoted context omitted.

> with LLM as a judge For anyone else interested, prompt is here [0]. The model used was gemini-2.0-flash-001. Benchmarks are hard, and I understand the appeal of having something that seems vaguely deterministic rather than having a human in the loop, but I have a very hard time accepting any LLM-judged benchmarks at face value. This is doubly true when we're talking about something like OCR which, as you say, is a…

You can use structured outputs, or something like my https://arthurcolle--dynamic-schema.modal.run/ to extract real data from unstructured text (like that producted from an LLM) to make benchmarks slightly easier if you have a schema

What is the project? It just returns a vanilla html page saying:

Dynamic Schema API API is running. See documentation for available endpoints.

Re: Mistral OCR

#413

Post by InvidFlower »

hdjrudni wrote: post #359

Still terrible at handwriting. I signed up for the API, cobbled together from their tutorial ( https://docs.mistral.ai/capabilities/document/ ) -- why can't they give the full script instead of little bits? Tried uploading a tiff, they rejected it. Tried upload JPG, they rejected it (even though they supposed support images?). Tried resaving as PDF. It took that, but the output was just bad. Then tried ChatGPT on the…

It is confusing, but they have diff calls for pdfs vs images. In their example google colab: https://colab.research.google.com/drive/11NdqWVwC_TtJyKT6cmu...

The first couple of sections are for pdfs and you need to skip all that (search for "And Image files...") to find the image extraction portion. Basically it needs ImageURLChunk instead of DocumentURLChunk.

Re: Mistral OCR

#414

Post by InvidFlower »

While it is nice to have more options, it still definitely isn't at a human level yet for hard to read text. Still haven't seen anything that can deal with something like this very well: https://i.imgur.com/n2sBFdJ.jpeg

If I remember right, Gemini actually was the closest as far as accuracy of the parts where it "behaved", but it'd start to go off the rails and reword things at the end of larger paragraphs. Maybe if the image was broken up into smaller chunks. In comparison, Mistral for the most part (besides on one particular line for some reason) sticks to the same number of words, but gets a lot wrong on the specifics.

Re: Mistral OCR

#415

Post by thomasahle »

I'm surprised they didn't benchmark it against Pixtral.

They test it against a bunch of different Multimodal LLMs, so why not their own?

I don't really see the purpose of the OCR form factor, when you have multimodal LLMs. Unless it's significantly cheaper.

Re: Mistral OCR

#416

Post by ChrisRob »

vikp wrote: post #289

I ran a partial benchmark against marker - https://github.com/VikParuchuri/marker . Across 375 samples with LLM as a judge, mistral scores 4.32, and marker 4.41 . Marker can inference between 20 and 120 pages per second on an H100. You can see the samples here - https://huggingface.co/datasets/datalab-to/marker_comparison... . The code for the benchmark is here - https://github.com/VikParuchuri/marker/tree/master/ben…

Thanks for the tip. Marker solved a table conversion without LLM that docling wasn't able to solve.

Re: Mistral OCR

#417

Post by butovchenkoy »

lokl wrote: post #126

Tried with a few historical handwritten German documents, accuracy was abysmal.

For this task, general models will always perform poorly. My company trains custom gen ai models for document understanding. We recently trained a VLM for the German government to recognize documents written in old German handwriting, and it performed with exceptionally high accuracy.

Re: Mistral OCR

#418

Post by butovchenkoy »

thadt wrote: post #274
lokl wrote: post #126

Tried with a few historical handwritten German documents, accuracy was abysmal.

Also working with historical handwritten German documents. So far Gemini seems to be the least wrong of the ones I've tried - any recommendations?

my recommendation is to train a custom model

Re: Mistral OCR

#419

Post by notepad0x90 »

devmor wrote: post #291
notepad0x90 wrote: post #212

Earlier quoted context omitted.

Then I think you misunderstand. The ML system would know when you want things digested to you or not. Right now companies are assuming this and forcing LLM interaction. But when properly done, the system would know based on your behavior or explicit prompts what you want and provide the service. If you're staring at a paragraph intently and confused, it might start highlighting common phrases or parts of the text/pic…

No, I fully understand. I am saying that this type of system, that deprives the user of problem solving, is itself a problem. A detriment to the very essence of human intelligence.

I just look at it as allowing the user to focus on problems that aren't already easily solved. Like using a calculator instead of calculating manually on paper.

Re: Mistral OCR

#420

Post by culi »

lingjiekong wrote: post #403
themanmaran wrote: post #371

Earlier quoted context omitted.

Yup, surprising results! We were able to dig in a bit more. Main culprit is the overzealous "image extraction". Where if Mistral classifies something as an image, it will replace the entire section with (image)[image_002). And it happened with a lot of full documents as well. Ex: most receipts got classified as images, and so it didn't extract any text.

where do you find this regarding "Where if Mistral classifies something as an image, it will replace the entire section with (image)[image_002)."?

themanmaran works at Omni so presumably they have access to the actual resulting data from this study
Post reply on HN