Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral OCR

mistral.ai

Post reply on HN
Print view View original

341–350 of 450 posts

Re: Mistral OCR

#341

Post by arthurcolle »

lolinder wrote: post #332
vikp wrote: post #289

I ran a partial benchmark against marker - https://github.com/VikParuchuri/marker . Across 375 samples with LLM as a judge, mistral scores 4.32, and marker 4.41 . Marker can inference between 20 and 120 pages per second on an H100. You can see the samples here - https://huggingface.co/datasets/datalab-to/marker_comparison... . The code for the benchmark is here - https://github.com/VikParuchuri/marker/tree/master/ben…

> with LLM as a judge For anyone else interested, prompt is here [0]. The model used was gemini-2.0-flash-001. Benchmarks are hard, and I understand the appeal of having something that seems vaguely deterministic rather than having a human in the loop, but I have a very hard time accepting any LLM-judged benchmarks at face value. This is doubly true when we're talking about something like OCR which, as you say, is a…

You can use structured outputs, or something like my https://arthurcolle--dynamic-schema.modal.run/

to extract real data from unstructured text (like that producted from an LLM) to make benchmarks slightly easier if you have a schema

Re: Mistral OCR

#342

Post by [deleted] »

vikp wrote: post #289

I ran a partial benchmark against marker - https://github.com/VikParuchuri/marker . Across 375 samples with LLM as a judge, mistral scores 4.32, and marker 4.41 . Marker can inference between 20 and 120 pages per second on an H100. You can see the samples here - https://huggingface.co/datasets/datalab-to/marker_comparison... . The code for the benchmark is here - https://github.com/VikParuchuri/marker/tree/master/ben…

[deleted]

Re: Mistral OCR

#343

Post by simonw »

gatienboquet wrote: post #195
simonw wrote: post #182

Earlier quoted context omitted.

What do you mean by agent?

La Plateforme agent builder - https://console.mistral.ai/build/agents/new

Oh neat, thanks - I hadn't seen that. Looks like their version of an "agent" is a model with pre-baked system prompt and some examples.

Re: Mistral OCR

#344

Post by codelion »

vikp wrote: post #289

I ran a partial benchmark against marker - https://github.com/VikParuchuri/marker . Across 375 samples with LLM as a judge, mistral scores 4.32, and marker 4.41 . Marker can inference between 20 and 120 pages per second on an H100. You can see the samples here - https://huggingface.co/datasets/datalab-to/marker_comparison... . The code for the benchmark is here - https://github.com/VikParuchuri/marker/tree/master/ben…

Really interesting benchmark, thanks for sharing! It's good to see some real-world comparisons. The hallucinations issue is definitely a key concern with LLM-based OCR, and it's important to quantify that risk. Looking forward to seeing the full benchmark results.

Re: Mistral OCR

#345

Post by raunakchowdhuri »

We ran some benchmarks comparing against Gemini Flash 2.0. You can find the full writeup here: https://reducto.ai/blog/lvm-ocr-accuracy-mistral-gemini

A high level summary is that while this is an impressive model, it underperforms even current SOTA VLMs on document parsing and has a tendency to hallucinate with OCR, table structure, and drop content.

Re: Mistral OCR

#346

Post by bredren »

groby_b wrote: post #39
bredren wrote: post #14

Earlier quoted context omitted.

Ya, presumably it is missing the number `1.00`.

Not really. When you go 60 mph (or km/h) you don't specify the 1.00 for the hours either. pages/$ is the unit, 1000 is the value.

But you do for virtually all other cloud pricing pages.

Re: Mistral OCR

#347

Post by siva7 »

qwertox wrote: post #167

We developers seem to really dislike PDFs, to a degree that we'll build LLMs and have them translate it into Markdown. Jokes aside, PDFs really serve a good purpose, but getting data out of them is usually really hard. They should have something like an embedded Markdown version with a JSON structure describing the layout, so that machines can easily digest the data they contain.

> We developers seem to really dislike PDFs, to a degree that we'll build LLMs and have them translate it into Markdown.

Why Jokes aside? Markdown/html is better suited for the web than pdf

Re: Mistral OCR

#349

Post by DonHopkins »

ralusek wrote: post #120
kapitalx wrote: post #104

Earlier quoted context omitted.

From my testing so far, it seems it's super fast and responded synchronously. But it decided that the entire page is an image and returned `![img-0.jpeg](img-0.jpeg)` with coordinates in the metadata for the image, which is the entire page. Our tool, doctly.ai is much slower and async, but much more accurate and gets you the content itself as an markdown.

I thought we stopped -ly company names ~8 years ago?

But doctr.ai was taken.

Re: Mistral OCR

#350

Post by merb »

revnode wrote: post #232
merb wrote: post #215

Earlier quoted context omitted.

Mistral is Europe based where invoices are more or less sent digitally in like 95% of all the cases anyway. Some are even digital invoices, which will at some point in the eu be mandatory. For orders there are proposals for that, too. And basically invoice data extraction is a different beast.

So an invoice attached to an email as a PDF is sent digitally ... those unfamiliar with PDF will think text and data extraction is trivial then, but this isn't true. You can have a fully digital, non-image PDF that is vector based and has what looks like text, but doesn't have a single piece of extractable text in it. It's all about how the PDF was generated. Tables can be formatted in a million ways, etc. Your best…

This is simply not true. Maybe it’s easier and you do not need 100% precision. But it is actually possible to extract text and layout of digital pdfs. Else it would be impossible to display it. Of course some people still add image fragments to a pdf, but that practice is basically dying. I did not see a single pdf the last year we‘re it was impossible to extract the layout.
Post reply on HN