Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nanonets-OCR-s – OCR model that transforms documents into structured markdown

huggingface.co

Post reply on HN
Print view View original

61–70 of 85 posts

Re: Nanonets-OCR-s – OCR model that transforms documents into structured markdown

#61

Post by lukev »

mgr86 wrote: post #29
souvik3333 wrote: post #23

Earlier quoted context omitted.

Actually, we have trained the model to convert to markdown and do semantic tagging at the same time. Eg, the equations will be extracted as LaTeX equations, and images (plots, figures, and so on) will be described within the ` ` tags. Same with ` `, ` `, . Also, we extract the tables as HTML tables instead of markdown for complex tables.

Have you considered XML. TEI, for example, is very robust and mature for marking up documents.

Yeah this really hurts. If your goal is to precisely mark up a document with some structural elements, XML is strictly superior to Markdown.

The fact that someone would go to all the work to build a model to extract the structure of documents, then choose an output format strictly less expressive than XML, speaks poorly of the state of cross-generational knowledge sharing within the industry.

Re: Nanonets-OCR-s – OCR model that transforms documents into structured markdown

#62

Post by jtbayly »

souvik3333 wrote: post #32
jtbayly wrote: post #24

Earlier quoted context omitted.

What happens to footnotes?

They will be extracted in a new line as normal text. It will be the last line.

So I’m left to manually link them up?

Have you considered using something like Pandoc’s method of marking them up? Footnotes are a fairly common part of scanned pages, and markdown that doesn’t indicate that a footnote is a footnote can be fairly incomprehensible.

Re: Nanonets-OCR-s – OCR model that transforms documents into structured markdown

#63

Post by [deleted] »

generalizations wrote: post #58
j45 wrote: post #57

Earlier quoted context omitted.

If you are after extracting images from pdfs there’s plenty of tools that do that just fine without LLMs.

I mean, ideally it would be in context, so the generated markdown references the correct image at the correct location in the doc. Unless that's what you're talking about? In which case I don't know about those tools.

[deleted]

Re: Nanonets-OCR-s – OCR model that transforms documents into structured markdown

#64

Post by gibsonf1 »

nattaylor wrote: post #59
gibsonf1 wrote: post #56

Earlier quoted context omitted.

Does it hallucinate with the LLM being used?

The base model is Qwen2.5-VL-3B and the announcement says a limitation is "Model can suffer from hallucination"

Seems a bit scary that the "source" text from the pdfs could actually be hallucinated.

Re: Nanonets-OCR-s – OCR model that transforms documents into structured markdown

#65

Post by michaelt »

gibsonf1 wrote: post #56
PixelPanda wrote: post #2

Full disclaimer: I work at Nanonets Excited to share Nanonets-OCR-s, a powerful and lightweight (3B) VLM model that converts documents into clean, structured Markdown. This model is trained to understand document structure and content context (like tables, equations, images, plots, watermarks, checkboxes, etc.). Key Features: LaTeX Equation Recognition Converts inline and block-level math into properly formatted LaTe…

Does it hallucinate with the LLM being used?

Sometimes. I just fed the huggingface demo an image containing some rather improbable details [1] and it OCRed "Page 1000000000000" with one extra trailing zero.

Honestly I was expecting the opposite - a repetition penalty to kick in having repeated zero too many times, resulting in too few zeros - but apparently not. So you might want to steer clear of this model if your document has a trillion pages.

Other than that, it did a solid job - I've certainly seen worse attempts to OCR a table.

[1] https://imgur.com/a/8rJeHf8

Re: Nanonets-OCR-s – OCR model that transforms documents into structured markdown

#66

Post by prats226 »

gibsonf1 wrote: post #64
nattaylor wrote: post #59

Earlier quoted context omitted.

The base model is Qwen2.5-VL-3B and the announcement says a limitation is "Model can suffer from hallucination"

Seems a bit scary that the "source" text from the pdfs could actually be hallucinated.

Given that input is image and not raw pdf, its not completely unexpected

Re: Nanonets-OCR-s – OCR model that transforms documents into structured markdown

#67

Post by prats226 »

lukev wrote: post #61
mgr86 wrote: post #29

Earlier quoted context omitted.

Have you considered XML. TEI, for example, is very robust and mature for marking up documents.

Yeah this really hurts. If your goal is to precisely mark up a document with some structural elements, XML is strictly superior to Markdown. The fact that someone would go to all the work to build a model to extract the structure of documents, then choose an output format strictly less expressive than XML, speaks poorly of the state of cross-generational knowledge sharing within the industry.

I think the choice mainly stems from how you want to use the output. If the output is going to get fed to another LLM, then you want to select markup language where 1) the grammer would not cause too many issues with tokenization 2) which LLM has seen a lot in past 3) generates minimal number of tokens. I think markdown fits it much better compared to other markup languages.

If goal is to parse this output programmatically, then I agree a more structured markup language is better choice.

Re: Nanonets-OCR-s – OCR model that transforms documents into structured markdown

#69

Post by vikp »

nxobject wrote: post #34
mvac wrote: post #18

How does it compare to Datalab/Marker https://github.com/datalab-to/marker ? We evaluated many PDF->MD converters and this one performed the best, though it is not perfect.

As anecdotal evidence, it serves my complex-enough purposes very well - mathematics and code interspersed together. One of my "litmus test" papers is this old paper on a Fortran inverse-Laplace transform algorithm [1] that intersperses inline and display equations, and monospace code blocks, while requiring OCR from scratch, and very few models currently do a satisfactory job, i.e. in the following page transcribed b…

Hi, author of marker here - I tried your image, and I don't see the issues you're describing with the newest version of marker (1.7.5).

I ran both with no setting specified, and with force_ocr, and I didn't see the issues either time.

Re: Nanonets-OCR-s – OCR model that transforms documents into structured markdown

#70

Post by jwr »

Thank you! This is very interesting — I'm just curious, why use such a small model?

I can comfortably run 27B models on my Mac and I'd much rather process my PDF library with something that is less prone to hallucinations and handles multiple languages better…

Post reply on HN