Live data from Hacker News

Marker: Convert PDF to Markdown quickly with high accuracy

github.com

61–70 of 101 posts

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#61
Can someone help me understand the line

>Due to the licensing of the underlying models like layoutlmv3 and nougat, this is only suitable for noncommercial usage.

Does this mean it isn't suitable if I wanted to use it in a product for sale or I cannot use it for tasks at my work? I would like to try to use this at work to convert vendor documentation to include in our internal wiki.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#62
post #4

Great work! I am a bit confused with the comparison with nougat throughout the repo. Nougat was specifically trained for academic documents, and I don't think anyone ever claimed Nougat was the best OCR model out there. That's kinda clear in your benchmark too where you mention that nougat has higher accuracy on arxiv documents. You also mention that marker will convert fewer equations when compared to nougat, and ye…

Author here: for my use case (converting scientific PDFs in bulk), nougat was the best solution, so I compared to it as the default. I also compare to naive text extraction further down.

Nougat is a great model, and converts a lot of PDFs very well. I just wanted something faster, and more generalizable.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#63
post #37
post #35

Earlier quoted context omitted.

Yeah, I know, but a lot of this content can be pretty sensitive, and might not be allowed to upload outside organization networks sometimes (hospitals, governments etc).

Like most software, LLMs can be run locally, or on private infrastructure. This was on the front page yesterday, which is not the only way to run an LLM locally, but about the easiest way possible: https://news.ycombinator.com/item?id=38464057

Thanks! Well, yea, I just thought the quality of offline models might not yet be good enough. By I'm glad to be told otherwise :)

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#64

Question for the author: Why to markdown? It seems to me the hard part of this tool is parsing pdfs with high accuracy, not whatever you do with them. As such, I would love if this tool allowed the user to choose the output format. I know that I would use a high accuracy pdf parser to render into epub.

I chose markdown because I wanted to preserve equations (fenced by $/$$), tables, bold/italic information, and headers. I haven't looked into epub output, but this ruled out plain text.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#65
post #58

Earlier quoted context omitted.

I did it on mac without any issues. Are you using Mac or Linux? what is the issue?

I'm using Ubuntu 22.04. I encountered several errors with Poetry and attempted to fix them but eventually gave up.

(author) Please feel free to open an issue if you try again. Poetry can be painful, I might just switch to a requirements.txt file in the future. (you can skip poetry if you want by just pulling everything in pyproject.toml into a requirements.txt file also)

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#68
post #61

Can someone help me understand the line >Due to the licensing of the underlying models like layoutlmv3 and nougat, this is only suitable for noncommercial usage. Does this mean it isn't suitable if I wanted to use it in a product for sale or I cannot use it for tasks at my work? I would like to try to use this at work to convert vendor documentation to include in our internal wiki.

If your work is commercial then you cannot use it. Think of it this way, is your work being used in a commercial business. Then it cannot be used. If you are using this for personal use or anything that is not part of a business, its ok.
Post reply on HN