Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Marker: Convert PDF to Markdown quickly with high accuracy

github.com

Post reply on HN
Print view View original

51–60 of 101 posts

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#51

Post by Finnucane »

sertbdfgbnfgsd wrote: post #46

Question for the author: Why to markdown? It seems to me the hard part of this tool is parsing pdfs with high accuracy, not whatever you do with them. As such, I would love if this tool allowed the user to choose the output format. I know that I would use a high accuracy pdf parser to render into epub.

You would want to have some kind fo markup that preserves structural markup as much as possible. I manage ebooks for a university press, and we have a deep backlist waiting for conversion, a lot of which only exists as page scans of old print volumes. I want to be able to offer them as epubs, which means I need to know where there are chapter breaks, heads, tables, charts, math, blockquotes, and so on and so forth. I have vendors that can do this for me, but it costs more than we'd get for some of these books in sales. I'd love to be able to do soem of this myself.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#52

Post by sertbdfgbnfgsd »

carschno wrote: post #49
sertbdfgbnfgsd wrote: post #46

Question for the author: Why to markdown? It seems to me the hard part of this tool is parsing pdfs with high accuracy, not whatever you do with them. As such, I would love if this tool allowed the user to choose the output format. I know that I would use a high accuracy pdf parser to render into epub.

I agree, the intermediate format should be plain text that could optionally be converted to any other format. I suppose that Markdown, however, is used as intermediate format here. It is close to plain text while it can preserve simple layout information. In practice, I would use the Markdown output and plug it into any tool that converts that into the desired final output format.

That sounds reasonable. I might explore pdf -> markdown -> epub.

I wonder if this could somehow be used directly by calibre. I think calibre's pdf->epub conversion isn't amazing. In particular, tables often end up broken.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#54

Post by hashemian »

Amazing work. Thank you.

I have a set of PDF files, and this week was thinking how I can link them to an LLM and be able to ask questions about them. So this was very timely.

I did a quick side-by-side testing against Nougat, and it clearly works better. On a handful of PDFs I tested, Marker extracted considerably more text (the text did not have any math, just academic papers), finished the job faster, and did not crash on any pdf, while Nougat took a lot longer to finish, and sometimes crashed due to out-of-memory error (could not allocate more than 7GB RAM!)

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#56

Post by hashemian »

s1291 wrote: post #55

I'm curious if anyone has had any success building this package. I've spent a lot of time trying to build it myself, but unfortunately haven't been able to get it to work. Has anyone else had better luck?"

I did it on mac without any issues. Are you using Mac or Linux? what is the issue?

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#57

Post by [deleted] »

sertbdfgbnfgsd wrote: post #47
Alex3917 wrote: post #45

Earlier quoted context omitted.

> we are talking about freeing up tons of knowledge from a "good for consumption/bad for distribution" format. FWIW PDF is actually great for distribution. It allows you to invisibly embed all the raw data used to generate the document that the end user is seeing, in whatever format you want. So if you are generating your PDFs by using PrinceXML to render HTML, you can embed the raw JSON used to generate all of the t…

pdfs don't play well with ereaders.

[deleted]

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#58

Post by s1291 »

hashemian wrote: post #56
s1291 wrote: post #55

I'm curious if anyone has had any success building this package. I've spent a lot of time trying to build it myself, but unfortunately haven't been able to get it to work. Has anyone else had better luck?"

I did it on mac without any issues. Are you using Mac or Linux? what is the issue?

I'm using Ubuntu 22.04. I encountered several errors with Poetry and attempted to fix them but eventually gave up.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#60

Post by vikp »

mannycalavera42 wrote: post #31

Let's not underestimate the impact of such tool: we are talking about freeing up tons of knowledge from a "good for consumption/bad for distribution" format. I'm very excited about it. Let's build a pipeline: all the pdfs -> markdown them all -> archive.org them all

Author here - this is one of the reasons I made this. Also see https://github.com/VikParuchuri/libgen_to_txt , although I haven't integrated marker with it yet (it uses naive text extraction).
Post reply on HN