Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Marker: Convert PDF to Markdown quickly with high accuracy

github.com

Post reply on HN
Print view View original

81–90 of 101 posts

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#81

Post by solardev »

Alex3917 wrote: post #48
sertbdfgbnfgsd wrote: post #47

Earlier quoted context omitted.

pdfs don't play well with ereaders.

Are the standards for building accessible PDFs worse than the standards for building accessible websites, or are they just not as commonly implemented?

(anecdotally) PDFs usually come from many people, departments, companies, and apps. It's hard to shoehorn in accessibility if someone didn't add it in at the origin (like in indesign or whatever app they used). Or if they printed to PDF, whatever accessibility they had would probably be lost. Much of the time it's like working with a raster image with some embedded text. Not really the same as being able to edit a proper semantic document.

With a website and available source code, any dev working on it later on can still add accessibility, tweak contrasts and fonts and add screen reader hints, etc.

It's much harder to do so for PDFs after the fact. And PDF viewer apps may or may not even support the accessibility annotations. By contrast all the major browsers and operating systems have OK support for web accessibility.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#82

Post by chaxor »

Alex3917 wrote: post #45
mannycalavera42 wrote: post #31

Let's not underestimate the impact of such tool: we are talking about freeing up tons of knowledge from a "good for consumption/bad for distribution" format. I'm very excited about it. Let's build a pipeline: all the pdfs -> markdown them all -> archive.org them all

> we are talking about freeing up tons of knowledge from a "good for consumption/bad for distribution" format. FWIW PDF is actually great for distribution. It allows you to invisibly embed all the raw data used to generate the document that the end user is seeing, in whatever format you want. So if you are generating your PDFs by using PrinceXML to render HTML, you can embed the raw JSON used to generate all of the t…

Yeah, totally. PDFs are wonderful for archiving.*

They can hold so many different types of data so that they're extremely difficult to parse.

Because of this, you can put several malicious programs into them for RCE.

That way, if someone archives many PDFs, there can be a plethora of different RCE vulnerabilities just waiting for the user to discover.

It's a wonderful dream for any malicious actor.

* /s

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#83

Post by vikp »

civilitty wrote: post #79
vikp wrote: post #62

Earlier quoted context omitted.

Author here: for my use case (converting scientific PDFs in bulk), nougat was the best solution, so I compared to it as the default. I also compare to naive text extraction further down. Nougat is a great model, and converts a lot of PDFs very well. I just wanted something faster, and more generalizable.

Great work! I just tried it on Linux for System Administrators and it did a great job properly picking up on code and config text. I noticed marker downloaded a PyTorch checkpoint called `nougat-0.1.0-small`, do you use nougat under the hood too or is that just a coincidence?

Yes, nougat is used as part of the pipeline to convert the equations (basically marker detects the equations then passes those regions to nougat). It's a great model for this.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#84

Post by vikp »

s1291 wrote: post #71
vikp wrote: post #65

Earlier quoted context omitted.

(author) Please feel free to open an issue if you try again. Poetry can be painful, I might just switch to a requirements.txt file in the future. (you can skip poetry if you want by just pulling everything in pyproject.toml into a requirements.txt file also)

Is there a plan to release this package as a docker image?

Yes, this is on my list of things to do :)

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#85

Post by Gabrys1 »

mannycalavera42 wrote: post #31

Let's not underestimate the impact of such tool: we are talking about freeing up tons of knowledge from a "good for consumption/bad for distribution" format. I'm very excited about it. Let's build a pipeline: all the pdfs -> markdown them all -> archive.org them all

Finally a good usecase for AI/ML/LLM.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#87

Post by afandian »

samuell wrote: post #39
afandian wrote: post #22

Earlier quoted context omitted.

My current workflow (for getting a magazine onto a website) is Calibre's HTMLZ export, then through Pandoc to markdown. It produces good enough Markdown to feed in to Hugo, and extracts images. I've been through a number of options in the past and this is what I've settled on.

Interesting! I tried it, but it seems to struggle with multi-column layouts (lines get intermingled). Is that something you tried?

No, only standard paragraphs.

My workflow still takes manual tweaking. When I find floated figures with captions, the lines get intertwingled and need to be unintertwingled. So I'm not surprised it didn't work for you.

Good luck, report back if you find what you're looking for. I'm always on the lookout for a better way.

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#88

Post by is_true »

iamflimflam1 wrote: post #26

How good is tesseract for OCR nowadays? I tried using it a while back and it was nowhere near as good as the online offerings from AWS, Azure and GCP.

It requires quite a bit of preprocessing. I've only tried GCP's solution which it's better in my experience

Re: Marker: Convert PDF to Markdown quickly with high accuracy

#90

Post by habosa »

iamflimflam1 wrote: post #26

How good is tesseract for OCR nowadays? I tried using it a while back and it was nowhere near as good as the online offerings from AWS, Azure and GCP.

I found it to be surprisingly good and I was very impressed with the in-browser performance. It is very very sensitive to resolution though. Once my images got down to a certain size they produced garbage from Tesseract even though they were very human readable.
Post reply on HN