Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

metachris.com

Post reply on HN
Print view View original

11–20 of 36 posts

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#11

Post by afandian »

rossmounce wrote: post #9

Most PDF scientific articles sadly don't have good embedded metadata [0], so this & the DOI issue make this not very useful (at least for the journals I read). I also would have implemented this with a simpler shell script calling exiftool[1] and pdftotext[2], but hey; fun to have a python-based implementation :) [0] http://rossmounce.co.uk/2012/12/31/pdf-metadata-why-so-poor/ [1] http://www.sno.phy.queensu.ca/~phil/…

What's "the DOI issue"?

(Also, if your favourite publisher isn't putting the metadata you want in PDFs, write to them and ask! It may make a difference...)

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#12

Post by rossmounce »

afandian wrote: post #11
rossmounce wrote: post #9

Most PDF scientific articles sadly don't have good embedded metadata [0], so this & the DOI issue make this not very useful (at least for the journals I read). I also would have implemented this with a simpler shell script calling exiftool[1] and pdftotext[2], but hey; fun to have a python-based implementation :) [0] http://rossmounce.co.uk/2012/12/31/pdf-metadata-why-so-poor/ [1] http://www.sno.phy.queensu.ca/~phil/…

What's "the DOI issue"? (Also, if your favourite publisher isn't putting the metadata you want in PDFs, write to them and ask! It may make a difference...)

the DOI issue is as described by "aroch"

Quickscrape is capable of downloading PDFs, given a DOI I think: https://github.com/ContentMine/quickscrape

If publishers listened to their readers we would have had 100% Open Access ten years ago. Traditional academic publishers do not listen and don't care.

Even if I could convince say PLOS to do something about this it wouldn't change much. We need all or the majority of publishers to provide good embedded metadata. Not just an isolated one or two. I don't see a good mechanism for making that happen, sadly.

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#13

Post by aroch »

IanCal wrote: post #6
aroch wrote: post #2

While nice, it really only works with the reference has a direct link to the PDF while the majority of citations use the DOI in the sciences. DOI traversal would be required

DOIs are great for humans, unfortunately they'll take you to the publishers webpage and I don't know of a standard way of getting an actual PDF from a DOI. Maybe with PLOS, I know they're good at serving up different versions (xml with a different accept header iirc). Searching for something on the page that looks like a download PDF button and trying that might get you 80% of the way there, along with at least givin…

afandian's reply below is exactly how I would go about it. Most DOI's in science papers are crossmark OR convertable to a crossmark DOI

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#14

Post by adelevie »

This is really neat! For work, I've found myself from time to time exploring the tech around PDFs. I find this tech strangely fascinating. It's like a shim on top of something old and ugly that enables integration with much more modern systems.

Some quick feedback (and a shameless plug):

The CLI interface should output JSON. It would be nice to combin with a CLI JSON parser such as jq[0].

Shameless plug: I've been working on a PDF CLI aimed at making it easier to programmatically fill out PDF forms: https://github.com/adelevie/pdfq. It provides an interface and some wrappers on top of the main pdf form-filling tool, pdftk. For example, you can get json out of a pdf form like this:

    pdftk hello.pdf dump_data_fields | pdfq
Or you can generate FDF from a json file:

    cat hello.json | pdfq json_to_fdf
You can also fill a pdf without touching an fdf code:

    pdfq set foo bar input.pdf output.pdf
[0] https://stedolan.github.io/jq/

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#15

Post by metachris »

adelevie wrote: post #14

This is really neat! For work, I've found myself from time to time exploring the tech around PDFs. I find this tech strangely fascinating. It's like a shim on top of something old and ugly that enables integration with much more modern systems. Some quick feedback (and a shameless plug): The CLI interface should output JSON. It would be nice to combin with a CLI JSON parser such as jq[0]. Shameless plug: I've been wo…

pdfx will output json if you use the -j flag!

    pdfx -j
jq looks neat btw.

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#16

Post by IanCal »

afandian wrote: post #7
IanCal wrote: post #6

Earlier quoted context omitted.

DOIs are great for humans, unfortunately they'll take you to the publishers webpage and I don't know of a standard way of getting an actual PDF from a DOI. Maybe with PLOS, I know they're good at serving up different versions (xml with a different accept header iirc). Searching for something on the page that looks like a download PDF button and trying that might get you 80% of the way there, along with at least givin…

The standard way for getting the actual PDF from a DOI, when it's a Crossref DOI (which it probably is) is to use the full-text link, available in the CrossRef API. For DOI 10.1155/2010/963926 http://api.crossref.org/works/10.1155/2010/963926 From the returned JSON message -> link -> there's the PDF! [ { intended-application: "text-mining", content-version: "vor", content-type: "application/pdf", URL: "http://downloa…

Thanks, I'd not thought of the crossref api for this. I use the API pretty heavily for other things though, really good work!

Just noticed this part of the response:

    "affiliation": [],
How well filled in is that? I find it's currently a really poorly provided thing on many sites (although there are metatags, they're often wrong).

> When a PDF has Crossref CrossMark, the DOI is embedded in the metadata (I can't say how but I can find out)

That's likely to come in really useful, thanks.

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#17

Post by adelevie »

metachris wrote: post #15
adelevie wrote: post #14

This is really neat! For work, I've found myself from time to time exploring the tech around PDFs. I find this tech strangely fascinating. It's like a shim on top of something old and ugly that enables integration with much more modern systems. Some quick feedback (and a shameless plug): The CLI interface should output JSON. It would be nice to combin with a CLI JSON parser such as jq[0]. Shameless plug: I've been wo…

pdfx will output json if you use the -j flag! pdfx -j jq looks neat btw.

I should have read the [] manual :)

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#18

Post by chriswarbo »

On a related note, these past couple of weeks I've found myself wanting to import several years' worth of accumulated PDFs into a BibTeX file. This has involved metadata extraction, text scraping, querying Google Scholar (good, but rate-limited) and CrossRef (no limit, but not as accurate).

I've written a very rough guide to the approaches I've taken so far at http://chriswarbo.net/essays/pdf-tools.html , with a bunch of links to external tools, some NixOS package definitions, commandline snippets and descriptions of Emacs macros.

Not quite the same problem as the author's, but the tools and scripts I've been using can do similar things :)

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#19

Post by afandian »

IanCal wrote: post #16
afandian wrote: post #7

Earlier quoted context omitted.

The standard way for getting the actual PDF from a DOI, when it's a Crossref DOI (which it probably is) is to use the full-text link, available in the CrossRef API. For DOI 10.1155/2010/963926 http://api.crossref.org/works/10.1155/2010/963926 From the returned JSON message -> link -> there's the PDF! [ { intended-application: "text-mining", content-version: "vor", content-type: "application/pdf", URL: "http://downloa…

Thanks, I'd not thought of the crossref api for this. I use the API pretty heavily for other things though, really good work! Just noticed this part of the response: "affiliation": [], How well filled in is that? I find it's currently a really poorly provided thing on many sites (although there are metatags, they're often wrong). > When a PDF has Crossref CrossMark, the DOI is embedded in the metadata (I can't say ho…

    http://api.crossref.org/works?filter=has-affiliation:true

    => total-results: 964,696
Do ask us questions on labs@crossref.org or raise a ticket on https://github.com/CrossRef/rest-api-doc

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#20

Post by prolepunk »

metachris wrote: post #3

Glad if this tool/lib is useful to some. I'm happy to answer any and all questions! A Kivy [1] based cross-platform GUI would be a nice addition at some point. [1] http://kivy.org

How does this tool handle XFA forms?

Or I should ask how does any tool besides Adobe LiveCycle designer handles XFA. I've been using Apache Tika with Apache Solr to index PDF documents -- it worked quite well up until I tried to index XFA-based documents and got nothing, well I got metdatada, but not the content.

[pdftk](https://www.pdflabs.com/docs/pdftk-man-page/#dest-drop-xfa) seems to have an option called drop_xfa, which does just that.

Post reply on HN