Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

metachris.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#21

Post by afandian »

rossmounce wrote: post #12
afandian wrote: post #11

Earlier quoted context omitted.

What's "the DOI issue"? (Also, if your favourite publisher isn't putting the metadata you want in PDFs, write to them and ask! It may make a difference...)

the DOI issue is as described by "aroch" Quickscrape is capable of downloading PDFs, given a DOI I think: https://github.com/ContentMine/quickscrape If publishers listened to their readers we would have had 100% Open Access ten years ago. Traditional academic publishers do not listen and don't care. Even if I could convince say PLOS to do something about this it wouldn't change much. We need all or the majority of pu…

Well hopefully the coverage of full-text link metadata in Crossref will increase over time. Until then, best of luck to ContentMine!

Improving metadata coverage is a different issue than changing business model. Objectively, one's easier to implement than the other.

It is indeed tricky getting ~5,000 publishers to provide optimal metadata, but it's a good thing to have an industry-standard platform to do it in.

(I work at Crossref)

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#22

Post by retSava »

Slightly off-topic, pardon me. But, does someone have any good tips on how to remove pdf security?

Let me clarify why; I frequently come across datasheets (eg to flash memory ICs) that have security enabled for some strange reason. Nothing secret, just plainly downloaded from the Internet. I can open, and print, but not highlight or add remarks.

Existing solutions I've found so far are inadequate since they typically are either 'download this obscure-sounding executable', 'upload and convert on this sketchy possibly-malware-injecting-website', or resort to printing the entire thing to a new pdf document (eg via PDF creator) - but this makes text un-highlightable.

I don't mind anything involving hex-editing, some node.js or python-lib, or chanting and dancing, as long as it gets the job done.

I just want to be able to highlight and copy text :(

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#23

Post by PaulHoule »

adelevie wrote: post #14

This is really neat! For work, I've found myself from time to time exploring the tech around PDFs. I find this tech strangely fascinating. It's like a shim on top of something old and ugly that enables integration with much more modern systems. Some quick feedback (and a shameless plug): The CLI interface should output JSON. It would be nice to combin with a CLI JSON parser such as jq[0]. Shameless plug: I've been wo…

PDF is less proprietary than most people think. It is an ISO standard after all and it is a bit complicated but it does solve the problem of making "printable" documents produced by all sorts of tools available online.

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#24

Post by fit2rule »

Nice work Chris .. doesn't work on all my PDF's, though:

    j@w1x8-dev:~/Documents/PDF Documents {}
    $ pdfx xhyve\ –\ Lightweight\ Virtualization\ on\ OS\ X\ Based\ on\ bhyve\ _\ pagetable.pdf
    Traceback (most recent call last):
      File "/usr/local/bin/pdfx", line 9, in 
    load_entry_point('pdfx==1.0.1', 'console_scripts', 'pdfx')()
    File "build/bdist.macosx-10.10-x86_64/egg/pdfx/cli.py", line 66, in main
    File "build/bdist.macosx-10.10-x86_64/egg/pdfx/__init__.py", line 137, in __init__
    AttributeError: 'NoneType' object has no attribute 'items'
    j@w1x8-dev:~/Documents/PDF Documents {}
If you want some sample PDF's on which it is borked, just let me know .. in the meantime I'm using pdf_scraper for most of these ..

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#25

Post by metachris »

fit2rule wrote: post #24

Nice work Chris .. doesn't work on all my PDF's, though: j@w1x8-dev:~/Documents/PDF Documents {} $ pdfx xhyve\ –\ Lightweight\ Virtualization\ on\ OS\ X\ Based\ on\ bhyve\ _\ pagetable.pdf Traceback (most recent call last): File "/usr/local/bin/pdfx", line 9, in load_entry_point('pdfx==1.0.1', 'console_scripts', 'pdfx')() File "build/bdist.macosx-10.10-x86_64/egg/pdfx/cli.py", line 66, in main File "build/bdist.macos…

Thanks for the stack trace. Yes please, sample PDFs with problems would be great! You can find my email in my profile.

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#26

Post by metachris »

prolepunk wrote: post #20
metachris wrote: post #3

Glad if this tool/lib is useful to some. I'm happy to answer any and all questions! A Kivy [1] based cross-platform GUI would be a nice addition at some point. [1] http://kivy.org

How does this tool handle XFA forms? Or I should ask how does any tool besides Adobe LiveCycle designer handles XFA. I've been using Apache Tika with Apache Solr to index PDF documents -- it worked quite well up until I tried to index XFA-based documents and got nothing, well I got metdatada, but not the content. [pdftk]( https://www.pdflabs.com/docs/pdftk-man-page/#dest-drop-xfa ) seems to have an option called drop…

I don't know! If you could send me sample PDFs I'm more than happy to look into it!

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#28

Post by lcswi »

retSava wrote: post #22

Slightly off-topic, pardon me. But, does someone have any good tips on how to remove pdf security? Let me clarify why; I frequently come across datasheets (eg to flash memory ICs) that have security enabled for some strange reason. Nothing secret, just plainly downloaded from the Internet. I can open, and print, but not highlight or add remarks. Existing solutions I've found so far are inadequate since they typically…

Use a reader like evince and toggle the checkbox.

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#29

Post by metachris »

chriswarbo wrote: post #18

On a related note, these past couple of weeks I've found myself wanting to import several years' worth of accumulated PDFs into a BibTeX file. This has involved metadata extraction, text scraping, querying Google Scholar (good, but rate-limited) and CrossRef (no limit, but not as accurate). I've written a very rough guide to the approaches I've taken so far at http://chriswarbo.net/essays/pdf-tools.html , with a bunc…

Thanks, interesting read.

Re: Show HN: PDFx – Extract Metadata and URLs from PDFs, and Download Referenced PDFs

#30

Post by retSava »

lcswi wrote: post #28
retSava wrote: post #22

Slightly off-topic, pardon me. But, does someone have any good tips on how to remove pdf security? Let me clarify why; I frequently come across datasheets (eg to flash memory ICs) that have security enabled for some strange reason. Nothing secret, just plainly downloaded from the Internet. I can open, and print, but not highlight or add remarks. Existing solutions I've found so far are inadequate since they typically…

Use a reader like evince and toggle the checkbox.

Nice! Thanks for that! If you only knew how much I've looked for something like this, and, not to brag or anything, my internet-search-skills are prettttttty sharp.

For others information (from wikipedia): "Evince used to obey the DRM restrictions of PDF files, which may prevent copying, printing, or converting some PDF files, however this has been made optional, and turned off by default[...]"

Post reply on HN