Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

github.com

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

#41

Post by halomru »

scrollaway wrote: post #27
nemild wrote: post #16

For those interested in converting PDF tables into CSV, there's also Tabula ( http://tabula.technology/ ) (Used by many journalists to analyze the data in PDFs)

I find it absolutely ridiculous that we have to resort to these kinds of tools :/ We have digital formats, and we decided to standardize document distribution on the one that makes it as hard to extract data as if it were on physical paper.

PDF is a perfectly fine and rich digital format. It also allows you to do proper copy and paste, which is much saner than anything paper offers.

Sure, PDF is a light on context clues for automation and is targeted purely at humans. But formats targeted at both computers and humans consistently fail (XML with accompanying XSLT comes to mind), and/or only have terrible tools for creating files (easily parsable, pretty HTML).

Either there is very little real demand or we consistently fail at making alternatives viable.

Re: Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

#42

Post by logicallee »

yeldarb wrote: post #25
logicallee wrote: post #11

Earlier quoted context omitted.

if you really want to rake it in, serve, at static speeds (meaning instantly, I swear, boot a ramdrive (Tmpfs) and serve static html from nginx all from RAM), text versions of the top 10,000 web sites. there is so much crap on most sites. re-crawl hourly. monetize via Google adwords. EDIT: I'm not sure why I'm being downvoted. I am not suggesting serving PDF's. I am suggesting serving tiny text renders of top sites,…

Sounds cool and all but two huge problems: 1) Copyright: completely re-serving the complete content of the top 100 sites with your own ads does not fall under fair use and would almost certainly be a magnet for lawsuits. 2) Distribution: how do you find your niche of people with poor internet connections and get them to use your mirror instead of whatever site it is they want to read?

no clue on 2. for 1, you could have it be "opera mini/turbo as a service" so that you are arguing you are just shifting the viewer to the site, but it's still the user doing the viewing. it helps if you preserve any text ads on the site (or links, with alt-text, given you're probably not doing images. you could also replace images with a grainy black-and-white very low-fidelity version, this also shifts most ads on the original site, without adding hugely to your footprint.) To be honest I also thought perhaps javascript etc could be run, so that the heaviest sites of all are still downloaded and then turned into text versions. In many cases that can let someone browse a site that is otherwise incredibly slow.

This isn't legal advice, just the approach I would use off of the top of my head. I agree with you that it's hard. with the framework "opera minifier/turbofier as a service" it could work, though. Like a remote browser. (in a VM). Like, present it as "lynx as a service." (Lynx being an old terminal-based text browser.) Something like that, anyway.

Re: Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

#44

Post by TeMPOraL »

halomru wrote: post #41
scrollaway wrote: post #27

Earlier quoted context omitted.

I find it absolutely ridiculous that we have to resort to these kinds of tools :/ We have digital formats, and we decided to standardize document distribution on the one that makes it as hard to extract data as if it were on physical paper.

PDF is a perfectly fine and rich digital format. It also allows you to do proper copy and paste, which is much saner than anything paper offers. Sure, PDF is a light on context clues for automation and is targeted purely at humans. But formats targeted at both computers and humans consistently fail (XML with accompanying XSLT comes to mind), and/or only have terrible tools for creating files (easily parsable, pretty…

> It also allows you to do proper copy and paste, which is much saner than anything paper offers.

Technically it does that, yes. Rarely do I see people taking advantage of it, though; most of the times I tried to copy some text out of PDF, the result had to undergo a significant cleanup before becoming usable.

Re: Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

#45

Post by marak830 »

jlink wrote: post #43
marak830 wrote: post #3

Ahh this will be useful for my kitchen receipts. Thanks. Now I just need to roll that with an auto translator too.(I guess I have my day off project now :-) )

Happy to know it could help you. Good cooking to you!

Both I and my accountant thank you haha.

Re: Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

#46

Post by krylon »

dmoo wrote: post #26
tyingq wrote: post #7

Curious if this works better than the pdftotext utility that comes in the Debian poppler-utils package. That has a --layout option that works really well sometimes and really terrible other times. Doesn't seem to be related to document complexity either.

Also available for windows and mac at http://www.foolabs.com/xpdf/download.html

Last year, my boss gave me a task that looked simple enough at first glance - get data on how many vacation days each employee has in total, how many they have used in the current year, and how many they have left, and put that data in our SharePoint server (so people can see when filling out a vacation request if they actually have enough days left).

Most of that was fairly easy, except that the POS program that sits in the actual data only allows exporting data in one single format - PDF. Converting that PDF file to a CSV that I can feed into SharePoint was one of the nastiest things I did last year. I did manage to get it to work though, by toying around with pdftotext for a while and exploring its command line parameters.

It was a pleasure to use! It took me a while to discover the correct set of command line parameters I needed, but I got it to work! Thanks, xpdf!

Re: Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

#47

Post by ptaipale »

halomru wrote: post #41
scrollaway wrote: post #27

Earlier quoted context omitted.

I find it absolutely ridiculous that we have to resort to these kinds of tools :/ We have digital formats, and we decided to standardize document distribution on the one that makes it as hard to extract data as if it were on physical paper.

PDF is a perfectly fine and rich digital format. It also allows you to do proper copy and paste, which is much saner than anything paper offers. Sure, PDF is a light on context clues for automation and is targeted purely at humans. But formats targeted at both computers and humans consistently fail (XML with accompanying XSLT comes to mind), and/or only have terrible tools for creating files (easily parsable, pretty…

PDF is not very fine. Copy-paste from PDF very often results in complete rubbish, even when it is not deliberately prevented (which the format allows, and then you have to do OCR).

Re: Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

#48

Post by krakaukiosk »

nemild wrote: post #16

For those interested in converting PDF tables into CSV, there's also Tabula ( http://tabula.technology/ ) (Used by many journalists to analyze the data in PDFs)

Tabula is a great tool. In my experience it's the most reliable open source software for extracting tables from PDFs. We are using their underlying Tabula-Java library for some parts of https://docparser.com and are happily sponsoring their project.

Re: Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

#49

Post by jlink »

tyingq wrote: post #7

Curious if this works better than the pdftotext utility that comes in the Debian poppler-utils package. That has a --layout option that works really well sometimes and really terrible other times. Doesn't seem to be related to document complexity either.

During the development I compared my results with the ones of pdftotext utility and i obtained more or less similar results. The objective of my code was to have an equivalent tool easily embeddable in any java/android project and to learn more about apache pdfbox.

Re: Show HN: PDFLayoutTextStripper – Converts PDF to text while keeping the layout

#50

Post by halomru »

ptaipale wrote: post #47
halomru wrote: post #41

Earlier quoted context omitted.

PDF is a perfectly fine and rich digital format. It also allows you to do proper copy and paste, which is much saner than anything paper offers. Sure, PDF is a light on context clues for automation and is targeted purely at humans. But formats targeted at both computers and humans consistently fail (XML with accompanying XSLT comes to mind), and/or only have terrible tools for creating files (easily parsable, pretty…

PDF is not very fine. Copy-paste from PDF very often results in complete rubbish, even when it is not deliberately prevented (which the format allows, and then you have to do OCR).

People purposefully disallowing copy-paste isn't a problem with PDF: in other formats they would have embedded a picture, at least with PDF you get the other advantages of proper text: infinite zoom and great compression. Sadly there's also a lot of PDFs that are little more than a picture collection that looks like text, but that's hardly the file format's fault.

It really is a problem with PDF that it's too easy to get a file where copy and paste yields a different result than what's displayed. But this varies widely with the software used for creating the file (e.g. latex ligatures never work in copy-paste)

Post reply on HN