Live data from Hacker News

Show HN: Parsing horse racing charts with Apache PDFBox

github.com

11–20 of 32 posts

Re: Show HN: Parsing horse racing charts with Apache PDFBox

#11
post #10

I still don't understand how PDF could become one of the standards for publishing documents. Well structured content gets converted into PDF which loses most of that structure. And then a lot of work is done to guess that structure from PDF and convert it back to a better file format. It just shows that successful solutions don't have to be technically good.

Sadly I think that often the publishers actually want it that way, i.e. the they do not want the data to be easily parsable...

Re: Show HN: Parsing horse racing charts with Apache PDFBox

#12
post #9

Very interesting! I had never heard of Apache PDFBox before, I must give it a try. I have a similar program that parses horse racing PDFs from sites such as www.racehorserunner.com - which are of a much simpler format, but cause endless problems for me when the PDFs have layout problems. For example, issues like one column being too long and overlapping with another, e.g the last race on http://www.racehorserunner.co…

Huh, interesting. I was looking around for PDF libs previously and PDFBox didn't show up in google results. pdftk was the only one that showed up in Google results anywhere useful.

Edit: Looks like it's on the second page of results and I never made it that far, heh. Goes to show how biasing the first page of results is.

Re: Show HN: Parsing horse racing charts with Apache PDFBox

#13
post #11
post #10

I still don't understand how PDF could become one of the standards for publishing documents. Well structured content gets converted into PDF which loses most of that structure. And then a lot of work is done to guess that structure from PDF and convert it back to a better file format. It just shows that successful solutions don't have to be technically good.

Sadly I think that often the publishers actually want it that way, i.e. the they do not want the data to be easily parsable...

I think it's more that they want consistency in rendering across devices and media.

Re: Show HN: Parsing horse racing charts with Apache PDFBox

#15
post #13
post #11

Earlier quoted context omitted.

Sadly I think that often the publishers actually want it that way, i.e. the they do not want the data to be easily parsable...

I think it's more that they want consistency in rendering across devices and media.

For legal documents (where PDF was used first as far as I know) this may make sense but for manuals and other documents it makes no sense at all.

Re: Show HN: Parsing horse racing charts with Apache PDFBox

#18
post #10

I still don't understand how PDF could become one of the standards for publishing documents. Well structured content gets converted into PDF which loses most of that structure. And then a lot of work is done to guess that structure from PDF and convert it back to a better file format. It just shows that successful solutions don't have to be technically good.

The keyword is "publishing" --- as in, producing human-readable physical copies, not electronic ones. It just so happens that the format was relatively suitable for the latter too (because it actually looks like a printed document rendered on the screen --- unlike HTML or other formats around at the time), which is why that use-case became popular. PDF is basically a descendant of PostScript, which was designed to control printers.

(Its PostScript origins may also explain the bizarre mix of text and binary that constitute the file format. For example, page contents are in a relatively free-form PostScript-ish RPN-like textual language, but are found in "content streams" which may be compressed or encoded into a binary format. Data "object" structures include things like 'Then there are things like the cross-reference table of all objects in the file, which is an array of fixed-width textual numbers representing file offsets, e.g. "0000001056 00000 n" refers to something 1056 bytes from the start of the file. Reactions of WTF!? from those working with the format for the first time are not uncommon.)

Re: Show HN: Parsing horse racing charts with Apache PDFBox

#19
post #5

Earlier quoted context omitted.

PDF is used primarily for pre-paginated media and does not reflow text; if the PDF author wanted the pages in landscape orientation, or using some other paper dimensions, he would have specified that. Same goes for margins and the like.

Doesn't mean the author isn't wrong for my reading situation... Or isn't doing it just cause everyone else is.

I think what the parent is trying to say is that PDF is not like HTML or text or other formats where the viewer is primarily responsible for a lot of the formatting --- in fact, a PDF page contains not much more than primitive instructions of the form "move to X, Y"; "set font to F"; "draw text 'Some text here'" (some pathological cases issue individual moves and draws for each character) --- so expecting all PDF viewers to be able to somehow "reverse-engineer" or "decompile" that set of low-level drawing instructions into more semantic entities like lines of text or even words in order to reformat the text is a little too much.

Anyone who has tried selecting text from a two-column PDF page will also quickly realise the nature of the problem.

Re: Show HN: Parsing horse racing charts with Apache PDFBox

#20

Impressive! Seems like you can't just use PDFBox out of the box (no pun intended) and need to write some custom code specific to the PDF itself per the chart-parser commits[1] [1] https://github.com/robinhowlett/chart-parser/tree/master/src...

Author here; well, PDFBox is good for simple text stripping. If I wanted to print all the text on the PDF, that would be very straightforward and not much code. However, the PDF chart here is in essence a representation of structured data. I wanted to get the content in that format so that I could both serialize to JSON plus have an SDK to boot.
Post reply on HN