Show HN: Gutenberg – A simple interface to the Project Gutenberg corpus • Hacker News Board

sethish wrote: post #3

This is fantastic! I just made a github repo for each Gutenber book: https://github.com/GITenberg This will be very helpful, the XML/RDF files are a hassle.

Are the github repos intended to collect errata? Do you know of a database which has metadata for all the Gutenberg books?

Re: Show HN: Gutenberg – A simple interface to the Project Gutenberg corpus

#5

Post by voltagex_ » Mon, Aug 04, 2014, 6:45 AM UTC

sethish wrote: post #3

This is fantastic! I just made a github repo for each Gutenber book: https://github.com/GITenberg This will be very helpful, the XML/RDF files are a hassle.

Why is Git in uppercase?

Re: Show HN: Gutenberg – A simple interface to the Project Gutenberg corpus

#6

Post by c-w » Mon, Aug 04, 2014, 11:09 AM UTC

walterbell wrote: post #4

sethish wrote: post #3

This is fantastic! I just made a github repo for each Gutenber book: https://github.com/GITenberg This will be very helpful, the XML/RDF files are a hassle.

Are the github repos intended to collect errata? Do you know of a database which has metadata for all the Gutenberg books?

There's a database of RDF files that describe the books (http://www.gutenberg.org/cache/epub/feeds/rdf-files.tar.bz2), but its a bit of a pain to use and doesn't link the books back to the API that should be used for crawling Project Gutenberg (http://www.gutenberg.org/robot/harvest).

Re: Show HN: Gutenberg – A simple interface to the Project Gutenberg corpus

#7

Post by c-w » Mon, Aug 04, 2014, 11:20 AM UTC

Hey all, OP here.

I built this because I think that Project Gutenberg is a great resource for NLP (e.g. stylometry, tracking writing styles over time, authorship detection, ...) - I wanted to use the data on Project Gutenberg a number of times in the past but always ended up using another corpus because there wasn't an easy way to access the Project Gutenberg data. Hopefully this library fixes that.

The project currently is "works on my machine" quality, so please do report any bugs you stumble across.

Also, if you can think of any use-cases for the Project Gutenberg data that aren't easily doable using the functionality that is currently available in the library, please let me know (e.g. by filing a ticket on the Bitbucket repo).

Re: Show HN: Gutenberg – A simple interface to the Project Gutenberg corpus

#8

Post by sethish » Mon, Aug 04, 2014, 11:48 AM UTC

walterbell wrote: post #4

sethish wrote: post #3

This is fantastic! I just made a github repo for each Gutenber book: https://github.com/GITenberg This will be very helpful, the XML/RDF files are a hassle.

Are the github repos intended to collect errata? Do you know of a database which has metadata for all the Gutenberg books?

The github repos are intended to collect issues and received pull requests. Project Gutenberg doesn't have a public bugtracker, nor do they use version control.

Re: Show HN: Gutenberg – A simple interface to the Project Gutenberg corpus

#9

Post by sethish » Mon, Aug 04, 2014, 11:55 AM UTC

c-w wrote: post #6

walterbell wrote: post #4

Earlier quoted context omitted.

Are the github repos intended to collect errata? Do you know of a database which has metadata for all the Gutenberg books?

There's a database of RDF files that describe the books ( http://www.gutenberg.org/cache/epub/feeds/rdf-files.tar.bz2 ), but its a bit of a pain to use and doesn't link the books back to the API that should be used for crawling Project Gutenberg ( http://www.gutenberg.org/robot/harvest ).

I think the previous version of the metadata included a path to the ftp server. Splitting the book id (4443 -> 4/4/4/4443) works for _most_ books, but there were somewhere between 800 and 3000 books organized in a different folder structure that I still need to track down.

Re: Show HN: Gutenberg – A simple interface to the Project Gutenberg corpus

#10

Post by sethish » Mon, Aug 04, 2014, 11:56 AM UTC

voltagex_ wrote: post #5

sethish wrote: post #3

This is fantastic! I just made a github repo for each Gutenber book: https://github.com/GITenberg This will be very helpful, the XML/RDF files are a hassle.

Why is Git in uppercase?

To better differentiate from Project Gutenberg, GIT + Gutenberg. Case is ambigious on github, so I can change it later without breaking anyone's URLs.