Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Case for a Git-Powered Project Gutenberg

neosmart.net

Post reply on HN
Print view View original

11–20 of 47 posts

Re: The Case for a Git-Powered Project Gutenberg

#11

Post by felix_faber »

Project Gutenberg is a complete mess.

It took them years of discussion and they haven't yet chosen a master file format for their books.[1] As a consequence, they are hosting thousands of books in many different file-formats provided by distributed proofreaders[2] and some other sources. But since no master-format exists and conversion is .. well .. buggy, they have no leverage on their quality problems. The textual content may be fine, but the reading experience for the casual user shows how big the iceberg is under the surface.

Having seen endless flamewars[3] and several unsuccessful approaches to their formatting and complexity problems, I propose clear reset.

What about a nice fork with only a few important books that get proper treatment in regard to

- master file format

- conversion (mainly html, pdf and ereader formats)

- design and page layout (e.g. for pdf versions)

From then on, one could build a growing git repository of _nice_ books and build an infrastructure around that. Tackling their current mess directly (and taking on the burden of their internal politics and historical toolchain) will be completely in vain, as the past has shown.

I don't have much time to spare. But for a really good cause (and the future of bookreading is a really good cause), I am willing to invest.

Who is in?

[1] This one is the latest candidate: http://www.gutenberg.org/wiki/Gutenberg:RST Before that, there was one guy working on a simplified TEI format. But that never took off.

[2] http://www.pgdp.net/c/

[3] The home of many-a-flamewar: http://blog.gmane.org/gmane.culture.literature.e-books.guten...

Re: The Case for a Git-Powered Project Gutenberg

#12

Post by wladimir »

ComputerGuru wrote: post #3
zdw wrote: post #2

Version control is superior than other "sync" solutions when the data is: - Modified mainly by humans, at human speeds - Mistakes can be made and need reverted atomically - Data integrity and chains of changes need to be preserved I agree with the article that PG is a great example of where version control could work with great effect. The downside is that teaching non-techies how to make it work is difficult, and th…

I'm the author of TFA and I must completely agree with you on this point. Is there a human-friendly VCS out there? I don't care how featured or not it is, as Git can be configured to pull from it (a la git-svn). Just anything that non-tech-related persons (or even just non-coders!) can use without wanting to pull their hair out and feeling very stupid? For example, one of my software products has translations powered…

Depends on what features you need. I've seen wikis used as "human-friendly" VCSes. There is no branching, forking and automated merging in most implementations... But that might be good things, based on who your target group is.

Re: The Case for a Git-Powered Project Gutenberg

#13

Post by ComputerGuru »

felix_faber wrote: post #11

Project Gutenberg is a complete mess. It took them years of discussion and they haven't yet chosen a master file format for their books.[1] As a consequence, they are hosting thousands of books in many different file-formats provided by distributed proofreaders[2] and some other sources. But since no master-format exists and conversion is .. well .. buggy, they have no leverage on their quality problems. The textual…

Any reboot of PG (if any at all) should from the ground up have a single master format for any given title, these master formats under DVCS, and an integrated automated build system for conversion to auxiliary formats. These are the very basic components of any large scale library being designed in this day and age, and ensure integrity, posterity, and availability.

Re: The Case for a Git-Powered Project Gutenberg

#14

Post by donatzsky »

ComputerGuru wrote: post #3
zdw wrote: post #2

Version control is superior than other "sync" solutions when the data is: - Modified mainly by humans, at human speeds - Mistakes can be made and need reverted atomically - Data integrity and chains of changes need to be preserved I agree with the article that PG is a great example of where version control could work with great effect. The downside is that teaching non-techies how to make it work is difficult, and th…

I'm the author of TFA and I must completely agree with you on this point. Is there a human-friendly VCS out there? I don't care how featured or not it is, as Git can be configured to pull from it (a la git-svn). Just anything that non-tech-related persons (or even just non-coders!) can use without wanting to pull their hair out and feeling very stupid? For example, one of my software products has translations powered…

Mercurial (Hg) with TortoiseHg might be a good option. You could even create a pre-configured PG version, to make it even easier for novices. Along with a guide on the basics, I think that could end up working quite well. Hg also has the benefit, compared to Git, of being strictly cross-platform.

Re: The Case for a Git-Powered Project Gutenberg

#15

Post by SoftwareMaven »

felix_faber wrote: post #11

Project Gutenberg is a complete mess. It took them years of discussion and they haven't yet chosen a master file format for their books.[1] As a consequence, they are hosting thousands of books in many different file-formats provided by distributed proofreaders[2] and some other sources. But since no master-format exists and conversion is .. well .. buggy, they have no leverage on their quality problems. The textual…

I would be interested. PG is amazing wrt the content, but the experience of actually reading the books is awful. My email is in my profile.

Re: The Case for a Git-Powered Project Gutenberg

#16

Post by danking00 »

SoftwareMaven wrote: post #15
felix_faber wrote: post #11

Project Gutenberg is a complete mess. It took them years of discussion and they haven't yet chosen a master file format for their books.[1] As a consequence, they are hosting thousands of books in many different file-formats provided by distributed proofreaders[2] and some other sources. But since no master-format exists and conversion is .. well .. buggy, they have no leverage on their quality problems. The textual…

I would be interested. PG is amazing wrt the content, but the experience of actually reading the books is awful. My email is in my profile.

Agreed. Let's do something about our shared cultural history. Email is also in my profile.

Re: The Case for a Git-Powered Project Gutenberg

#17

Post by pronoiac »

I'm not sure git's the best choice for handling multiple gigabytes of data. That estimate's based on their DVD releases. Oddly, I'm not positive that they're compressed.

I've seen a proposal for using Usenet (or something like it) for Wikipedia; perhaps that would be appropriate here? Surfacing changes would be easier. And if you didn't want to follow each and every format available, you could. (Having every format in the same silo seems painfully redundant to me.)

Re: The Case for a Git-Powered Project Gutenberg

#18

Post by DanBC »

felix_faber wrote: post #11

Project Gutenberg is a complete mess. It took them years of discussion and they haven't yet chosen a master file format for their books.[1] As a consequence, they are hosting thousands of books in many different file-formats provided by distributed proofreaders[2] and some other sources. But since no master-format exists and conversion is .. well .. buggy, they have no leverage on their quality problems. The textual…

A decent master file system is good. Including conversion is sub-optimal. Have a sister project that creates and maintains good software for conversion.

That would draw 2 crowds; people wanting to help PG and people unwilling to continue using Calibre.

Re: The Case for a Git-Powered Project Gutenberg

#19

Post by haberman »

CPDL (the Choral Public Domain Library), which is like Project Gutenberg except for Choral Music, has exactly the same problem. I can't tell you how many scores I've used from CPDL that have obvious errors in them that we end up fixing in rehearsal. I have no way of contributing back these fixes except to send a message to the score's author which is not worth the hassle so I don't do it.

Re: The Case for a Git-Powered Project Gutenberg

#20

Post by ComputerGuru »

pronoiac wrote: post #17

I'm not sure git's the best choice for handling multiple gigabytes of data. That estimate's based on their DVD releases. Oddly, I'm not positive that they're compressed. I've seen a proposal for using Usenet (or something like it) for Wikipedia; perhaps that would be appropriate here? Surfacing changes would be easier. And if you didn't want to follow each and every format available, you could. (Having every format i…

You wouldn't place the "binary" output in the repository, just the whatever source format you're using to compile the rest. Say (for the sake of the example) LaTeX or, heck, plain UTF-8 text. Only that is stored in the git repo, the rest are generated on the fly with a continuous build system.

I'm not sure how big that would be, but it's not going to be a lot. Currently, they have 38,000 titles so that's 38,000 files of, I dunno, 50,000 words on average? Probably less, as a lot of the PG works are rather short (novellas and short stories), though some are also equally monstrously long.

Post reply on HN