Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Books in .txt format for AI training purposes

twitter.com

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Books in .txt format for AI training purposes

#81

Post by sailingparrot »

sillysaurusx wrote: post #79
sailingparrot wrote: post #77

Earlier quoted context omitted.

And again, I find books3 extremely cool and important work on your part, looking forward to the rest. I just have a minor gripe with saying that "now we can train world class GPT model" thanks to that, as you said it's just one piece, and as a typicial HNist I had to point it out :).

Believe it or not, I appreciate and relate to that sentiment. But after spending roughly one year acquiring knowledge related to this work, I feel I can say with a fairly high degree of certainty that this dataset alone is enough to train a model that will achieve "world class" status in some area. Writing books, perhaps. Which part of my logic do you feel is mistaken, and why? I am actually quite interested to hear…

I don't think you are mistaken, I guess it's just that there is just so much information you can convey in a tweet, when I read "world class GPT model" I understand a model that will beat (or equal) on general NLG, which is not what you meant it seems.

Re: Books in .txt format for AI training purposes

#82

Post by PeterisP »

yodon wrote: post #5
webmaven wrote: post #3

Earlier quoted context omitted.

> Always impressive how the AI community, both inside large companies and not, seems to assume data just exists for them to use, copyright or personal rights be damned. Are you objecting to the use of data for training, or to the compilation of training datasets?

> Are you objecting to the use of data for training, or to the compilation of training datasets? The GP is objecting to the casualness with which privacy rights and intellectual property rights are ignored by so many in the AI community, not to a choice of whether they object to one or another specific manifestation of how the community is doing so. [edit added]: Those who would argue "no, there is no casual ignoring…

IMHO the research community in general is definitely not ignoring intellectual property rights - for natural language processing, there are papers and publication tracks in major conferences specifically about the legal issues of corpora; many researchers are quite painfully aware of all the various legal restrictions that exist.

However, the legal environment is very different than in the commercial world or consumer piracy, as it generally involves various legal exemptions (differing between locales) that do allow such usage. For example, I work in NLP research on aspects that involve handling large corpora of copyrighted text. It's easier to do it with cooperation of the publishers for various practical reasons, however, we still can and do use also the works of the publishers who would refuse to grant any permission, because local copyright law has specific exceptions that allows the usage these works for noncommercial research purposes. Doing so is not ignoring their rights, their rights are not violated but rather they are limited; their exclusivity right (privilege would be a more appropriate word) to make copies is not absolute. There are even some countries with explicit legal duty for the publishers to provide digital versions of their works to national corpora where they will be used for (among other things) machine learning models.

The specific consequence, however, is that we can't legally share the full datasets which we are using with the public, like it was done in the original post with this particular dataset, as that would be a violation of the publishers' commercial rights; we can provide them to specific researchers for limited noncommercial purposes only. But I can download this dataset or one like this and use it my research legally; just as I can rip up a physical book, scan it, make a digital copy and OCR it, and use it in a research corpus (with copies distributed to other researchers) even if the publisher disapproves.

Re: Books in .txt format for AI training purposes

#83

Post by Shared404 »

pixl97 wrote: post #74
detaro wrote: post #2

Always impressive how the AI community, both inside large companies and not, seems to assume data just exists for them to use, copyright or personal rights be damned.

Eh, this entire comment thread is leading into "RMS was right" territory. Copyright and our machine intelligence future can't mix well at all.

To be fair, I'm pretty sure that's a pretty common sentiment around HN.

Re: Books in .txt format for AI training purposes

#84

Post by nialv7 »

gamblor956 wrote: post #71
nialv7 wrote: post #67

Earlier quoted context omitted.

That's a bold claim to make, given we ourselves don't even understand what understanding, or intelligence is. So how can you support your claim that what GPT-3 does is not intelligence.

I'm not the only one. It's industry consensus. https://www.technologyreview.com/2020/08/22/1007539/gpt3-ope...

That doesn't answer the question.

Re: Books in .txt format for AI training purposes

#85

Post by sillysaurusx »

WClayFerguson wrote: post #16

Is there a torrent link? Also, I know IPFS is still new, but growing rapidly, so some day each of these books will just have a CID on IPFS, and basically to publish all the books as a list you'd simply publish a list of the CIDs.

No torrent link yet, partly because I trust the-eye. They are fearless in a way that I haven't seen elsewhere.

Time will tell if fear would have been wise. But for now, I'm curious to see whether "the link will last for years" is true.

If it goes down, I'll append instructions to the original twitter thread on how to access it elsewhere. I also don't have any experience setting up the alternatives you mention; if you do, please feel free to mention it somewhere (twitter DM is always a reliable way to reach me) and I'll highlight it.

As for a torrent link specifically, I did request that the-eye have a torrent ready to go on day one, for just such an eventuality. I was confused when they didn't seem worried. After spending almost an hour describing in detail the kinds of repercussions that might inevitably follow, he simply said that fear does not control him the way it controls me, and that he will ensure it remains. (Probably via torrent, if such a thing becomes necessary.)

I've thought a lot about what he said, and how he phrased it. I decided to trust them to take care of the data. Mostly, though, I made the decision out of intellectual curiosity to see how true their ambition really is.

In the meantime, joining their community on Discord and/or donating to them would be helpful (though to my surprise, they didn't seem too interested in monetary concerns either). http://the-eye.eu/

Re: Books in .txt format for AI training purposes

#86

Post by gwern »

ralph87 wrote: post #40

The entirety of Library Genesis (about 2.7 million books, fairly poorly curated) can also be downloaded, its somewhere around 40 TB altogether of significantly more recent books.

The problem with LibGen, and why EleutherAI has been avoiding it, is that most of it is PDFs and most of the PDFs are scans; OCR layers are typically incredibly crummy. Even if you re-OCR them all with Tesseract or something (which will take quite a while, based on how long Tesseract takes to OCR my books even using parallelism on a Threadripper), the OCR will still be awful data. Do you really want to add that to your training dataset...? Far from obvious, and not when there are so many other pools of text like Arxiv which aren't so insoluble.

Re: Books in .txt format for AI training purposes

#87

Post by naniwaduni »

wyldfire wrote: post #19
detaro wrote: post #2

Always impressive how the AI community, both inside large companies and not, seems to assume data just exists for them to use, copyright or personal rights be damned.

Aside: it's interesting to think about the legal ramifications of copyrights in the face of AI. If my AI model "reads" a bunch of text, is the model now a "derived work" of the copyrighted work? Perhaps. But of course humans don't infringe when they watch/listen/read a copyrighted work and it changes their state of mind. Perhaps the model can be said not to infringe if it sufficiently abstracts concepts of the art in…

> But of course humans don't infringe when they watch/listen/read a copyrighted work and it changes their state of mind.

You're not generally distributing, performing, &c. your state of mind. On the other hand, if you then go and generate work based on what you read, then distributing that work certainly can be infringing. Thus "clean room" techniques, where one team reads copyrighted text, writes up a spec (which may then be checked off by lawyers), and then another team, without reading the copyrighted text, implements based off the spec, are sometimes used to attempt to launder copyright taint.

Taking a step back, the fact that well-known authors have infamously declared that they don't read fanfiction as a CYA move against accusations of plagiarism suggests that yes, brains that have read text are widely considered derivative.

Re: Books in .txt format for AI training purposes

#88

Post by fakedang »

naniwaduni wrote: post #87
wyldfire wrote: post #19

Earlier quoted context omitted.

Aside: it's interesting to think about the legal ramifications of copyrights in the face of AI. If my AI model "reads" a bunch of text, is the model now a "derived work" of the copyrighted work? Perhaps. But of course humans don't infringe when they watch/listen/read a copyrighted work and it changes their state of mind. Perhaps the model can be said not to infringe if it sufficiently abstracts concepts of the art in…

> But of course humans don't infringe when they watch/listen/read a copyrighted work and it changes their state of mind. You're not generally distributing, performing, &c. your state of mind. On the other hand, if you then go and generate work based on what you read, then distributing that work certainly can be infringing. Thus "clean room" techniques, where one team reads copyrighted text, writes up a spec (which ma…

Distributing an exact copy or paraphrased content of an original work would be subject to copyright, but if a model just generates its own inferences, I don't see how it's different from say a human generated review.

Re: Books in .txt format for AI training purposes

#89

Post by fakedang »

lacker wrote: post #25
nix23 wrote: post #6

Earlier quoted context omitted.

The question is..if a AI reads a book is it against copyright? Or is the trained model a derived work of those books?

It is probably not a derived work. To be a derived work it is not enough to simply use the original work in its construction; it must contain major copyrightable elements of the original work. The weights and parameters present in the model do not seem to fit that description; generally you can’t copyright a bunch of numbers. That said, I don’t think this has been confirmed by a major court case yet, and I wouldn’t b…

But for a dynamically learning machine model, can you even copyright the numbers?
Post reply on HN