Always impressive how the AI community, both inside large companies and not, seems to assume data just exists for them to use, copyright or personal rights be damned.
You just described Google.
Books in .txt format for AI training purposes
41–50 of 97 posts
Re: Books in .txt format for AI training purposes
#42The entirety of Library Genesis (about 2.7 million books, fairly poorly curated) can also be downloaded, its somewhere around 40 TB altogether of significantly more recent books.
In plain text format though?
AFAIK, most is in PDF, EPUB or mobi format. I'd presume it's not too difficult to extract text from the latter two, but extracting text from PDFs is far from simple, and something you get working for 1 PDF won't necessarily work on another.
Re: Books in .txt format for AI training purposes
#43Always impressive how the AI community, both inside large companies and not, seems to assume data just exists for them to use, copyright or personal rights be damned.
Aside: it's interesting to think about the legal ramifications of copyrights in the face of AI. If my AI model "reads" a bunch of text, is the model now a "derived work" of the copyrighted work? Perhaps. But of course humans don't infringe when they watch/listen/read a copyrighted work and it changes their state of mind. Perhaps the model can be said not to infringe if it sufficiently abstracts concepts of the art in…
It is derivative. There’s then a question of whether the derived work is sufficiently transformative to be fair use, which depends on what the model is outputting.
But yeah that’s separate from the question of whether you properly licensed the data to train on in the first place.
A big chunk of the computing community seems to approach licensing as “I can see it, so I can use it.” (See GPL code used where it shouldn’t be.)
Re: Books in .txt format for AI training purposes
#44There is always Project Gutenberg [1]. I suppose there may be pros and cons to each, with Project Gutenberg being generally older books. [1]. https://www.gutenberg.org/ Fun trivia: I once took an NLP class and discovered that I could predict whether a text was written by Edgar Allan Poe or H. G. Wells with ~86% accuracy based on the presence of one word: "whereupon". The language had changed enough in 60 years such t…
I’ve noticed that “presently” (in the meaning “soon”) is one such indicator for post-WWII English. British novels written in the 1950s consistently use “presently” to indicate a short passage of time. In the following decades it seems to have been removed from editors’ style guides and replaced mostly by plain old “soon”.
Wonder what other words that could be found that do something similar.
Re: Books in .txt format for AI training purposes
#45Always impressive how the AI community, both inside large companies and not, seems to assume data just exists for them to use, copyright or personal rights be damned.
Another related aspect of this is the hiQ vs. LinkedIn decision, which many interpreted as now setting precedent for allowing scraping of public information.
It's not that simple unfortunately, and companies could still send a C&D/threaten invoking the CFAA, and it would still be a mess. Although in practice it's not worth the effort for companies to sue as long as the scraper is not monetarily benefiting from it (which is how the court case happened).
Re: Books in .txt format for AI training purposes
#46This is very cool and useful, thanks. However, saying: "Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data. Now you do. Now everyone does" Is quite a stretch. You need much, much more that 36GB of quality data to train a world-class GPT model.
AI models suffer massive diminishing returns on both model size and amount of input text, which fortunately gives fresh models a chance to compete (the benefits of GPT-2/Transformer architectures is that they are capable of scaling indefinitely, but they still have diminishing returns)
Re: Books in .txt format for AI training purposes
#47I’ve tried some sci-fi nonsense on GPT-2 in the past and it returned flattened out phpBB replies, complete with usernames and headers. So whatever they are trained with must include web crawls.
GPT-2 was trained on web links linked to from Reddit which would explain your output.
GPT-3 was trained on the Common Crawl + books.
Re: Books in .txt format for AI training purposes
#48Always impressive how the AI community, both inside large companies and not, seems to assume data just exists for them to use, copyright or personal rights be damned.
> Always impressive how the AI community, both inside large companies and not, seems to assume data just exists for them to use, copyright or personal rights be damned. Are you objecting to the use of data for training, or to the compilation of training datasets?
[deleted]
Re: Books in .txt format for AI training purposes
#49Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use.
With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title?
Also, what's in that dmca.mp4 file? All I hear is music.
Re: Books in .txt format for AI training purposes
#50Earlier quoted context omitted.
The question is..if a AI reads a book is it against copyright? Or is the trained model a derived work of those books?
The question is, "Do the creators of bibliotik have proper licenses to aggregate those books into a dataset?" The dataset is certainly a derived work. Edit: https://the-eye.eu/public/Books/humble_books_20180509/ Cue the "they're depriving me of my income" complaints from authors.
[deleted]