61–70 of 97 posts
What is ".txt format"? I'm genuinely curious what they went with here, but don't want to download 36 GB to find out. There isn't a .txt standard, is there? I would hope for UTF-8, but given the old-school, Windows-y .txt extension, it could just as easily be Windows-1252 or something. LF or CRLF line endings?
It looks like they are unix line endings, UTF-8, and markdown formatted.
Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use. With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title? Also, what's in that dmca.mp4 file? All I hear is music.
I'm gonna take you at your word on blindness and help out here. There are a small assembly women in a semi-circle on stage, singing with stearn and serious faces what is an aria like song piece, all while vigorously gesturing what can universally be described male masterbation. The camera pans into their faces, and back out to reveal the small group of women to be a chorus of fapping.
This is very cool and useful, thanks. However, saying: "Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data. Now you do. Now everyone does" Is quite a stretch. You need much, much more that 36GB of quality data to train a world-class GPT model.
No, You don't need more. This is compressed text. All of English Wikipedia is roughly 20 GB. You can train amazing and state of the art text models with this data in addition. Openai states themselves that they trained on "40GB of internet text" https://openai.com/blog/better-language-models/
Your link is for GPT-2, GPT-3 used much, much more data.
GPT-3 was trained in part on "books2" corpus which is not public but seems to basically be the same thing as this: 200k books * 100k words per book on average * ~3 token per words = 60B tokens, books2 is 55B tokens so it checks out.
The total amount of tokens that GPT-3 was trained on from all sources is a combined 500B tokens, this is merely 10% of what they have.
Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use. With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title? Also, what's in that dmca.mp4 file? All I hear is music.
I spent four or five days intensely working on the script to convert an .epub to .txt. It's a surprisingly hard problem, so I tried to be correct in every detail. It was important to me that an AI learn the way humans learn.
Therefore, yes, I have found that these books are quite readable. However there are some unique challenges due to the nature of it being .txt, which I would love to help you navigate.
Firstly, the top and the bottom of each file is usually "repetitive". By that I mean, there tends to be a lot of metadata that might be hard to skip.
However, the neat thing is, these aren't actually .txt files -- it's Markdown. And we can take advantage of the markdown format to help you out.
Here is a direct link to "A Mathematician's Apology": https://gist.githubusercontent.com/shawwn/1b399325e866731165...
I like this file as an example, because it illustrates both problems that will be challenging for you. Firstly, the book itself doesn't begin until halfway down the file. There is a ridiculously long "forward" section, written by someone else. Normally this would be easy to skip.
The cheat code I use to read this book is to search for "# 1". Double quote, followed by pound sign (or "hashtag" as the kids say nowadays, ha), followed by a space, then the number one.
That brings you straight to chapter one.
However, not all books use that format for chapter one. So I would recommend searching for a regex, if you can: "beginning of line" followed by pound sign.
That'll usually do the trick. And yes, I find all of these to be quite readable, which I was proud of.
The code is courtesy of Aaron Swartz, by the way. I merely refined it. I am so proud of him for what he was able to achieve during his lifetime. This is "html-to-text" with some modifications.
If you want to convert .epub files to .txt for your own use, the script is here: https://github.com/shawwn/scrap/blob/master/epub2txt-all
It has a rather cryptic name of "epub2txt-all". Sorry about that. But the script itself should be just a matter of running it.
Here are some notes on what I did with this script, in case you find it helpful: https://github.com/soskek/bookcorpus/issues/27
Happy reading! Please let me know any other questions you might have.
By the way, dmca.mp4 is a joke video. Sort of. It's a good question of what is actually going on there. As far as I can tell, it's a dozen professionals who have gathered together to simulate jacking themselves off while singing "ahhhhhhHHhhH" for ten minutes. I don't really know anything beyond that, but the-eye.eu community seems rather proud of it.
This is very cool and useful, thanks. However, saying: "Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data. Now you do. Now everyone does" Is quite a stretch. You need much, much more that 36GB of quality data to train a world-class GPT model.
AI models suffer massive diminishing returns on both model size and amount of input text, which fortunately gives fresh models a chance to compete (the benefits of GPT-2/Transformer architectures is that they are capable of scaling indefinitely, but they still have diminishing returns)
This is very cool and useful, thanks. However, saying: "Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data. Now you do. Now everyone does" Is quite a stretch. You need much, much more that 36GB of quality data to train a world-class GPT model.
The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When you look at power burned doing it, the gap is even wider. Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.
That's not a fair comparison. The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution.
Earlier quoted context omitted.
The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When you look at power burned doing it, the gap is even wider. Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.
GPT and most modern "AI" approaches suffer from the same problem: they're just brute force algorithmic approaches that don't mimic "intelligence" in any way. Since there is no understanding, they can't learn in the same way humans or other living creatures do (in general terms, by extrapolation). The ironic thing is that ML became popular because we didn't have the technology back then to properly model actual neuron…
Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use. With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title? Also, what's in that dmca.mp4 file? All I hear is music.
- 0/0.4 - Mike Lancaster.epub.txt
- 0/0.721 - Gary Webster.epub.txt
- 0/01 - Alec Dunn.epub.txt
- 0/01 Kai_ Ninja Of Fire (Scholastic) - Greg Farshtey (retail).epub.txt
- 0/02 Crescendo - Becca Fitzpatrick.epub.txt
- 0/03 Cole_ Ninja Of Earth (Scholastic) - Greg Farshtey (retail).epub.txt
- 0/03 - Jean-Christophe Valtat.epub.txt
- 0/04. R. W. Peake - Antony and Cleopatra Part I Antony (Marching With Caesar, Book 4) [Retail].epub.txt
- 0/05. R. W. Peake - Antony and Cleopatra Part II Cleopatra (Marching With Caesar, Book 5) [Retail].epub.txt
- 0/05 LEGO Ninjago - Snake Attack! (Scholastic) - Tracey West (retail).epub.txt
Second edit - Books3 is much cleaner than Books1, I am editing my negative opinion to a more positive one to reflect that.
This is very cool and useful, thanks. However, saying: "Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data. Now you do. Now everyone does" Is quite a stretch. You need much, much more that 36GB of quality data to train a world-class GPT model.
Earlier quoted context omitted.
The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When you look at power burned doing it, the gap is even wider. Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.
> with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. That's not a fair comparison. The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution.