Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Books in .txt format for AI training purposes

twitter.com

Post reply on HN
Print view View original

71–80 of 97 posts

Re: Books in .txt format for AI training purposes

#71

Post by gamblor956 »

nialv7 wrote: post #67
gamblor956 wrote: post #52

Earlier quoted context omitted.

GPT and most modern "AI" approaches suffer from the same problem: they're just brute force algorithmic approaches that don't mimic "intelligence" in any way. Since there is no understanding, they can't learn in the same way humans or other living creatures do (in general terms, by extrapolation). The ironic thing is that ML became popular because we didn't have the technology back then to properly model actual neuron…

That's a bold claim to make, given we ourselves don't even understand what understanding, or intelligence is. So how can you support your claim that what GPT-3 does is not intelligence.

I'm not the only one. It's industry consensus.

https://www.technologyreview.com/2020/08/22/1007539/gpt3-ope...

Re: Books in .txt format for AI training purposes

#72

Post by curtainsforus »

corobo wrote: post #38
wyldfire wrote: post #19

Earlier quoted context omitted.

Aside: it's interesting to think about the legal ramifications of copyrights in the face of AI. If my AI model "reads" a bunch of text, is the model now a "derived work" of the copyrighted work? Perhaps. But of course humans don't infringe when they watch/listen/read a copyrighted work and it changes their state of mind. Perhaps the model can be said not to infringe if it sufficiently abstracts concepts of the art in…

I'd imagine one day this will be determined by whether you can copy the resulting 'brain' to another system It'll work well until we can upload ourselves to the cloud and we'll have to revisit it

As technical civilization gains power, it becomes able to make things inbetween social categories. Is a simulation of a human brain a person? And one compressed to half-size? And compressed again? And again?

It's a grains-to-heap problem; eventually, you have to make an ugly choice.

Re: Books in .txt format for AI training purposes

#73

Post by sailingparrot »

asutekku wrote: post #69
sailingparrot wrote: post #7

This is very cool and useful, thanks. However, saying: "Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data. Now you do. Now everyone does" Is quite a stretch. You need much, much more that 36GB of quality data to train a world-class GPT model.

36gb of compressed text is a lot though.

That's about 50B tokens. It can be a lot depending on what you want to do, but It's 10% of what GPT-3 (the definition of world class GPT model) used.

Re: Books in .txt format for AI training purposes

#74

Post by pixl97 »

detaro wrote: post #2

Always impressive how the AI community, both inside large companies and not, seems to assume data just exists for them to use, copyright or personal rights be damned.

Eh, this entire comment thread is leading into "RMS was right" territory. Copyright and our machine intelligence future can't mix well at all.

Re: Books in .txt format for AI training purposes

#75

Post by sillysaurusx »

sailingparrot wrote: post #73
asutekku wrote: post #69

Earlier quoted context omitted.

36gb of compressed text is a lot though.

That's about 50B tokens. It can be a lot depending on what you want to do, but It's 10% of what GPT-3 (the definition of world class GPT model) used.

For what it's worth, we're serious about replicating GPT-3. books3 is just one piece. You will notice I never claimed equivalency to GPT-3's training data.

books3 may be 10%, but The Pile is building the rest:

https://twitter.com/arankomatsuzaki/status/13204141418954874...

https://github.com/EleutherAI/The-Pile

https://www.eleuther.ai/get-involved

https://media.discordapp.net/attachments/735217892517216366/...

https://media.discordapp.net/attachments/735217892517216366/...

Re: Books in .txt format for AI training purposes

#76

Post by heimatau »

ralph87 wrote: post #40

The entirety of Library Genesis (about 2.7 million books, fairly poorly curated) can also be downloaded, its somewhere around 40 TB altogether of significantly more recent books.

Is there a way to submit to it? Also, is there a way to download the entire thing?

Re: Books in .txt format for AI training purposes

#77

Post by sailingparrot »

sillysaurusx wrote: post #75
sailingparrot wrote: post #73

Earlier quoted context omitted.

That's about 50B tokens. It can be a lot depending on what you want to do, but It's 10% of what GPT-3 (the definition of world class GPT model) used.

For what it's worth, we're serious about replicating GPT-3. books3 is just one piece. You will notice I never claimed equivalency to GPT-3's training data. books3 may be 10%, but The Pile is building the rest: https://twitter.com/arankomatsuzaki/status/13204141418954874... https://github.com/EleutherAI/The-Pile https://www.eleuther.ai/get-involved https://media.discordapp.net/attachments/735217892517216366/... https:…

And again, I find books3 extremely cool and important work on your part, looking forward to the rest.

I just have a minor gripe with saying that "now we can train world class GPT model" thanks to that, as you said it's just one piece, and as a typicial HNist I had to point it out :).

Re: Books in .txt format for AI training purposes

#78

Post by webmaven »

nialv7 wrote: post #66
bigbubba wrote: post #32

Earlier quoted context omitted.

The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When you look at power burned doing it, the gap is even wider. Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.

> with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. That's not a fair comparison. The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution.

> The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution.

It is worth noting that the training data used for training humans has been undergoing it's own optimization process for a while too (arguably for about 40ky, since the Upper Paleolithic Revolution, which is roughly when cultural evolution started taking over). In AI adjusting the training data is called Curriculum Learning. Right now most curriculum learning is done just by adjusting the order of training samples, rather than creating samples specifically optimized to facilitate learning (although GANs might be considered a Socratic approach to the latter, if you squint at it).

Re: Books in .txt format for AI training purposes

#79

Post by sillysaurusx »

sailingparrot wrote: post #77
sillysaurusx wrote: post #75

Earlier quoted context omitted.

For what it's worth, we're serious about replicating GPT-3. books3 is just one piece. You will notice I never claimed equivalency to GPT-3's training data. books3 may be 10%, but The Pile is building the rest: https://twitter.com/arankomatsuzaki/status/13204141418954874... https://github.com/EleutherAI/The-Pile https://www.eleuther.ai/get-involved https://media.discordapp.net/attachments/735217892517216366/... https:…

And again, I find books3 extremely cool and important work on your part, looking forward to the rest. I just have a minor gripe with saying that "now we can train world class GPT model" thanks to that, as you said it's just one piece, and as a typicial HNist I had to point it out :).

Believe it or not, I appreciate and relate to that sentiment.

But after spending roughly one year acquiring knowledge related to this work, I feel I can say with a fairly high degree of certainty that this dataset alone is enough to train a model that will achieve "world class" status in some area. Writing books, perhaps.

Which part of my logic do you feel is mistaken, and why? I am actually quite interested to hear thoughts from someone who is very pedantic about such things.

Re: Books in .txt format for AI training purposes

#80

Post by jononor »

nialv7 wrote: post #66
bigbubba wrote: post #32

Earlier quoted context omitted.

The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When you look at power burned doing it, the gap is even wider. Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.

> with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. That's not a fair comparison. The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution.

Also, humans created all the text and the language and structures used within, for the purpose of communicating - so it is likely to be a good match for our cognitive systems.
Post reply on HN