Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Books in .txt format for AI training purposes

twitter.com

Post reply on HN
Print view View original

91–97 of 97 posts

Re: Books in .txt format for AI training purposes

#91

Post by bigbubba »

nialv7 wrote: post #67
gamblor956 wrote: post #52

Earlier quoted context omitted.

GPT and most modern "AI" approaches suffer from the same problem: they're just brute force algorithmic approaches that don't mimic "intelligence" in any way. Since there is no understanding, they can't learn in the same way humans or other living creatures do (in general terms, by extrapolation). The ironic thing is that ML became popular because we didn't have the technology back then to properly model actual neuron…

That's a bold claim to make, given we ourselves don't even understand what understanding, or intelligence is. So how can you support your claim that what GPT-3 does is not intelligence.

Personally I think our common definitions of intelligence are too narrowly defined, we have a bias towards recognizing intelligence similar to our own and a blindspot for intelligent systems unlike our own. I agree with Paul Stamets that mycelia networks may have a form of intelligence quite unlike our own. Large corporations likely have intelligence of their own ('slow AIs') and may in aggregate pursue goals that no individual participant in the organization agrees with (preference falsification demonstrates how such a thing might occur.) Closer to the traditional, I think reptiles may be far more intelligent that we usually give them credit for since we have a mammalian bias against their kind. Mammalians seem to have empathy circuits that fire more often for other mammals, and this inclines us to not consider the possible intelligence of non-mammals.

So, GPT? I think that's probably a form of intelligence too, but not one that's particularly similar to our own. I also doubt that we'll have much luck getting it to ever perform at near-human levels, when you consider the data and power requirements. I think it's not simply a matter of silicon vs wetware; I think the GPT approach is substantially more different from our own than the hardware it runs on. It's a form of intelligence but that doesn't mean it is like our own.

Re: Books in .txt format for AI training purposes

#92

Post by ralph87 »

gwern wrote: post #86
ralph87 wrote: post #40

The entirety of Library Genesis (about 2.7 million books, fairly poorly curated) can also be downloaded, its somewhere around 40 TB altogether of significantly more recent books.

The problem with LibGen, and why EleutherAI has been avoiding it, is that most of it is PDFs and most of the PDFs are scans; OCR layers are typically incredibly crummy. Even if you re-OCR them all with Tesseract or something (which will take quite a while, based on how long Tesseract takes to OCR my books even using parallelism on a Threadripper), the OCR will still be awful data. Do you really want to add that to yo…

It of course depends entirely on what kind of tool you are trying to build. PDFs capture a ton of visual information lost in plain text. I know the parent article is discussing construction of a text model, but what about say, if you were attempting to build the ultimate AI page reflow tool? etc.

Re: Books in .txt format for AI training purposes

#93

Post by ralph87 »

heimatau wrote: post #76
ralph87 wrote: post #40

The entirety of Library Genesis (about 2.7 million books, fairly poorly curated) can also be downloaded, its somewhere around 40 TB altogether of significantly more recent books.

Is there a way to submit to it? Also, is there a way to download the entire thing?

They release periodic delta torrents of the entire collection. I won't link it here, but it's trivial to find

Re: Books in .txt format for AI training purposes

#94

Post by WClayFerguson »

sillysaurusx wrote: post #85
WClayFerguson wrote: post #16

Is there a torrent link? Also, I know IPFS is still new, but growing rapidly, so some day each of these books will just have a CID on IPFS, and basically to publish all the books as a list you'd simply publish a list of the CIDs.

No torrent link yet, partly because I trust the-eye. They are fearless in a way that I haven't seen elsewhere. Time will tell if fear would have been wise. But for now, I'm curious to see whether "the link will last for years" is true. If it goes down, I'll append instructions to the original twitter thread on how to access it elsewhere. I also don't have any experience setting up the alternatives you mention; if you…

I just assumed all of that was in the public domain and legal to download. The only reason I was asking about Torrent was because of the large size of the file, and because I'm a developer working in IPFS world and very interested in decentralization.

I've developed a web platform that handles books online in a very unique way, and my demo 'book' for that technology was War and Peace:

https://quanta.wiki/n/war-and-peace

but probably if I need to add more books I can go to Project Gutenberg. Anyway, appreciated the reply

Re: Books in .txt format for AI training purposes

#95

Post by naniwaduni »

fakedang wrote: post #88
naniwaduni wrote: post #87

Earlier quoted context omitted.

> But of course humans don't infringe when they watch/listen/read a copyrighted work and it changes their state of mind. You're not generally distributing, performing, &c. your state of mind. On the other hand, if you then go and generate work based on what you read, then distributing that work certainly can be infringing. Thus "clean room" techniques, where one team reads copyrighted text, writes up a spec (which ma…

Distributing an exact copy or paraphrased content of an original work would be subject to copyright, but if a model just generates its own inferences, I don't see how it's different from say a human generated review.

Human generated reviews are generally considered derivative. That's why fair use &c. doctrines have factors expressly to enable them.

Re: Books in .txt format for AI training purposes

#96

Post by wyldfire »

naniwaduni wrote: post #87
wyldfire wrote: post #19

Earlier quoted context omitted.

Aside: it's interesting to think about the legal ramifications of copyrights in the face of AI. If my AI model "reads" a bunch of text, is the model now a "derived work" of the copyrighted work? Perhaps. But of course humans don't infringe when they watch/listen/read a copyrighted work and it changes their state of mind. Perhaps the model can be said not to infringe if it sufficiently abstracts concepts of the art in…

> But of course humans don't infringe when they watch/listen/read a copyrighted work and it changes their state of mind. You're not generally distributing, performing, &c. your state of mind. On the other hand, if you then go and generate work based on what you read, then distributing that work certainly can be infringing. Thus "clean room" techniques, where one team reads copyrighted text, writes up a spec (which ma…

> You're not generally distributing, performing, &c. your state of mind.

I don't think I agree. My employer values my contribution in some part as an oracle. People at work ask me questions and I answer them. Those answers come from the sum of my experiences (a biochemical 'model'). Other people more directly conduct public performances of their talent.

If nothing else this will likely blur the lines on what's considered original work.

Re: Books in .txt format for AI training purposes

#97

Post by webmaven »

isoprophlex wrote: post #34
yodon wrote: post #5

Earlier quoted context omitted.

> Are you objecting to the use of data for training, or to the compilation of training datasets? The GP is objecting to the casualness with which privacy rights and intellectual property rights are ignored by so many in the AI community, not to a choice of whether they object to one or another specific manifestation of how the community is doing so. [edit added]: Those who would argue "no, there is no casual ignoring…

Sweet jesus those instructions go on forever, imagine performing that (without breaking into laughter)

I'm pretty sure that it's the same few seconds looped for ten minutes.
Post reply on HN