Live data from Hacker News

Books in .txt format for AI training purposes

twitter.com

51–60 of 97 posts

Re: Books in .txt format for AI training purposes

#51
post #23

What is ".txt format"? I'm genuinely curious what they went with here, but don't want to download 36 GB to find out. There isn't a .txt standard, is there? I would hope for UTF-8, but given the old-school, Windows-y .txt extension, it could just as easily be Windows-1252 or something. LF or CRLF line endings?

It looks like they are unix line endings, UTF-8, and markdown formatted.

Re: Books in .txt format for AI training purposes

#52

This is very cool and useful, thanks. However, saying: "Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data. Now you do. Now everyone does" Is quite a stretch. You need much, much more that 36GB of quality data to train a world-class GPT model.

The GPT approach must be very poor; with only a fraction of that information you can train a human to perform better than any GPT model shown thus far. When you look at power burned doing it, the gap is even wider. Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.

GPT and most modern "AI" approaches suffer from the same problem: they're just brute force algorithmic approaches that don't mimic "intelligence" in any way. Since there is no understanding, they can't learn in the same way humans or other living creatures do (in general terms, by extrapolation).

The ironic thing is that ML became popular because we didn't have the technology back then to properly model actual neuronal activity. Now we do, but ML is so established that modeling "true" intelligence is no longer de jure.

Re: Books in .txt format for AI training purposes

#53

Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use. With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title? Also, what's in that dmca.mp4 file? All I hear is music.

The dmca.mp4 file is a video of performance art in which about eight women in fancy dresses chant while vigorously mimicking male masturbation. I didn't watch to the end, but the activity appears to go on for a full ten minutes, and I bet there's a suitable finale.

Presumably this is a "fuck you" to the DMCA law.

EDIT: there is unfortunately no "suitable finale."

Re: Books in .txt format for AI training purposes

#54
post #8

Earlier quoted context omitted.

Of course it is a derived work.

I read a Java book and created programs using knowledge given in the book. Is my program a derived work?

A book on Java was written to teach you Java, so it would be foolish to claim any copyright on your program.

Re: Books in .txt format for AI training purposes

#55

Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use. With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title? Also, what's in that dmca.mp4 file? All I hear is music.

I’m downloading it right now, I’ll try to come back and let you know what it looks like.

Also the dmca.mp4 is a video of like 5 or 10 people singing into microphones while pretending to jack off

Re: Books in .txt format for AI training purposes

#56

Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use. With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title? Also, what's in that dmca.mp4 file? All I hear is music.

The filenames are readable and informative. The dmca.mp4 is a 10 minute video of a group of women standing in a half-circle, singing and doing "spank the monkey" gestures.

Re: Books in .txt format for AI training purposes

#57
post #24

Earlier quoted context omitted.

I’ve noticed that “presently” (in the meaning “soon”) is one such indicator for post-WWII English. British novels written in the 1950s consistently use “presently” to indicate a short passage of time. In the following decades it seems to have been removed from editors’ style guides and replaced mostly by plain old “soon”.

Wonder what other words that could be found that do something similar.

You may enjoy this article - https://en.wikipedia.org/wiki/Hapax_legomenon

Re: Books in .txt format for AI training purposes

#58

Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use. With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title? Also, what's in that dmca.mp4 file? All I hear is music.

I'm gonna take you at your word on blindness and help out here. There are a small assembly women in a semi-circle on stage, singing with stearn and serious faces what is an aria like song piece, all while vigorously gesturing what can universally be described male masterbation. The camera pans into their faces, and back out to reveal the small group of women to be a chorus of fapping.

Re: Books in .txt format for AI training purposes

#59
post #58

Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use. With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title? Also, what's in that dmca.mp4 file? All I hear is music.

I'm gonna take you at your word on blindness and help out here. There are a small assembly women in a semi-circle on stage, singing with stearn and serious faces what is an aria like song piece, all while vigorously gesturing what can universally be described male masterbation. The camera pans into their faces, and back out to reveal the small group of women to be a chorus of fapping.

[deleted]

Re: Books in .txt format for AI training purposes

#60

Blind person here. I don't need fancy shmancy formatting, so this could even come in handy for my own, personal use. With that said, does anyone know if the filename structure inside that file makes any sense for a human? Can you find a book by author/title? Also, what's in that dmca.mp4 file? All I hear is music.

> what's in that dmca.mp4 file? All I hear is music.

I wish you could see it, here's a basic description.

It begins with a close up of a person singing while performing a gesture of masturbation then zooms out to three people signing and then a full group of around 10 people. It's basically telling whoever is trying to DMCA to jerk themselves off.

Post reply on HN