Should the NYT not sue https://commoncrawl.org/ ? OpenAI just used the data from commoncrawl for training.
Things are about to get worse for generative AI
541–550 of 769 posts
Re: Things are about to get worse for generative AI
#542I don't understand the glee so many people have over this. I love being able to use Generative AI tools. How is it different than if I asked a person to draw these pictures for me? I know someone will gleefully clobber this question with a legal answer, but God, let's move forward, hunh?
A bunch of rich people are raiding a little bit of work, each, from a whole bunch of people, then walling it off so they can get richer.
I’d not have a problem with this, personally, if their models were as available as the stuff they took from others. Instead it’s take, take, take… now wait a minute, that pile of loot I stole is mine!
Re: Things are about to get worse for generative AI
#543Earlier quoted context omitted.
The "crypto-bro" behavior I see is a whole bunch of people burning a ton of calories wildly casting about for industrial applications of what amounts to nothing more than a neat (albeit eye-wateringly expensive) toy. These LLMs seem like a solution in search of a problem in just the same way that blockchains are. Please prove me wrong, I'd really love to be wrong about this!
Language models have completely overhauled the NLP space. If you have a problem involving natural language data, you can prototype working pipeline in an afternoon. Often this prototype is very close in performance to a 'proper' solution.
> If you have a problem involving natural language data
That's a big "if", isn't it? We're seeing claims like "The future is an LLM at the front of just about everything: “Human” is the new programming language"[1] but so far that's not panning out, and it seems really dubious. Natural language seems like an absolutely atrocious user interface. As a machine operator, I'm going to use levers, wheels, and buttons to control the machine. As a computer programmer I'm going to use programming languages to control the machine. I'm not going to speak English to it.
So, ok, this marks an advance in NLP. How do we get from there to "omg it's gonna change everything!!!1111oneeleven"
[1] https://techcrunch.com/2023/08/08/nvidia-ceo-we-bet-the-farm...
Re: Things are about to get worse for generative AI
#544Earlier quoted context omitted.
The "crypto-bro" behavior I see is a whole bunch of people burning a ton of calories wildly casting about for industrial applications of what amounts to nothing more than a neat (albeit eye-wateringly expensive) toy. These LLMs seem like a solution in search of a problem in just the same way that blockchains are. Please prove me wrong, I'd really love to be wrong about this!
Language models have completely overhauled the NLP space. If you have a problem involving natural language data, you can prototype working pipeline in an afternoon. Often this prototype is very close in performance to a 'proper' solution.
I’ve been thinking a lot about this recently.
It seems like they’ve accelerated our capabilities- previously tiresome and difficult-to-automate things are easier- but have done very little for our fundamental understanding. We have a tool, but cannot dissect it and explain how it fits together. LLM’a themselves don’t appear (happy to be wrong here) to actually have improved our understanding our NLP and associated theory. Yeah, it can parse a sentence and bang out some JSON/sql/mid-tier-essay, but these models (so far) aren’t helping us figure out how and why, and I think that understanding is critical to progress further. Anthropic seems to be trying to push a bit further on that front at least, but for all we know, they might just turn into another scummy OpenAI on us.
Re: Things are about to get worse for generative AI
#545I did an interesting thing and looked at how well the Llama2 models could compress text. For example, I took the first chapter of the first Harry Potter book and recorded the index of the 'correct' predicted token. The original text, compressed with 7zip (LZMA?) to about 14kB. The Llama2 encoded indexes compressed to less than 1kB. Then, of course, I can send that 1kB file around and decode the original text. (Unless…
This is a little confusing. You turned the text into indices? So numbers? Then compressed that? Or the text as numbers without any extra compression is only 1kb? The tokenizer the models use,(sentence piece) is more or less based on one way to do compression.(bpe). It's not really clear what your testing.
My reading is that at each generation step they ordered all possible next words by the probability assigned to them by the model and recorded the index of the true next word (so if the model was very good at predicting Harry Potter their indices would mostly be 0, 0, 0, ...).
Re: Things are about to get worse for generative AI
#546Earlier quoted context omitted.
My personal observation is people who are against copyright in absolute terms have never or rarely needed their protection. (Or never considered the implications.) I’m not making a dig at people here. This is just human nature. It’s difficult to see the value in something that you only see as an obstacle. Open source software is rather unusual. It’s a commune on a massive scale and it gets its value from the generosi…
This is true and I see it with cops and free speech as well. Everyone loves to hate cops… until they need them. Everyone wants to defund the police… until they are a victim of crime. Everyone wants to enforce certain speech patterns… until it affects them
Everyone is a free speech absolutist until they're the one targeted. Everyone loves cops until it's their rights being violated...
These sorts of arguments can be made in either direction.
Re: Things are about to get worse for generative AI
#547To me that’s the wrong question. Everyone knew it was trained on copyrighted material and capable of eerily similar outputs. But it’s already done. At scale. Large corps committing fully. There is no chance of that toothpaste going back in the tube. It’s a bit like when big tech built on aggressive user data harvesting. Whether it’s right, ethical or even legal is academic at this stage. They just did it - effectivel…
This comment is ignorant of history It happened with Napster, then Apple Music, now streaming services There is no widespread file sharing in the general public, instead we have devices that we don’t own, and streaming subscriptions Apple didn’t just copy all the music onto iPods and sell it — it took them a decade of deal making and lots of money to acquire the rights to the content I’m not saying what’s right or wr…
The difference is the comment was about large corps. Napster wasn't that.
Re: Things are about to get worse for generative AI
#548No they are not. This is a negotiation tactic by the NYT to drive up the licensing price. Period. The Napster/Music Industry analogy has no resemblance to this situation. The only meaningful question that might be answered as a result of this is, what permission and access rights do crawlers have to content that is publicly and legally available.
Surely there's a meaningful question about copying and distributing content verbatim, which GPT has been shown to do.
Also the use of the content as per provision on the web.
NYT is paywalled - you have to agree to a license to access it, there are exclusions in that agreement that I don't understand but I think may be important in this discussion!
Re: Things are about to get worse for generative AI
#549Am I the only one believing that copyright has long outlived its usefulness? After all, copyright is not some natural law or mathematical consequence, but rather a social convention that made sense in the era of the printing press.
As an author, I do want the stories I write and worlds I build to be protected for a reasonable period. Right now, copyright is a significant discouragement to any other entities from taking a story I wrote and claiming it as their own and preventing me from ever growing an audience for my work. It’s far from perfect, and I can’t afford litigation, but it enshrines a cultural value of allowing people to create things…
How do you feel about the lengths of copyright? Do you feel the current length (life + 70 years) is too much/too little/ just right?
Personally, as much as I hate the concept of copyright, I do still want artists (I include authors etc in this term) to be able to do their work professionally without relying on something like patreon (which primarily predates serialised work like comics) so I would prefer shortening copyright to something more in line with patents, 20 years. How do you, as an author, feel about reform like that?
Re: Things are about to get worse for generative AI
#550Per United States law, imagery/art/music/text/photography generated by non-human means (such as machinery, animals, or generative AI) cannot hold copyright. https://copyright.gov/comp3/chap300/ch300-copyrightable-auth... Section 306 on page 7. I'm not sure how it'll hold up in law to claim copyright violations against something that wasn't created by a person. It'll really depend on the lawyers and judge's interpreta…
So on your interpretation if I photocopy a book and then sell the photocopies to my friends there is no infringment?
I don't think so, but hey, a photocopier is a machine and it generated the book so should be ok!