Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book

understandingai.org

Post reply on HN
Print view View original

321–326 of 326 posts

Re: Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book

#321

Post by fennecfoxy »

otabdeveloper4 wrote: post #236
fennecfoxy wrote: post #228

Earlier quoted context omitted.

Has nothing to do with "cloud providers". The randomness is inherent to the sampler, using a sampler that picks top probability for next token would result in lower quality output as I have definitely seen it get stuck in certain endless sequences when doing that. Ie you get something like "Complete this poem 'over yonder hills I saw' output: a fair maiden with hair of gold like the sun gold like the sun gold like th…

> would result in lower quality output No it wouldn't. > seen it get stuck in certain endless sequences when doing that Yes, and infinite loops is just an inherent property of LLMs, like hallucinations.

How would it not result in lower quality output? You're reducing the set of tokens that may be selected to 1. The pool isn't necessarily synonyms but words that share some semantic connection to the previous word, but the selection of one word in particular can certainly impact the word that is selected next.

Explain your reasoning otherwise.

Re: Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book

#322

Post by otabdeveloper4 »

fennecfoxy wrote: post #321
otabdeveloper4 wrote: post #236

Earlier quoted context omitted.

> would result in lower quality output No it wouldn't. > seen it get stuck in certain endless sequences when doing that Yes, and infinite loops is just an inherent property of LLMs, like hallucinations.

How would it not result in lower quality output? You're reducing the set of tokens that may be selected to 1. The pool isn't necessarily synonyms but words that share some semantic connection to the previous word, but the selection of one word in particular can certainly impact the word that is selected next. Explain your reasoning otherwise.

> You're reducing the set of tokens that may be selected to 1.

Yes, reducing it to 1 token that is deemed to be the optimal token according to the model.

Re: Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book

#323

Post by aspenmayer »

aprilthird2021 wrote: post #318
aspenmayer wrote: post #316

Earlier quoted context omitted.

I didn’t ask for info, I asked for your views. I gave you all the info anyone has publicly, so you have enough to comment. I suspect that it was a limited hangout self-own by Meta to claim that they aren’t responsible, and then they are doing research on a leaked LLM that they developed, but then was leaked, so they can claim that the subsequent research is not tainted by the fruit of the poisonous tree legal doctrin…

I'm not really sure what you are insinuating? You think Meta leaked LAMA so they could claim, legally that they are in the clear for copyright violation? Sorry, I just don't really get what you want me to opine about. If that is what you are asking, I don't think that's what happened. It's far more likely that it was just leaked or grabbed by a hacker

I just thought the whole situation was interesting. You commented about the current LLM research being clean, while being based on prior LLMs which were perhaps less clean, so I thought that it was a curious coincidence how torrents kept popping up.

Re: Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book

#324

Post by int_19h »

Lerc wrote: post #247
Retric wrote: post #218

Earlier quoted context omitted.

I think you may have something with that line of reasoning. The threshold for transformative for fictional works is fairly high unfortunately. Fan fiction and reasonably distinct works with excessive inspiration are both copyright infringing. https://en.wikipedia.org/wiki/Tanya_Grotter > Models themselves are very clearly transformative. A near word for word copy of large sections of a work seems nowhere near that th…

Models are not word for word copies of large sections of text. They are capable of emitting that text though. It would be interesting to look at what legal precidents were set regarding mp3s or other encodings. Is the encoding itself an infringement, or is it the decoding, or is it the distribution of a decodable form of a work. There is also the distinction with a lossy encoding that encodes a single work. There is…

> When the encoding acts as a bulk archive, does the responsibility shift to those who choose what to extract from the archive?

If you take many gigabytes of, say, public domain music, and stick them on a flash drive with just one audio file that is an unlicensed copy of a copyrighted song, distributing that drive would constitute copyright infringement, quite obviously so. I don't see why it'd matter what else the model can produce, if it can produce that one thing verbatim by itself.

(If you could only prompt the model to regurgitate the original text with a framing of, say, critical analysis of said text around it, and not in any other context, then I think there would be a stronger fair use argument here.)

Re: Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book

#325

Post by int_19h »

Lerc wrote: post #294
rpd9803 wrote: post #281

Earlier quoted context omitted.

Copyright fair use rules are tools designed to govern how humans use protected works in dervied works. AI is not human use, therefore the rules are only coincidentally correct for AI use where it even is.

If you take that approach to fair use, don't you open the door to the same argument for copyright itself? How do you distinguish between a tool and the director of a tool? I doubt people would say that a person is immune to copyright or fair use rules because it was the pen that wrote the document, not the person.

I think it's a valid question. Suppose you have two LLMs interacting with each other in a loop, and one randomly prompts the other to reproduce the entire text of Harry Potter, which the other then does. However, the chat log isn't actually stored anywhere, it's just a transient artifact of the interaction - so no human ever sees it nor can see it even in principle. Is it a copyright violation then? If it is, what are the damages?

Re: Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book

#326

Post by int_19h »

paxys wrote: post #75
OtherShrezzing wrote: post #68

Earlier quoted context omitted.

I think the argument is less about piracy and more that the model(s output) is a derivative work of Harry Potter, and the rights holder should be paid accordingly when it’s reproduced.

That may be relevant in the NYT vs OpenAI case, since NYT was supposedly able to reproduce entire articles in ChatGPT. Here Llama is predicting one sentence at a time when fed the previous one, with 50% accuracy, for 42% of the book. That can easily be written off as fair use.

If it can predict the next sentence reliably, that sentence then becomes part of the context, so if you just continue inference, it would eventually produce the entire text verbatim, no?
Post reply on HN