Can Gemini 1.5 read all the Harry Potter books at once?
1–10 of 40 posts
Re: Can Gemini 1.5 read all the Harry Potter books at once?
#2There might be enough Harry Potter related content in its training set that it's not really "reading" the books in its context.
Re: Can Gemini 1.5 read all the Harry Potter books at once?
#3How much of that character map is already in its training data and how much of it is actually read from the input prompt?
I’m always suspicious of these kinds of tests. It needs to be run with an unpublished book, not one of the most popular series in the 21st century.
Re: Can Gemini 1.5 read all the Harry Potter books at once?
#4ML 101: Do not evaluate on the training data.
Yes of course it can, because they fit in the context window. But this is an awful test of the model's capabilities because it was certainly trained on these books and websites talking about the books and the HP universe.
Re: Can Gemini 1.5 read all the Harry Potter books at once?
#5Answer: No (but almost)
Re: Can Gemini 1.5 read all the Harry Potter books at once?
#6Not sure about all of the Harry Potter books, but I gave it My entire data export from ChatGPT and handled it very well. I was able to search through it and have conversation again from past conversations. It was good.
Re: Can Gemini 1.5 read all the Harry Potter books at once?
#7How much of that character map is already in its training data and how much of it is actually read from the input prompt? I’m always suspicious of these kinds of tests. It needs to be run with an unpublished book, not one of the most popular series in the 21st century.
This, Harry potter is a terrible example, even weak models know the rough story of Harry Potter unprompted.
If you want a real test, go test it on some Japanese light novel, or some harry potter fanfiction, and see if the model actually understands the plot details.
For reference, Opus/GPT-4 know the rough story of moderately popular light novels/mangas without any context given. They however do not precisely understand the fine-grained details of the story, like which character will win in a fight.
Re: Can Gemini 1.5 read all the Harry Potter books at once?
#8Not sure about all of the Harry Potter books, but I gave it My entire data export from ChatGPT and handled it very well. I was able to search through it and have conversation again from past conversations. It was good.
I'm curious if that would work for me. How many megabytes was the export?
By way of reference, mine is currently around 7mb.
Re: Can Gemini 1.5 read all the Harry Potter books at once?
#9How much of that character map is already in its training data and how much of it is actually read from the input prompt? I’m always suspicious of these kinds of tests. It needs to be run with an unpublished book, not one of the most popular series in the 21st century.
This, Harry potter is a terrible example, even weak models know the rough story of Harry Potter unprompted. If you want a real test, go test it on some Japanese light novel, or some harry potter fanfiction, and see if the model actually understands the plot details. For reference, Opus/GPT-4 know the rough story of moderately popular light novels/mangas without any context given. They however do not precisely underst…
Japanese light novels were almost certainly in the training set, either in their original Japanese, an English translation in that Books2 pile, or in a fan translation that happened to get scraped.
Re: Can Gemini 1.5 read all the Harry Potter books at once?
#10ML 101: Do not evaluate on the training data. Yes of course it can, because they fit in the context window. But this is an awful test of the model's capabilities because it was certainly trained on these books and websites talking about the books and the HP universe.
Not so fast. If you were evaluating the model on its ability to predict the next word in a Harry Potter book, you'd be right, because it's already seen the entire book, but that's not what's happening here.
The linked X post shows that the user asked the model to generate a graph of the characters, which was presumably a novel question. This is a legitimate test of the model's ability to understand and answer questions about the training data. Repeating the books in the prompt for emphasis makes sense, since the model probably didn't memorize all the relevant details.