Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can Gemini 1.5 read all the Harry Potter books at once?

twitter.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Can Gemini 1.5 read all the Harry Potter books at once?

#21

Post by paxys »

munchler wrote: post #10
rryan wrote: post #4

ML 101: Do not evaluate on the training data. Yes of course it can, because they fit in the context window. But this is an awful test of the model's capabilities because it was certainly trained on these books and websites talking about the books and the HP universe.

Not so fast. If you were evaluating the model on its ability to predict the next word in a Harry Potter book, you'd be right, because it's already seen the entire book, but that's not what's happening here. The linked X post shows that the user asked the model to generate a graph of the characters, which was presumably a novel question. This is a legitimate test of the model's ability to understand and answer questio…

It could have been trained on this exact picture created by a fan and uploaded to some forum. Ultimately it is impossible to know unless testing with brand new material.

I have the same problem with benchmarks that use real world tests (like SAT/LSAT/GRE or whatever else). The model got a good score, sure, but how many thousands of variations of this exact test was it trained on? How many questions did it encounter that were similar or the same?

Re: Can Gemini 1.5 read all the Harry Potter books at once?

#22

Post by anonylizard »

kmeisthax wrote: post #9
anonylizard wrote: post #7

Earlier quoted context omitted.

This, Harry potter is a terrible example, even weak models know the rough story of Harry Potter unprompted. If you want a real test, go test it on some Japanese light novel, or some harry potter fanfiction, and see if the model actually understands the plot details. For reference, Opus/GPT-4 know the rough story of moderately popular light novels/mangas without any context given. They however do not precisely underst…

Japanese light novels were almost certainly in the training set, either in their original Japanese, an English translation in that Books2 pile, or in a fan translation that happened to get scraped.

That's expected, and why the model can reproduce the basic details.

But those Japanese light novels don't have millions of forum discussions and essays written on it. So it shows how well the model can recall sparse data in its training dataset, rather than recalling a dataset that basically shows up 100000 times in different forms.

Re: Can Gemini 1.5 read all the Harry Potter books at once?

#23

Post by poglet »

munchler wrote: post #10
rryan wrote: post #4

ML 101: Do not evaluate on the training data. Yes of course it can, because they fit in the context window. But this is an awful test of the model's capabilities because it was certainly trained on these books and websites talking about the books and the HP universe.

Not so fast. If you were evaluating the model on its ability to predict the next word in a Harry Potter book, you'd be right, because it's already seen the entire book, but that's not what's happening here. The linked X post shows that the user asked the model to generate a graph of the characters, which was presumably a novel question. This is a legitimate test of the model's ability to understand and answer questio…

You could modify the source material (change a name or character relationship) and see if it correctly reports the modification in the graph.

Re: Can Gemini 1.5 read all the Harry Potter books at once?

#24

Post by aikinai »

xmprt wrote: post #15
munchler wrote: post #10

Earlier quoted context omitted.

Not so fast. If you were evaluating the model on its ability to predict the next word in a Harry Potter book, you'd be right, because it's already seen the entire book, but that's not what's happening here. The linked X post shows that the user asked the model to generate a graph of the characters, which was presumably a novel question. This is a legitimate test of the model's ability to understand and answer questio…

It's a novel question and impressive that Gemini was able to solve it but the tweet's author is claiming that this is because of the large context window and not because of all the Harry Potter related training data that was available to it.

The generic models definitely know a lot about Harry Potter without any additional context.

Probably 80% of my questions to ChatGPT were about Harry Potter plot and character details as my kid was reading the books. It was extremely knowledgeable about all the minutiae, probably thanks to all the online discussion more than the books themselves. It was actually the first LLM killer app for me.

Re: Can Gemini 1.5 read all the Harry Potter books at once?

#25

Post by HlessClaudesman »

OK so my next question is what can you do with a model loaded with Harry Potter Context? Answer Harry Potter Trivia at a superhuman level? Write the next Harry Potter adventure?

Having used GPTs to do creative writing I can report that they are good for solving the tyranny of the blank page, but then you have to read and edit hundreds of pages of dank AI prose, which never quite aligns with your creative vision, to harvest a few nuggets of creativity. Does it end up saving any time?

Re: Can Gemini 1.5 read all the Harry Potter books at once?

#26

Post by earthnail »

HlessClaudesman wrote: post #25

OK so my next question is what can you do with a model loaded with Harry Potter Context? Answer Harry Potter Trivia at a superhuman level? Write the next Harry Potter adventure? Having used GPTs to do creative writing I can report that they are good for solving the tyranny of the blank page, but then you have to read and edit hundreds of pages of dank AI prose, which never quite aligns with your creative vision, to h…

Think about all the soulless powerpoint presentations this could replace. They were never written with a creative vision in mind, but you meed a tom of context for accurate information.

Re: Can Gemini 1.5 read all the Harry Potter books at once?

#28

Post by magospietato »

rryan wrote: post #4

ML 101: Do not evaluate on the training data. Yes of course it can, because they fit in the context window. But this is an awful test of the model's capabilities because it was certainly trained on these books and websites talking about the books and the HP universe.

I got some interesting results by feeding Claude 3 a very sparse primer for a conlang I wrote when I was 18.

There is zero chance this is anywhere in the models dataset, and we were able to perform basic translation to and from English.

Re: Can Gemini 1.5 read all the Harry Potter books at once?

#29

Post by barfbagginus »

JonSolomon wrote: post #19
barfbagginus wrote: post #8

Earlier quoted context omitted.

I'm curious if that would work for me. How many megabytes was the export? By way of reference, mine is currently around 7mb.

Mine was 8-9 mb, there’s an html in the zip that contains them all. What I’m not sure about is if 1.5 is truncating it.

You could do the needle recall test, detailed in the following article, and used in the Gemini technical report:

https://vladbogo.substack.com/p/gemini-15-unlocking-multimod...

https://arxiv.org/abs/2403.05530

In the needle test, we generate "needle" queries at each point in the context. We then graph the model's recall for each needle.

In this case we might

1. Generate needles: Iterate through your conversations one by one, and generate multiple choice questions for each conversation. You might need to break down conversations into chunks.

2. Test the haystack. Given the full context, run random batches of needle queries. Run multiple batches, to give good coverage of the context.

3. Visualize recall. Graph the conversation # vs the needle recall score for that conversation

Let's assume Gemini is truncating your context, but has perfect recall for no truncated context. Then your needle graph will be ~100% for all conversation in context, and then fall off like a cliff at the exact point of truncation.

My main concern with this approach is the cost, as you have to load the entire context for each batch of needles. It's likely that testing all needles at the same time would skew the results or exceed the allowed context.

I don't know how the authors deal with this issue, and I don't know if they have published code for needle testing. But if you're interested in working on this, I'd like to collaborate. We can look at the existing solutions, and if necessary we can build a needle testing fixture for working with GPT exports. I'd also be interested in supporting more broad needle testing use cases, like books, API docs, academic papers, etc.

Post reply on HN