What's the chance that these limericks are now in the training set? As others mention, it'd be interesting to come up with a way to synthesize something sufficiently interesting so it always evades training fit.
GPT-4o's Memory Breakthrough – Needle in a Needlestack
31–40 of 256 posts
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#32You need to know that this test set data wasn't included in the training data for this to be meaningful.
I thought the test limericks were autogenerated?
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#33Someone needs to come up with a "synthesis from haystack" test that tests not just retrieval but depth of understanding, connections, abstractions across diverse information. When a person reads a book, they have an "overall intuition" about it. We need some way to quantify this. Needle in haystack tests feel like a simple test that doesn't go far enough.
The needles form a graph and the prompt asks graph based tasks.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#34Someone needs to come up with a "synthesis from haystack" test that tests not just retrieval but depth of understanding, connections, abstractions across diverse information. When a person reads a book, they have an "overall intuition" about it. We need some way to quantify this. Needle in haystack tests feel like a simple test that doesn't go far enough.
My idea is to buy to a unpublished novel or screenplay with a detailed, internally consistent world built in to it and a cast of characters that have well crafted motivations and then ask it to continue writing from an arbitrary post-mid-point by creating a new plot line that combines two characters that haven't yet met in the story. If it understands the context it should be able to write a new part of the story and…
You can only use it for a short while, they get a copy as well.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#35Someone needs to come up with a "synthesis from haystack" test that tests not just retrieval but depth of understanding, connections, abstractions across diverse information. When a person reads a book, they have an "overall intuition" about it. We need some way to quantify this. Needle in haystack tests feel like a simple test that doesn't go far enough.
An elaborate Agatha Christie style whodunit, with a series of plot-twists and alibis which can be chopped off the end of the piece to modify who is the most likely suspect
Or a spot the difference.
Generate 1000 generic facts about Alice and the same 1000 facts about Eve. Randomise the order and change one minor detail then ask how they differ.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#36Someone needs to come up with a "synthesis from haystack" test that tests not just retrieval but depth of understanding, connections, abstractions across diverse information. When a person reads a book, they have an "overall intuition" about it. We need some way to quantify this. Needle in haystack tests feel like a simple test that doesn't go far enough.
Why can't you be that someone?
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#37Someone needs to come up with a "synthesis from haystack" test that tests not just retrieval but depth of understanding, connections, abstractions across diverse information. When a person reads a book, they have an "overall intuition" about it. We need some way to quantify this. Needle in haystack tests feel like a simple test that doesn't go far enough.
Why can't you be that someone?
lol, made me think of the euphemism: be the change you want to see.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#38I'd like to see this for Gemini Pro 1.5 -- I threw the entirety of Moby Dick at it last week, and at one point all books Byung Chul-Han has ever published, and it both cases it was able to return the single part of a sentence that mentioned or answered my question verbatim, every single time, without any hallucinations.
But this content is presumably in its training set, no? I'd be interested if you did the same task for a collection of books published more recently than the model's last release.
Gemini works with brand new books too; I've seen multiple demonstrations of it. I'll try hunting one down. Side note: this experiment is still insightful even using model training material. Just compare its performance with the uploaded book(s) to without.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#39Someone needs to come up with a "synthesis from haystack" test that tests not just retrieval but depth of understanding, connections, abstractions across diverse information. When a person reads a book, they have an "overall intuition" about it. We need some way to quantify this. Needle in haystack tests feel like a simple test that doesn't go far enough.
The needles form a graph and the prompt asks graph based tasks.
That is an interesting idea
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#40You need to know that this test set data wasn't included in the training data for this to be meaningful.
No you don't. Compare the model's performance before and after uploading the material.