You need to know that this test set data wasn't included in the training data for this to be meaningful.
I thought the test limericks were autogenerated?
They come from a database of 98k limericks -- https://zenodo.org/records/5722527
GPT-4o's Memory Breakthrough – Needle in a Needlestack
41–50 of 256 posts
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#42I'd like to see this for Gemini Pro 1.5 -- I threw the entirety of Moby Dick at it last week, and at one point all books Byung Chul-Han has ever published, and it both cases it was able to return the single part of a sentence that mentioned or answered my question verbatim, every single time, without any hallucinations.
Just put the 2500 example linked on the article through Gemini 1.5 Flash and it answered correctly ("The tree has diseased leaves and its bark is peeling.") https://aistudio.google.com/
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#43The needle in the haystack test gives a very limited view of the model’s actual long context capabilities. It’s mostly used because early models were terrible at it and it’s easy to test. In fact, most recent models now do pretty good at this one task, but in practice, their ability to do anything complex drops off hugely after 32K tokens.
RULER is a much better test:
https://github.com/hsiehjackson/RULER
> Despite achieving nearly perfect performance on the vanilla needle-in-a-haystack (NIAH) test, all models (except for Gemini-1.5-pro) exhibit large degradation on tasks in RULER as sequence length increases.
> While all models claim context size of 32k tokens or greater (except for Llama3), only half of them can effectively handle sequence length of 32K by exceeding a qualitative threshold, Llama2-7b performance at 4K (85.6%). The performance exceeding the threshold is underlined.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#44That's great to hear. My biggest issue with GPT-4.0 was that as the conversation got longer, the quality diminished (especially relevant for coding projects) I wonder if it'll be better now. Will test today.
That’s been my experience so far. My current conversations are crazy long compared to any of my gpt4 convos which I had to frequently copy context from and start over in a new chat
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#45[deleted]
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#46This is a very promising development. It would be wise for everyone to go back and revise old experiments that failed now that this capability is unlocked. It should also make RAG even more powerful now that you can load a lot more information into the context and have it be useful.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#47Earlier quoted context omitted.
I would hope that Byung-Chul Han would not be in the training set (at least not without his permission), given he's still alive and not only is the legal question still open but it's also definitely rude. This doesn't mean you're wrong, though.
It's pretty easy to confirm that copywritten material is in the training data. See the NYT lawsuit against OpenAI for example.
Part of that back-and-forth is the claim "this specific text was copied a lot all over the internet making it show up more in the output", and that means it's not a useful guide to things where one copy was added to The Pile and not removed when training the model.
(Or worse, that Google already had a copy because of Google Books and didn't think "might training on this explode in our face like that thing with the Street View WiFi scanning?")
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#48Anyone has done any benchmarks for RAG yet?
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#49How Do we know that gpt-4o.has not been trained on this dataset?
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#50Earlier quoted context omitted.
My idea is to buy to a unpublished novel or screenplay with a detailed, internally consistent world built in to it and a cast of characters that have well crafted motivations and then ask it to continue writing from an arbitrary post-mid-point by creating a new plot line that combines two characters that haven't yet met in the story. If it understands the context it should be able to write a new part of the story and…
You can only use it for a short while, they get a copy as well.
I have been thinking about this for use in evaluating locally run models, so I didn't make that connection in this case. I guess it would have limited utility.