GPT-4o's Memory Breakthrough – Needle in a Needlestack
nian.llmonpy.ai
GPT-4o's Memory Breakthrough – Needle in a Needlestack
1–10 of 256 posts
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#2The article shows how much better GPT-4o is at paying attention across its input window compared to GPT-4 Turbo and Claude-3 Sonnet.
We've needed an upgrade to needle in a haystack for a while and this "Needle In A Needlestack" is a good next step! NIAN creates a prompt that includes thousands of limericks and the prompt asks a question about one limerick at a specific location.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#3It's interesting (though perhaps not surprising) to see the variance in curve shape across models.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#4I'd like to see this for Gemini Pro 1.5 -- I threw the entirety of Moby Dick at it last week, and at one point all books Byung Chul-Han has ever published, and it both cases it was able to return the single part of a sentence that mentioned or answered my question verbatim, every single time, without any hallucinations.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#5I'd like to see this for Gemini Pro 1.5 -- I threw the entirety of Moby Dick at it last week, and at one point all books Byung Chul-Han has ever published, and it both cases it was able to return the single part of a sentence that mentioned or answered my question verbatim, every single time, without any hallucinations.
Wow. Cool. I have access to that model and have also seen some impressive context extraction. It also gave a really good summary of a large code base that I dumped in. I saw somebody analyze a huge log file, but we really need something like this needle in a needlestack to help identify when models might be missing something. At the very least, this could give model developers something to analyze their proposed models.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#6Would love to see Gemini there too!
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#7That's great to hear. My biggest issue with GPT-4.0 was that as the conversation got longer, the quality diminished (especially relevant for coding projects)
I wonder if it'll be better now. Will test today.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#8I'd like to see this for Gemini Pro 1.5 -- I threw the entirety of Moby Dick at it last week, and at one point all books Byung Chul-Han has ever published, and it both cases it was able to return the single part of a sentence that mentioned or answered my question verbatim, every single time, without any hallucinations.
If I had access to Gemini with a reasonable token rate limit, I would be happy to test Gemini. I have had good results with it in other situations.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#9Someone needs to come up with a "synthesis from haystack" test that tests not just retrieval but depth of understanding, connections, abstractions across diverse information.
When a person reads a book, they have an "overall intuition" about it. We need some way to quantify this. Needle in haystack tests feel like a simple test that doesn't go far enough.
Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack
#10That's great to hear. My biggest issue with GPT-4.0 was that as the conversation got longer, the quality diminished (especially relevant for coding projects) I wonder if it'll be better now. Will test today.
I had the same experience. With a 16k prompt, Turbo was nearly flawless. But it wasn't very good at 32k and not usable at 100+. You have to repeat information to get good results with longer prompts