Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4o's Memory Breakthrough – Needle in a Needlestack

nian.llmonpy.ai

Post reply on HN
Print view View original

51–60 of 256 posts

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#51

Post by irthomasthomas »

This is based on a limericks dataset published in 2021. https://zenodo.org/records/5722527

I think it very likely that gpt-4o was trained on this. I mean, why would you not? Innnput, innnput, Johnny five need more tokens.

I wonder why the NIAN team don't generate their limericks using different models, and check to make sure they're not in the dataset? Then you'd know the models couldn't possibly be trained on them.

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#54

Post by nsagent »

19h wrote: post #4

I'd like to see this for Gemini Pro 1.5 -- I threw the entirety of Moby Dick at it last week, and at one point all books Byung Chul-Han has ever published, and it both cases it was able to return the single part of a sentence that mentioned or answered my question verbatim, every single time, without any hallucinations.

A number of people in my lab do research into long context evaluation of LLMs for works of fiction. The likelihood is very high that Moby Dick is in the training data. Instead the people in my lab have explored recently published books to avoid these issues.

See BooookScore (https://openreview.net/forum?id=7Ttk3RzDeu) which was just presented at ICLR last week and FABLES (https://arxiv.org/abs/2404.01261) a recent preprint.

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#55

Post by sftombu »

irthomasthomas wrote: post #51

This is based on a limericks dataset published in 2021. https://zenodo.org/records/5722527 I think it very likely that gpt-4o was trained on this. I mean, why would you not? Innnput, innnput, Johnny five need more tokens. I wonder why the NIAN team don't generate their limericks using different models, and check to make sure they're not in the dataset? Then you'd know the models couldn't possibly be trained on them.

I tested the LLMs to make sure they could not answer the questions unless the limerick was given to them. Other than 4o, they do very badly on this benchmark, so I don't think the test is invalidated by their training.

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#56

Post by sftombu »

DominikPeters wrote: post #42
19h wrote: post #4

I'd like to see this for Gemini Pro 1.5 -- I threw the entirety of Moby Dick at it last week, and at one point all books Byung Chul-Han has ever published, and it both cases it was able to return the single part of a sentence that mentioned or answered my question verbatim, every single time, without any hallucinations.

Just put the 2500 example linked on the article through Gemini 1.5 Flash and it answered correctly ("The tree has diseased leaves and its bark is peeling.") https://aistudio.google.com/

Interesting!

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#57

Post by pushedx »

jddj wrote: post #35
jddj wrote: post #30

Earlier quoted context omitted.

An elaborate Agatha Christie style whodunit, with a series of plot-twists and alibis which can be chopped off the end of the piece to modify who is the most likely suspect

Or a spot the difference. Generate 1000 generic facts about Alice and the same 1000 facts about Eve. Randomise the order and change one minor detail then ask how they differ.

    sort alice.txt | diff -
That's not a task for an LLM

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#58

Post by yatz »

Well, I can now use GPT to transform raw dynamic data into beautiful HTML layouts on the fly for low-traffic pages, such as change/audit logs, saving a ton of development time and keeping my HTML updated even when the data structure has changed. My last attempt did not consistently work because GPT4-Turbo sometimes ignored the context and instructions almost entirely.

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#60

Post by sftombu »

cararemixed wrote: post #31

What's the chance that these limericks are now in the training set? As others mention, it'd be interesting to come up with a way to synthesize something sufficiently interesting so it always evades training fit.

Previous answer to this question:

https://news.ycombinator.com/item?id=40361419

Post reply on HN