30% drop in O1-preview accuracy when Putnam problems are slightly variated
1–10 of 558 posts
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#2Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#3I hope someone reruns this on o1 and eventually o3.
If o1-preview was the start like gpt1, then we should expect generalization to increase quickly.
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#4Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/
Not hardcoded, I think it's just likely that those problems exist in its training data in some form
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#5Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/
Not hardcoded, I think it's just likely that those problems exist in its training data in some form
Isn't that just the LLM equivalent of hardcoding though?
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#6Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/
Not hardcoded, I think it's just likely that those problems exist in its training data in some form
Seems a bit picky. If the bot has seen the exact problem before it's not really doing anything more than recall to solve it.
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#7This result is the same as a recent test of the same method+hypothesis from a group at Apple, no? I don’t have that reference handy but I don’t think I’m making it up.
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#8Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/
Not hardcoded, I think it's just likely that those problems exist in its training data in some form
If I temember well this is call overfitting [1].
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#9I would love to see how well Deepseek V3 do on this.
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#10Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/
Not hardcoded, I think it's just likely that those problems exist in its training data in some form
I've always assumed they removed it, because it's such a basic and fundamental part of ML training that you separate your test and train data. And yet I never see any papers even mention if/how they do this. And I wonder if they do, how do they guarantee with high reliability that their massive terabytes of data don't contain the answer.