Performance of these LLMs on real life tasks feels very much like students last-minute cramming for Asian style exams. The ability to perfectly regurgitate, while no concept of meaning.
Look into JEE Advanced.
30% drop in O1-preview accuracy when Putnam problems are slightly variated
21–30 of 558 posts
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#22Or it's time to step back and call it what it is - very good pattern recognition. I mean, that's cool... we can get a lot of work done with pattern recognition. Most of the human race never really moves above that level of thinking in the workforce or navigating their daily life, especially if they default to various societally prescribed patterns of getting stuff done (eg. go to college or the military , find a job…
> Or it's time to step back and call it what it is - very good pattern recognition. Or maybe it's time to stop wheeling out this tedious and disingenuous dismissal. Saying it is just "pattern recognition" (or a "stochastic parrot") implies behavioural and performance characteristics that have very clearly been greatly exceeded.
Citation needed. Please be more specific, or else this is just a tedious and disingenuous advocacy.
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#23Hmmm without a human control it is not all that clear to me that the variation problems are not more difficult.
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#24trained on test. who even trusts OAI anymore?
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#25Earlier quoted context omitted.
Not hardcoded, I think it's just likely that those problems exist in its training data in some form
Seems a bit picky. If the bot has seen the exact problem before it's not really doing anything more than recall to solve it.
20 years ago in grad school we were doing a very early iteration of this where we built Markov chains with Shakespeare's plays and wanted to produce a plausibly "Shakespearian" clause given a single word to start and a bearish professor said "the more plausible it gets the more I worry people might forget plausibility is all that it promises".
(There was also a much earlier piece of software that would generate semi-intelligible Kant or Hegel one sentence at a time, though that was through a series of a priori generation rules and a large at the time dictionary of stock phrases. I wonder what ever happened to that.)
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#26Earlier quoted context omitted.
Not hardcoded, I think it's just likely that those problems exist in its training data in some form
I've always assumed they removed it, because it's such a basic and fundamental part of ML training that you separate your test and train data. And yet I never see any papers even mention if/how they do this. And I wonder if they do, how do they guarantee with high reliability that their massive terabytes of data don't contain the answer.
Imagine you have someone polluting your training data every day. That's what happens when you scrape any tech forum today.
The short version is that llm trainign data is the lowest quality data you are likely to see unless you engage in massive potential copyright infringement.
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#27There is also a curated benchmark just for those famous problems slightly variated: https://github.com/cpldcpu/MisguidedAttention/tree/main/eval
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#28Earlier quoted context omitted.
I've always assumed they removed it, because it's such a basic and fundamental part of ML training that you separate your test and train data. And yet I never see any papers even mention if/how they do this. And I wonder if they do, how do they guarantee with high reliability that their massive terabytes of data don't contain the answer.
Imagine you have someone polluting your training data every day. That's what happens when you scrape any tech forum today. The short version is that llm trainign data is the lowest quality data you are likely to see unless you engage in massive potential copyright infringement.
[deleted]
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#29Earlier quoted context omitted.
Seems a bit picky. If the bot has seen the exact problem before it's not really doing anything more than recall to solve it.
20 years ago in grad school we were doing a very early iteration of this where we built Markov chains with Shakespeare's plays and wanted to produce a plausibly "Shakespearian" clause given a single word to start and a bearish professor said "the more plausible it gets the more I worry people might forget plausibility is all that it promises". (There was also a much earlier piece of software that would generate semi-…
It became a successful consultant...
Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated
#30I hope someone reruns this on o1 and eventually o3. If o1-preview was the start like gpt1, then we should expect generalization to increase quickly.
You're assuming that openAI isn't just gonna add the new questions to the training data.
Their methodology shows they can create an infinite variety of problems.
This is the same thing as synthetic training data.
It doesn't matter if models are trained on the output of the generated data or not. If the model ends up being able to solve newly generated variations, you'd have to admit that it understands the underlying problems.