Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

30% drop in O1-preview accuracy when Putnam problems are slightly variated

openreview.net

Post reply on HN
Print view View original

301–310 of 558 posts

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#301

Post by kqr »

jerf wrote: post #197

I remember when this stuff was all coming out and people were finally excited about ChatGPT getting the problem with "which is heavier, a 10 pound bag of feathers or a 10 pound bag of bricks?" problem correct. But of course it got it correct. It was in the training set. Vary the problem slightly by just changing the nouns, or changing the numbers so that one in fact was heavier than the other, and performance went al…

Shows the importance of chain of thought! Forcing it to commit to an answer without deliberation is not playing to its strength.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#302

Post by riffraff »

qwertox wrote: post #300
jerf wrote: post #197

I remember when this stuff was all coming out and people were finally excited about ChatGPT getting the problem with "which is heavier, a 10 pound bag of feathers or a 10 pound bag of bricks?" problem correct. But of course it got it correct. It was in the training set. Vary the problem slightly by just changing the nouns, or changing the numbers so that one in fact was heavier than the other, and performance went al…

As long as an LLM is capable of inserting "9.99 > 10.01?" into an evaluation tool, we're on a good way. It feels a bit like "if all you have is a hammer, everything looks like a nail", where we're trying to make LLMs do stuff which it isn't really designed to do. Why don't we just limit LLMs to be an interface to use other tools (in a much more human way) and train them to be excellent at using tools. It would also m…

> As long as an LLM is capable of inserting "9.99 > 10.01?" into an evaluation tool, we're on a good way

chatgpt will switch to python for some arithmetic with the result that you get floating point math issues when a 8yo will get the result right. I think "switch to a tool" still requires understanding of which tool to get a reliable result, which in turn means understanding the problem. It's an interesting issue.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#303

Post by Dylan16807 »

redman25 wrote: post #251
klabb3 wrote: post #232

Earlier quoted context omitted.

Disagree. The AI we have is very useful for specific things. The pushback you see is not so much denying the milestones that have been surpassed, but rather the milestones that enthusiasts claim are near. And for good reason! Every time and in every field we’ve extrapolated an exponential-looking curve ad infinitum, it’s turned out to be S-shaped, and life goes on. > We had a threshold for intelligence. We’ve had man…

If you consider that evolution has taken millions of years to produce intelligent humans--that LLM training completed in a manner of months can produce parrots of humans is impressive by itself. Talking with the parrot is almost indistinguishable from talking with a real human. As far as pattern matching, the difference I see from humans is consciousness. That's probably the main area yet to be solved. All of our cur…

> If you consider that evolution has taken millions of years to produce intelligent humans--that LLM training completed in a manner of months can produce parrots of humans is impressive by itself.

I disagree that such a comparison is useful. Training should be compared to training, and LLM training feeds in so many more words than a baby gets. (A baby has other senses but it's not like feeding in 20 years of video footage is going to make an LLM more competent.)

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#304

Post by chongli »

qup wrote: post #296
chongli wrote: post #276

Earlier quoted context omitted.

It does not show the model version anywhere on the page on chatgpt.com, even when logged in.

Yes it does, at the top of every chat there is a drop-down to select the model, which displays the current model. It's been a constant part of the UI since forever.

No, it only says "ChatGPT Plus (Upgrade)" or "ChatGPT".

Maybe it's different if you have a paid account?

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#305

Post by bwfan123 »

yifanl wrote: post #2

Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/

this work is similar to the GSM symbolic paper (applied to putnam) https://arxiv.org/html/2410.05229v1

going forward, llm performance must be reported on the confounded benchmark as well

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#306

Post by make3 »

whimsicalism wrote: post #143
amluto wrote: post #140

Earlier quoted context omitted.

> ask it for a formula for mass-energy equivalence Way too easy. If you think that mass and energy might be equivalent, then dimensional analysis doesn’t give you too much choice in the formula. Really, the interesting thing about E=mc^2 isn’t the formula but the assertion that mass is a form of energy and all the surrounding observations about the universe. Also, the actual insight in 1905 was more about asking the…

but e=mc^2 is just an approximation e: nice, downvoted for knowing special relativity

it's a special case, not an approximation

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#307

Post by whimsicalism »

chongli wrote: post #304
qup wrote: post #296

Earlier quoted context omitted.

Yes it does, at the top of every chat there is a drop-down to select the model, which displays the current model. It's been a constant part of the UI since forever.

No, it only says "ChatGPT Plus (Upgrade)" or "ChatGPT". Maybe it's different if you have a paid account?

if i go to chatgpt.com on my phone not logged on at all it tells me very prominently at the top that i am using 4o mini

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#308

Post by whimsicalism »

make3 wrote: post #306
whimsicalism wrote: post #143

Earlier quoted context omitted.

but e=mc^2 is just an approximation e: nice, downvoted for knowing special relativity

it's a special case, not an approximation

its not an either/or, it is both. regardless, my point is that you cannot simply dimensional analysis your way to special relativity or the energy-momentum relation

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#309

Post by bwfan123 »

frikskit wrote: post #290

An interesting example of this is: There are 6 “a”s in the sentence: “How many ‘a’ in this sentence?” https://chatgpt.com/share/677582a9-45fc-8003-8114-edd2e6efa2... Whereas the typical “strawberry” variant is now correct. There are 3 “r”s in the word “strawberry.” Clearly the lesson wasn’t learned, the model was just trained on people highlighting this failure case.

Reminds me of software i have built which had some basic foundational problems. Each bug was fixed with a data-patch that fixed the symptom but not the cause.

hence we continually played whack-a-mole with bugs. we would squash one bug, and another one would appear.

same with llms, squash one problem with a data-fix, and another one pops-up.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#310

Post by whimsicalism »

MattDaEskimo wrote: post #280
whimsicalism wrote: post #230

Earlier quoted context omitted.

this thread is about math LLM capability, it’s a bit ridiculous to say that mentioning frontiermath is off topic but that’s just me

Just because you can generalize the topic doesn't mean you can ignore the specific conversation and choose your hill to argue. Additionally, the conversation of this topic is about the model's ability to generalize and it's potential overfitting, which is arguably more important than parroting mathematics.

performance on a held-out set (like frontiermath) compared to putnam (which is not held out) is obviously relevant to a model's potential overfitting.

i'm not going to keep replying, others can judge whether they think what i'm saying is "relevant at all."

Post reply on HN