Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

30% drop in O1-preview accuracy when Putnam problems are slightly variated

openreview.net

Post reply on HN
Print view View original

261–270 of 558 posts

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#261

Post by mmaunder »

jerf wrote: post #197

I remember when this stuff was all coming out and people were finally excited about ChatGPT getting the problem with "which is heavier, a 10 pound bag of feathers or a 10 pound bag of bricks?" problem correct. But of course it got it correct. It was in the training set. Vary the problem slightly by just changing the nouns, or changing the numbers so that one in fact was heavier than the other, and performance went al…

I've posted this before and I know it's a cliche, but this really is Goodhart's Law at work with the benchmarks becoming targets.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#262

Post by woopwoop »

InkCanon wrote: post #10
Trasmatta wrote: post #4

Earlier quoted context omitted.

Not hardcoded, I think it's just likely that those problems exist in its training data in some form

I've always assumed they removed it, because it's such a basic and fundamental part of ML training that you separate your test and train data. And yet I never see any papers even mention if/how they do this. And I wonder if they do, how do they guarantee with high reliability that their massive terabytes of data don't contain the answer.

I agree that openai is somewhat sketchy about this, but they're sketchy about everything. In the past though they have admitted up front to data contamination (e.g. the original gpt-4 press release did not use big-bench as a benchmark due to data contamination). For the Putnam in particular: this is not a benchmark that they use. There is no reason to exclude it since it is not part of the "test set" in any meaningful sense.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#263

Post by collinvandyck76 »

bufferoverflow wrote: post #223
collinvandyck76 wrote: post #218

Earlier quoted context omitted.

I reproduced this on Claude Sonnet 3.5, but found that changing your prompt to "Which is heavier, a 9.99-pound back of steel ingots or a 10.01-pound bag of fluffy cotton?" corrected its reasoning, after repeated tests. For some reason it was not able to figure out that "10.01" referred to pounds.

You wrote "back" instead of "bag".

Ah, good catch -- I had just copied the OP. Changing it to bag instead of back doesn't seem to change much. It still oscillates between the right and wrong answer, versus qualifying the 10.01 value as pounds, which has not yet given me the wrong answer (small sample size, of course).

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#264

Post by JumpCrisscross »

whimsicalism wrote: post #241
JumpCrisscross wrote: post #235

Earlier quoted context omitted.

> just asked o1-mini (not 4o mini) 5 times in a row (new chats obviously) and it got it right every time Could you try playing with the exact numbers and/or substances?

give me a query and i’ll ask it, but also i don’t want to burn through all of my o1mini allocation and have to use the pay-as-you-go API.

> give me a query and i’ll ask it

Which is heavier: an 11kg bag of lint or a 20lb bag of gold?

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#265

Post by whimsicalism »

JumpCrisscross wrote: post #264
whimsicalism wrote: post #241

Earlier quoted context omitted.

give me a query and i’ll ask it, but also i don’t want to burn through all of my o1mini allocation and have to use the pay-as-you-go API.

> give me a query and i’ll ask it Which is heavier: an 11kg bag of lint or a 20lb bag of gold?

yeah it gets it

https://chatgpt.com/share/67757720-3c7c-8010-a3e9-ce66fb9f17...

e: cool, this gets downvoted

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#266

Post by stavros »

mitemte wrote: post #67
coder543 wrote: post #53

Earlier quoted context omitted.

One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the fa…

Wow, this seems ridiculous. The expected answer is basically finding a loophole in the problem. I can imagine how worthless all of these models would be if they behaved that way.

It's not a loophole, the question is "how can he get the goat across?". The answer is he just takes it across.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#267

Post by s1mplicissimus »

jokethrowaway wrote: post #15
obblekk wrote: post #3

I hope someone reruns this on o1 and eventually o3. If o1-preview was the start like gpt1, then we should expect generalization to increase quickly.

I don't think llm generalise much, that's why they're not creative and can't solve novel problems. It's pattern matching with a huge amount of data. Study on the topic: https://arxiv.org/html/2406.15992v1 This would explain o1 poor performance with problems with variations. o3 seems to be expensive brute forcing in latent space followed by verification which should yield better results - but I don't think we can call…

the fact that this (and tons of other legitimate critique) got downvoted into greytext speaks so much louder to me than all benchmarks in the world

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#268

Post by whimsicalism »

x_may wrote: post #56
ankit219 wrote: post #31

They are highly effective pattern matchers. You change the pattern, it won't work. I don't remember who, but most likely @tszzl (roon), commented on x that they still trained the traditional way, and there is no test time compute (TTC) or Montecarlo Tree search (like Alpha Go) in o1 or o3. If that is true, then it's still predicting the next word based on it's training data. Likely to follow the most probable path -…

I believe they are using scalable TTC. The o3 announcement released accuracy numbers for high and low compute usage, which I feel would be hard to do in the same model without TTC. I also believe that the 200$ subscription they offer is just them allowing the TTC to go for longer before forcing it to answer. If what you say is true, though, I agree that there is a huge headroom for TTC to improve results if the huggi…

ttc is an incredibly broad term and it is broadening as the hype spreads. people are now calling CoT “TTC” because they are spending compute on reasoning tokens before answering

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#269

Post by chongli »

mortehu wrote: post #213
jerf wrote: post #204

Earlier quoted context omitted.

That appears to be the same model I used. This is why I emphasized I didn't "go shopping" for a result. That was the first result I got. I'm not at all surprised that it will nondeterministically get it correct sometimes. But if it doesn't get it correct every time, it doesn't "know". (In fact "going shopping" for errors would still even be fair. It should be correct all the time if it "knows". But it would be differ…

While this may be true, it's a very common problem that people who want to demonstrate how bad a model is fail to provide a direct link or simply state the name of the model.

OpenAI is not doing us any favours by using confusing naming schemes for their models and obscuring which models people are actually working with.

If I didn’t know any better, I’d say OpenAI doesn’t want us doing these tests accurately and is trying to hide something.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#270

Post by whimsicalism »

4ad wrote: post #260
lukeschlather wrote: post #246

Earlier quoted context omitted.

I usually test models using the OpenAI API which doesn't offer links the way I think you mean. If I provide some output I got from a particular model you're just going to have to take my word for it.

You can use https://lluminous.chat (bring your own key) to link to chats using any model across all LLMs.

open router is the more standard solution
Post reply on HN