Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

30% drop in O1-preview accuracy when Putnam problems are slightly variated

openreview.net

Post reply on HN
Print view View original

71–80 of 558 posts

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#71

Post by cogman10 »

coder543 wrote: post #53
WiSaGaN wrote: post #27

There is also a curated benchmark just for those famous problems slightly variated: https://github.com/cpldcpu/MisguidedAttention/tree/main/eval

One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the fa…

No. Simply plug in the prompt to chat gpt and see what happens.

The llm isn't getting confused by the meaning of "item". It's recognizing a common problem and not picking up on the fact that the farmer just needs to transport the goat and nothing else.

Instead, it gives the standard answer for how to transport everything across.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#72

Post by sirolimus »

mupuff1234 wrote: post #19
obblekk wrote: post #3

I hope someone reruns this on o1 and eventually o3. If o1-preview was the start like gpt1, then we should expect generalization to increase quickly.

You're assuming that openAI isn't just gonna add the new questions to the training data.

Exactly. The naivity is just sky-high

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#73

Post by sirolimus »

Lerc wrote: post #64
mupuff1234 wrote: post #42

Earlier quoted context omitted.

I think what it shows that it has minimal "understanding" of the problem - otherwise such small variations wouldn't pose a challenge. Training it to handle these specific small variations doesn't change that. It's good in automation, not understanding.

If it were a complete failure on variations I would be inclined to agree. Instead it was a 30% drop in performance. I would characterise that as limited understanding.

Fully agree with this

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#74

Post by coder543 »

kccqzy wrote: post #68
coder543 wrote: post #53

Earlier quoted context omitted.

One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the fa…

The problem is to ask the farmer to transport the goat. So the farmer indeed gets in the boat with the goat. The unstated gotcha is that the farmer is willing to abandon the wolf and the cabbage. A heavily pattern-matching LLM or human would immediately assume that the farmer needs to transport all three.

Yep, and that gotcha got me, as a perfectly non-silicon human. My bad everyone.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#75

Post by ben_w »

Link title says "slightly", but the PDF says two different kinds of variations: variable names (slight) and problem constants (significant), and the 30% drop is on the combination of a 26 variable and also 26 variable + constant questions.

It's good to have a better test (though I bet this one will also be quickly saturated like all the others), but the title here doesn't seem justified by the page title there or the content.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#76

Post by sunir »

torben-friis wrote: post #66
coder543 wrote: post #53

Earlier quoted context omitted.

One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the fa…

>This is twisting the English language to assume that "item" only refers to non-living things. Not really. Unless I'm not reading correctly, most of the problem is irrelevant as you're only required to cross the boat with the goat, you don't care about the cabbage. The difficulty lies in the assumption you need to cross everything due to the resemblance with the bigger problem.

You’re reading it correctly. I read it again after your comment and I realized I too pattern matched to the typical logic puzzle before reading it carefully and exactly. I imagine the test here is designed for this very purpose to see if the model is pattern matching or reasoning.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#77

Post by n144q »

falcor84 wrote: post #33
cscurmudgeon wrote: post #18

Earlier quoted context omitted.

Good students are immune to variations that are discussed in the paper. But most academic tests may not differentiate between them and the crammers.

> Good students are immune to variations I don't believe that. I'd put some good money that if an excellent student is given an exact question from a previous year, they'll do better (faster & more accurate) on it, than when they're given a variation of it.

I don't think you are betting on the same thing the parent comment is talking about.

The assumptions aren't the same to begin with.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#78

Post by ryvi »

llm_trw wrote: post #26
InkCanon wrote: post #10

Earlier quoted context omitted.

I've always assumed they removed it, because it's such a basic and fundamental part of ML training that you separate your test and train data. And yet I never see any papers even mention if/how they do this. And I wonder if they do, how do they guarantee with high reliability that their massive terabytes of data don't contain the answer.

Imagine you have someone polluting your training data every day. That's what happens when you scrape any tech forum today. The short version is that llm trainign data is the lowest quality data you are likely to see unless you engage in massive potential copyright infringement.

> unless you engage in massive potential copyright infringement. And nobody is going to do that

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#79

Post by [deleted] »

Lerc wrote: post #64
mupuff1234 wrote: post #42

Earlier quoted context omitted.

I think what it shows that it has minimal "understanding" of the problem - otherwise such small variations wouldn't pose a challenge. Training it to handle these specific small variations doesn't change that. It's good in automation, not understanding.

If it were a complete failure on variations I would be inclined to agree. Instead it was a 30% drop in performance. I would characterise that as limited understanding.

[deleted]

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#80

Post by danielbln »

surgical_fire wrote: post #57
lompad wrote: post #55

Earlier quoted context omitted.

And within half an hour somebody invested in nvidia stock is going to swoop in and explain how they totally (trust me bro) made x thousand with an app written by llm. Every. Single. Time. Almost as if there was a financial incentive to do that.

It's rhe crypto bullshit all over again. Tech hype is becoming unbearable as time goes on.

What's with the bitterness? Maybe don't get blinded by the hype and bring a little bit of wonder (and humility) back.
Post reply on HN