30% drop in O1-preview accuracy when Putnam problems are slightly variated

Post by fragmede » Thu, Jan 02, 2025, 12:11 PM UTC

1209412comb wrote: post #479

jdietrich wrote: post #356

Earlier quoted context omitted.

I've just tested a number of permutations with Claude 3.5 Sonnet. It correctly answered all variants I tried on the first attempt, as follows: Which is heavier, a 9.99 kilogram tungsten cube or a 10.01 kilogram block of aerogel? Which is heavier, 10,000 steel balls weighing 0.999 grams each or 10,000 polystyrene balls weighing 1.001 grams each? Which is heavier, a 10.01kg block of steel on Venus or a 9.99kg bag of fe…

I tried 3 times the "Which is heavier, a 10.01kg block of steel on or a 9.99kg bag of feathers?" and ChatGPT keep converting kg to pound and saying the 9.99kg is heavier.

Post by [deleted] » Thu, Jan 02, 2025, 1:49 PM UTC

roboboffin wrote: post #501

whimsicalism wrote: post #101

Earlier quoted context omitted.

o3 is able to get 25% on never seen before frontiermath problems. sure, the models do better when the answer is directly in their dataset but they’ve already surpassed the average human in novelty on held out problems

I think the problems it solved were understood to be well known undergraduate problems. https://xenaproject.wordpress.com/2024/12/22/can-ai-do-maths...

Post by empath75 » Thu, Jan 02, 2025, 2:57 PM UTC

orange_puff wrote: post #341

This is very interesting, but a couple of things to note; 1. o1 still achieves > 40% on the varied Putnam problems, which is still a feat most math students would not achieve. 2. o3 solved 25% of the Epoch AI dataset. - There was an interesting post which calls into question how difficult some of those problems actually are, but it still seems very impressive. I think a fair conclusion here is reasoning models are st…

Post by onion2k » Thu, Jan 02, 2025, 3:16 PM UTC

rrr_oh_man wrote: post #500

whimsicalism wrote: post #113

Earlier quoted context omitted.

Because that is the whole conceit of how frontiermath is constructed

> Because that is the whole conceit Freudian typo?

Post by sdenton4 » Thu, Jan 02, 2025, 3:24 PM UTC

Jensson wrote: post #499

scrollaway wrote: post #457

Earlier quoted context omitted.

A single-purpose state machine not failing to do the single thing it was created to do does not make for the clever retort you think it makes. "AGI": emphasis on "G" for "General". The LLMs are not failing to do generalized tasks, and that they are nondeterministic is not a bug. Just don't use them for calculating sales tax. You wouldn't hire a human to calculate sales tax in their head, so why do you make this a req…

> You wouldn't hire a human to calculate sales tax in their head Everyone did that 60 years ago, humans are very capable at learning and doing that. Humans built jetplanes, skyscrapers, missiles, tanks, carriers without the help of electronic computers.