Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

30% drop in O1-preview accuracy when Putnam problems are slightly variated

openreview.net

Post reply on HN
Print view View original

531–540 of 558 posts

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#531

Post by p1esk »

csomar wrote: post #502
sholladay wrote: post #248

Earlier quoted context omitted.

ChatGPT Plus user here. The following are all fresh sessions and first answers, no fishing. GPT 4: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. The type of material doesn’t affect the weight comparison; it’s purely a matter of which bag weighs more on the scale. GPT 4o: The 10.01-pound bag of fluffy cotton is heavier. Weight is independent of the material, so the bag of cot…

o1-preview: To determine which bag is heavier when measured in air—the 9.99-pound bag of steel ingots or the 10.01-pound bag of fluffy cotton—we need to consider not just their masses but also the effect of air buoyancy on their apparent weights. *Step 1: Calculate the Masses* First, convert the given weights into masses using the conversion \(1 \text{ pound} = 0.453592 \text{ kg}\): - Mass of steel bag, \(m_s = 9.99…

This answer is probably better than 99.99% of human answers. Feel the AGI!

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#532

Post by kbelder »

thaumasiotes wrote: post #495
ted_dunning wrote: post #398

Earlier quoted context omitted.

Hmm... Gemini (1.5 Flash) just aced that exact question for me: These lines celebrate the victory of the British ship HMS Victory, led by the famous Admiral Lord Nelson, in the Battle of Trafalgar in 1805. "Here's success unto the Victory": This line directly praises the ship itself, acknowledging its role in the successful battle. "and crew of noble fame": This recognizes the bravery and skill of the sailors who ser…

That's not acing the question. It's completely incorrect. What do you think the singer in "Friends in Low Places" meant in the toast he gave after crashing his ex-girlfriend's wedding? And I saw the surprise and the fear in his eyes when I took his glass of champagne and I toasted you, said "Honey, we may be through but you'll never hear me complain"

Sounds like you just proved ted_dunning isn't sentient.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#533

Post by kbelder »

Workaccount2 wrote: post #480
deeviant wrote: post #210

Earlier quoted context omitted.

I don't believe that is the model that you used. I wrote a script and pounded 01 mini and gpt 4 with a wide vareity of tempature and top_p parameters, and was unable to get it to give the wrong answer a single time. Just a whole bunch of: (openai-example-py3.12) :~/code/openAiAPI$ python3 featherOrSteel.py Response 1: A 10.01-pound bag of fluffy cotton is heavier than a 9.99-pound bag of steel ingots. Response 2: A 1…

The elephant in the room is that HN is full of people facing an existential threat.

Who think they're facing an existential threat.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#534

Post by kbelder »

flatline wrote: post #299
qup wrote: post #294

Earlier quoted context omitted.

It's so weird that people use questions that are well-known for duping humans, who we all consider to be general intelligence. Getting this question wrong doesn't say much about the intelligence of humans, why would it say something about the AI?

Because for things like the Putnam questions, we are trying to get the performance of a smart human. Are LLMs just stochastic parrots or are they capable of drawing new, meaningful inferences? We keep getting more and more evidence of the latter, but things like this throw that into question.

It is perfectly possible for the first AGI to be stupid. A moron. In fact, I'd bet that's fairly likely.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#535

Post by kbelder »

trashtester wrote: post #520
jerf wrote: post #197

I remember when this stuff was all coming out and people were finally excited about ChatGPT getting the problem with "which is heavier, a 10 pound bag of feathers or a 10 pound bag of bricks?" problem correct. But of course it got it correct. It was in the training set. Vary the problem slightly by just changing the nouns, or changing the numbers so that one in fact was heavier than the other, and performance went al…

What weighs more, a 100kt aircraft carrier or a 200kt thermonuclear weapon?

How many (presumably conscious) people would answer that incorrectly?

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#536

Post by stefan_ »

scrollaway wrote: post #457
stefan_ wrote: post #348

Earlier quoted context omitted.

Why? Did a grocery store self checkout ever fail to calculate sales tax? Do I need to run a study on that? The people selling this could not make a car drive but now its AGI.

A single-purpose state machine not failing to do the single thing it was created to do does not make for the clever retort you think it makes. "AGI": emphasis on "G" for "General". The LLMs are not failing to do generalized tasks, and that they are nondeterministic is not a bug. Just don't use them for calculating sales tax. You wouldn't hire a human to calculate sales tax in their head, so why do you make this a req…

I do not get it, where are all the serious people? Just come out and say what it is and what it is not. Pretend for a moment you have the ethics and work philosophy of an undergraduate science student.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#537

Post by roboboffin »

whimsicalism wrote: post #523
roboboffin wrote: post #501

Earlier quoted context omitted.

I think the problems it solved were understood to be well known undergraduate problems. https://xenaproject.wordpress.com/2024/12/22/can-ai-do-maths...

that is not the correct takeaway from the blogpost at all… 1. it suggests it’s possible that more of the problems are IMO-esque than previously thought, we don’t know how the share of solved problems is. 2. calling IMO problems “well known undergraduate problems” is a bit much

The actual quote is that they are easier than IMO; but maybe not directly comparable.

https://x.com/littmath/status/1870848783065788644?s=46&t=foR...

I think it’s more probable that it would have solve the easier problems first, rather than some hard and only some easier; although that is supposition.

Reading this thread and the blog post gives more idea about what the problems might involve.

It’s difficult to judge without more information on the actual results, but that means we cannot draw any strong conclusions either way on what this means.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#538

Post by sdenton4 »

thaumasiotes wrote: post #529
sdenton4 wrote: post #521

Earlier quoted context omitted.

That requires knowing the song, beyond the words provided. Would you flunk an eighth grader for getting it wrong?

Well, the rest of the song helps, in that it specifies that (1) the toast upset the wedding, and (2) the singer responded to that by insulting "you", which is presumably one or more of the bride, the groom, and the guests. But I think specifying that the singer has crashed his ex-girlfriend's wedding is already enough that you deserve to fail if your answer is "he says he's not upset, so what he means is that he's no…

I was referring to the original query, of course, as any entity capable of reasoning could have figured out.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#539

Post by gmueckl »

deeviant wrote: post #373
gmueckl wrote: post #362

Earlier quoted context omitted.

Down voted for not actually countering the argument in question? The script doesn't alter the phrasing of the question itself. It just generates a randomized, irrelevant preamble.

Well, I understood the argument in question to be: was it possible for the model to be fooled by this question, not was it possible to prompt engineer it into failure. The parameter space I was exploring, then, was the different decoding parameters available during the invocation of the model, with the thesis that if were possible to for the model to generate an incorrect answer to the question, I would be able to re…

Intuitively, I wouldn't expect a wrong answer to show up that easily if the network was overfitted to that particular input token sequence.

The questions as I understand it is whether the network learned enough of a simulacrum of the concept of weight to answer similar questions correctly.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#540

Post by Dylan16807 »

ninetyninenine wrote: post #512
Dylan16807 wrote: post #303

Earlier quoted context omitted.

> If you consider that evolution has taken millions of years to produce intelligent humans--that LLM training completed in a manner of months can produce parrots of humans is impressive by itself. I disagree that such a comparison is useful. Training should be compared to training, and LLM training feeds in so many more words than a baby gets. (A baby has other senses but it's not like feeding in 20 years of video fo…

No, a baby is pre-trained. We know from linguistics that there is a natural language grammar template all humans follow. This template is intrinsic to our biology and is encoded and not learned through observation.

A baby has a template but so does an LLM.

The better comparison to the templating is all the labor that went into making the LLM, not how long the GPUs run.

Template versus template, or specific training versus specific training. Those comparisons make a lot more sense than going criss-cross.

Post reply on HN