Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

30% drop in O1-preview accuracy when Putnam problems are slightly variated

openreview.net

Post reply on HN
Print view View original

491–500 of 558 posts

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#491

Post by frikskit »

Isinlor wrote: post #419
frikskit wrote: post #290

An interesting example of this is: There are 6 “a”s in the sentence: “How many ‘a’ in this sentence?” https://chatgpt.com/share/677582a9-45fc-8003-8114-edd2e6efa2... Whereas the typical “strawberry” variant is now correct. There are 3 “r”s in the word “strawberry.” Clearly the lesson wasn’t learned, the model was just trained on people highlighting this failure case.

Transformers are very bad at counting due to how their internals work. But if you ask them to use explicit counter the problem disappears: https://chatgpt.com/share/6775c9a6-8cec-8007-b709-3431e7a2b2... Basically one feed forward is not Turing complete, but autoregressive (feeding previous output back into itself) are Turing complete.

This makes it worse IMO. I was starting to think it didn’t have a letter by letter representation of the tokens. It does. In which case the fact it didn’t decide to use it speaks even more towards its unsophistication.

Regardless, I’d love if you would explain a bit more why the transformer internals make this problem so difficult?

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#493

Post by sudosysgen »

whimsicalism wrote: post #473
sudosysgen wrote: post #466

Earlier quoted context omitted.

OpenAI never actually directly participated in the contests. OpenAI says they took (unspecified) problems and "simulated" the score it would have had. If they solved recent contests in a realistic contest simulation I would expect them to give the actual solutions and success rates as well, like they did for IOI problems, so I'm actually confused as to why they didn't.

very good clarification, thanks - they should absolutely release more details to provide more clarity and ideally just participate live? i suspect that the model takes a while for individual problems so time might be a constraint there

Yes, it would also probably be rate limited or timed out since they're doing 50+ submissions per problem

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#494

Post by thaumasiotes »

mikeruiz wrote: post #383
whimsicalism wrote: post #342

Earlier quoted context omitted.

i'd prefer an easily verifiable question rather than one where we can always go "no that's not what they really meant" but someone else with o1-mini quota can respond

“They’re toasting Admiral Nelson’s ship (HMS Victory) and its valiant crew, hailing the ship’s successes and Nelson’s heroism. In other words, the singer is offering tribute—“success unto the Victory”—to the vessel and its famed sailors, and “glory to the captain” who led them, namely the celebrated Admiral Horatio Nelson.” …but to your point, no idea if the artist intended some more obscure reference. o1-pro was als…

> no idea if the artist intended some more obscure reference.

No, but the answer is obviously wrong. Very, very wrong. The lines are not praise. The song's view of the Victory, and of the battle of Trafalgar, is overwhelmingly negative.

> o1-pro was also able to produce a relatively complete version of original source

...as should be clear, if you read the original source. (Or, heck, just the two lines following the ones in the question.) Being able to reproduce the source makes the error look that much worse.

> though, amusingly, referred to it as a ‘broadside’ rather than ‘broadsheet’.

Both terms are used. https://en.wikipedia.org/wiki/Broadside_ballad :

>> A broadside (also known as a broadsheet) is a single sheet of inexpensive paper printed on one side, often with a ballad, rhyme, news and sometimes with woodcut illustrations.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#495

Post by thaumasiotes »

ted_dunning wrote: post #398
thaumasiotes wrote: post #318

Earlier quoted context omitted.

>> so far o1-mini has bodied every task people are saying LLMs can’t do in this thread > give me a query and i’ll ask it Here's a query similar to one that I gave to Google Gemini (version unknown), which failed miserably: ---query--- Steeleye Span's version of the old broadsheet ballad "The Victory" begins the final verse with these lines: Here's success unto the Victory / and crew of noble fame and glory to the cap…

Hmm... Gemini (1.5 Flash) just aced that exact question for me: These lines celebrate the victory of the British ship HMS Victory, led by the famous Admiral Lord Nelson, in the Battle of Trafalgar in 1805. "Here's success unto the Victory": This line directly praises the ship itself, acknowledging its role in the successful battle. "and crew of noble fame": This recognizes the bravery and skill of the sailors who ser…

That's not acing the question. It's completely incorrect.

What do you think the singer in "Friends in Low Places" meant in the toast he gave after crashing his ex-girlfriend's wedding?

And I saw the surprise and the fear in his eyes

when I took his glass of champagne

and I toasted you, said "Honey, we may be through

but you'll never hear me complain"

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#496

Post by Isinlor »

frikskit wrote: post #491
Isinlor wrote: post #419

Earlier quoted context omitted.

Transformers are very bad at counting due to how their internals work. But if you ask them to use explicit counter the problem disappears: https://chatgpt.com/share/6775c9a6-8cec-8007-b709-3431e7a2b2... Basically one feed forward is not Turing complete, but autoregressive (feeding previous output back into itself) are Turing complete.

This makes it worse IMO. I was starting to think it didn’t have a letter by letter representation of the tokens. It does. In which case the fact it didn’t decide to use it speaks even more towards its unsophistication. Regardless, I’d love if you would explain a bit more why the transformer internals make this problem so difficult?

When Can Transformers Count to n?

https://arxiv.org/html/2407.15160v2

The Expressive Power of Transformers with Chain of Thought

https://arxiv.org/html/2310.07923v5

Transformer needs to retrieve letters per each token while forced to keep internal representation still aligned in length with the base tokens (each token also has finite embedding, while made out of multiple letters), and then it needs to count the letters within misaligned representation.

Autoregressive mode completely alleviate the problem as it can align its internal representation with the letters and it can just keep explicit sequential count.

BTW - humans also can't count without resorting to sequential process.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#497

Post by Isinlor »

tacitusarc wrote: post #475
Isinlor wrote: post #395

Earlier quoted context omitted.

BTW - the model may be wrong depending on the example. More voluminous objects displace more air and due to buoyancy are lighter for the same mass. The proper way to ask it would be to ask which object has more mass.

For what it’s worth, I asked it the following: Q: Which weighs more on earth: 100kg of steel or a 100kg polystyrene ball? Be as technically correct as possible A: Technically, 100 kg of steel and 100 kg of polystyrene weigh the same on Earth when measured on a scale, as they both experience the same gravitational force due to their equal mass. However, there are some nuances to consider: Buoyancy Effect Earth’s atmos…

Yup, the models are smart, but are trained to follow standard human patterns for this type of questions. And even on hackernews vast majority will not think that they would need to correct for buoyancy when actually attempting the experiment in standard conditions.

They very often get popular "tricky" questions wrong because they saw it so many times that they switch from internal reasoning to memorization/retrieval.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#498

Post by IshKebab »

kneegerman wrote: post #471
IshKebab wrote: post #20

Earlier quoted context omitted.

> Or it's time to step back and call it what it is - very good pattern recognition. Or maybe it's time to stop wheeling out this tedious and disingenuous dismissal. Saying it is just "pattern recognition" (or a "stochastic parrot") implies behavioural and performance characteristics that have very clearly been greatly exceeded.

so how much you have riding on nvidia bro?

Nothing. I just use ChatGPT and Claude so I am familiar with their capabilities and limitations.

Imagine if people who had never used VR kept saying it's just a TV on your face, or if people who had never used static types kept saying they're just extra work you have to do, or if people who had never had sex kept saying it's just a way of making babies.

It's a tedious claim when it's so easily disproven by going to a free website and trying it. Why are people so invested in AI being useless that they'll criticise it so confidently without even trying it?

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#499

Post by Jensson »

scrollaway wrote: post #457
stefan_ wrote: post #348

Earlier quoted context omitted.

Why? Did a grocery store self checkout ever fail to calculate sales tax? Do I need to run a study on that? The people selling this could not make a car drive but now its AGI.

A single-purpose state machine not failing to do the single thing it was created to do does not make for the clever retort you think it makes. "AGI": emphasis on "G" for "General". The LLMs are not failing to do generalized tasks, and that they are nondeterministic is not a bug. Just don't use them for calculating sales tax. You wouldn't hire a human to calculate sales tax in their head, so why do you make this a req…

> You wouldn't hire a human to calculate sales tax in their head

Everyone did that 60 years ago, humans are very capable at learning and doing that. Humans built jetplanes, skyscrapers, missiles, tanks, carriers without the help of electronic computers.

Re: 30% drop in O1-preview accuracy when Putnam problems are slightly variated

#500

Post by rrr_oh_man »

whimsicalism wrote: post #113
fldskfjdslkfj wrote: post #111

Earlier quoted context omitted.

> never seen before frontiermath problems How do you know that?

Because that is the whole conceit of how frontiermath is constructed

> Because that is the whole conceit

Freudian typo?

Post reply on HN