This doesn't feel like a "reasoning" challenge. The mental skill required to solve most of these seems to be the ability to loop over all known members of a category like "popular brand names" or "well-known actors" and see if they fit the clue. As a human, you'd expect to fail either because you didn't know a category member (e.g. as a non-American I have no idea WTF "Citgo" is; I could never get the answer to the f…
PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
41–50 of 85 posts
Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
#42This doesn't feel like a "reasoning" challenge. The mental skill required to solve most of these seems to be the ability to loop over all known members of a category like "popular brand names" or "well-known actors" and see if they fit the clue. As a human, you'd expect to fail either because you didn't know a category member (e.g. as a non-American I have no idea WTF "Citgo" is; I could never get the answer to the f…
Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
#43Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
#44What is so interesting to me is that the reasoning traces for these often have the correct answer, but the model fails to realize it. Problem 3 ("Dry Eye"), R1: "Wait, maybe "cubitus valgus" – no, too long. Wait, three letters each. Let me think again. Maybe "hay fever" is two words but not three letters each. Maybe "dry eye"? "Dry" and "eye" – both three letters. "Dry eye" is a condition. Do they rhyme? "Dry" (d-rye…
The other day I watched Gemini reasoning go in circles for ~30 seconds trying to determine if 9.11 was greater than 9.9. ---- "Realize a direct comparison of tenths is not enough: Wait, this is not right. In 9.9, the tenths digit is 9. In 9.11, the tenths digit is 1. This initial comparison is incorrectly stated. Let's re-examine. Re-examine tenths place comparison (correctly): In 9.9, the tenths digit is 9. In 9.11,…
Wonder if the same thing would have happened with 9.12.
What was your original prompt?
Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
#45The term "PhD Knowledge" makes me cringe. PhD is about contribution to a field, not a standardized test score or a weird status-bearing substitute for IQ.
In this case it is a shorthand for specialist level knowledge. Thre kind of information you would expect a person pursuing a PhD in a specific field to know but someone not doing that to not know. For example how in the “Humanity's Last Exam” AI evaluation dataset they have this question: “Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of…
Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
#46Are these really reasoning challenges? Seems like they are really solved via brute force or guess and check
Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
#47Earlier quoted context omitted.
In this case it is a shorthand for specialist level knowledge. Thre kind of information you would expect a person pursuing a PhD in a specific field to know but someone not doing that to not know. For example how in the “Humanity's Last Exam” AI evaluation dataset they have this question: “Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of…
That's a jargon and search problem at best. I'm not convinced at all.
What are you not convinced about?
Are you not convinced this is what they mean?
Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
#48What is so interesting to me is that the reasoning traces for these often have the correct answer, but the model fails to realize it. Problem 3 ("Dry Eye"), R1: "Wait, maybe "cubitus valgus" – no, too long. Wait, three letters each. Let me think again. Maybe "hay fever" is two words but not three letters each. Maybe "dry eye"? "Dry" and "eye" – both three letters. "Dry eye" is a condition. Do they rhyme? "Dry" (d-rye…
Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
#49What is so interesting to me is that the reasoning traces for these often have the correct answer, but the model fails to realize it. Problem 3 ("Dry Eye"), R1: "Wait, maybe "cubitus valgus" – no, too long. Wait, three letters each. Let me think again. Maybe "hay fever" is two words but not three letters each. Maybe "dry eye"? "Dry" and "eye" – both three letters. "Dry eye" is a condition. Do they rhyme? "Dry" (d-rye…
It feels like lot of the reasoning tokens go to waste on pure brute force approach - plugging in numbers and evaluating and comparing against the answer. "Nope, that didn't work, let's try 4 instead of 6 this time", etc. What if the reward function instead focuses on diversity of procedures within a token budged (10k - 20k tokens). I.e. RL rewards the model in trying different methods or generating different hypothes…
Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models
#50What is so interesting to me is that the reasoning traces for these often have the correct answer, but the model fails to realize it. Problem 3 ("Dry Eye"), R1: "Wait, maybe "cubitus valgus" – no, too long. Wait, three letters each. Let me think again. Maybe "hay fever" is two words but not three letters each. Maybe "dry eye"? "Dry" and "eye" – both three letters. "Dry eye" is a condition. Do they rhyme? "Dry" (d-rye…
The other day I watched Gemini reasoning go in circles for ~30 seconds trying to determine if 9.11 was greater than 9.9. ---- "Realize a direct comparison of tenths is not enough: Wait, this is not right. In 9.9, the tenths digit is 9. In 9.11, the tenths digit is 1. This initial comparison is incorrectly stated. Let's re-examine. Re-examine tenths place comparison (correctly): In 9.9, the tenths digit is 9. In 9.11,…