Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

arxiv.org

Post reply on HN
Print view View original

71–80 of 85 posts

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#71

Post by scotty79 »

m_kos wrote: post #61
Workaccount2 wrote: post #56

Earlier quoted context omitted.

"What is bigger, 9.9 or 9.11?"

Gemini 2 Pro and Mistral Chat both think 9.11 is bigger.

Only Gemini 2.0-flash and Claude 3.5-sonnet got this right for me. And only Claude seems to be getting such things with good consistency. It seems to have a strategy for it and applied it well while other models are basically guessing.

Also Deep Seek gets it right similarly to Claude but it's more verbose in standard mode and much more verbose in R1 deep thinking mode. The reasoning is verbose but nearly 100% sensible.

Especially when given follow up question "how about 9.8?"

Gemini-2.0-Flash is also constantly correct when "Think step by step." is appended to the prompt.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#72

Post by fzzzy »

Workaccount2 wrote: post #38
rahidz wrote: post #37

What is so interesting to me is that the reasoning traces for these often have the correct answer, but the model fails to realize it. Problem 3 ("Dry Eye"), R1: "Wait, maybe "cubitus valgus" – no, too long. Wait, three letters each. Let me think again. Maybe "hay fever" is two words but not three letters each. Maybe "dry eye"? "Dry" and "eye" – both three letters. "Dry eye" is a condition. Do they rhyme? "Dry" (d-rye…

The other day I watched Gemini reasoning go in circles for ~30 seconds trying to determine if 9.11 was greater than 9.9. ---- "Realize a direct comparison of tenths is not enough: Wait, this is not right. In 9.9, the tenths digit is 9. In 9.11, the tenths digit is 1. This initial comparison is incorrectly stated. Let's re-examine. Re-examine tenths place comparison (correctly): In 9.9, the tenths digit is 9. In 9.11,…

Training a model to reason that it should write and execute code is the solution here. Synthetic datasets showing lots of math questions followed by reasoning that writes and executes the program for the answer.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#73

Post by enum »

rthnbgrredf wrote: post #69
enum wrote: post #15

Earlier quoted context omitted.

There is likely a little bit of that, but I don't think that's the primary factor that makes these hard. If you look at the reasoning performed by R1 ("Show Thoughts" checkbox), it has no problem splitting and joining candidate words into letters, or even doing the letter-level manipulations. Just to give an example: Take a familiar brand name, seen along roads and highways in the United States. It has five letters,…

I'm still not convinced that this isn't a tokenizer issue. Were you able to find a substantial number of questions that do not fall into the letter countinh or word shuffling domsin - problems that are clearly unrelated to the fundamental tokenizer issue of modern LLMs? Otherwise, I would argue that your paper simply proves that the issue still exists.

It’s not that the benchmark is hard, but that the reasoning models do so much better than the non-reasoning models. That suggests it is testing a capability that reasoning models have that non-reasoning models do not.

Getting to 100% may require tokenization innovation, sure.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#74

Post by Workaccount2 »

windsignaling wrote: post #58

I'm not a fan of these "gotchas" because they don't test for what we really care about. Like counting the number of R's in strawberry, many of these are character-counting or character manipulation problems which tokenization is not well-suited for. I'm sure an engineer could come up with a clever way to train for this, but that seems like optimizing for the wrong thing. IMO these questions go in the wrong direction.…

Someone needs to make a data transformation benchmark.

"Here are a variety of personal documents about John Doe. Fill out the McDonalds job application with information retrieved from the document set."

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#75

Post by Jerrrry »

hto2i4j234234 wrote: post #50
Workaccount2 wrote: post #38

Earlier quoted context omitted.

The other day I watched Gemini reasoning go in circles for ~30 seconds trying to determine if 9.11 was greater than 9.9. ---- "Realize a direct comparison of tenths is not enough: Wait, this is not right. In 9.9, the tenths digit is 9. In 9.11, the tenths digit is 1. This initial comparison is incorrectly stated. Let's re-examine. Re-examine tenths place comparison (correctly): In 9.9, the tenths digit is 9. In 9.11,…

I wonder how much of this arithmetic confusion comes from software versioning numbers where the decimal character is actually a tuple-delimiter.

[dead]

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#76

Post by michaelt »

windsignaling wrote: post #58

I'm not a fan of these "gotchas" because they don't test for what we really care about. Like counting the number of R's in strawberry, many of these are character-counting or character manipulation problems which tokenization is not well-suited for. I'm sure an engineer could come up with a clever way to train for this, but that seems like optimizing for the wrong thing. IMO these questions go in the wrong direction.…

The problem is some of the "gotchas" seem rather important in nontrivial applications.

Imagine a model that isn't sure if 9.11 is greater than 9.9 - which is difficult to reason about, because tokens.

Could such a model coach kids in math? Could it proofread a paper, or sense-check a business plan? Could it summarise a long document about carbon emissions? Could it generate a GUI? Could it spot mistakes in an OCRed document? Spot an off-by-one error or divide-by-zero in computer code?

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#77

Post by szundi »

tkgally wrote: post #42
XCabbage wrote: post #19

This doesn't feel like a "reasoning" challenge. The mental skill required to solve most of these seems to be the ability to loop over all known members of a category like "popular brand names" or "well-known actors" and see if they fit the clue. As a human, you'd expect to fail either because you didn't know a category member (e.g. as a non-American I have no idea WTF "Citgo" is; I could never get the answer to the f…

I agree that recall seems to play an important role in solving these problems. Similar to how the ARC-AGI problems seem to depend on visual perception of shapes and colors. When I come up with the correct answers to such puzzles, I feel subjectively that the answers flashed into my mind, not that I reasoned my way to them.

Your base model is good then

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#78

Post by DiscourseFan »

klipt wrote: post #54
DiscourseFan wrote: post #34

Earlier quoted context omitted.

PhD students aren't necessarily the brightest (which is probably why many of them don't end up with tenure), but studying something at the graduate level does open your eyes to a particular kind of knowledge which is generally unavailable otherwise. So, yes, of course the work is more important than the credential, but the credential often stands in for the work that gives that knowledge.

> which is probably why many of them don't end up with tenure There are far fewer tenure track positions than PhD's, so most of them not ending up with tenure is tautologically guaranteed.

Well yes but it should be hard to get a tenure either way

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#79

Post by empath75 »

Workaccount2 wrote: post #38
rahidz wrote: post #37

What is so interesting to me is that the reasoning traces for these often have the correct answer, but the model fails to realize it. Problem 3 ("Dry Eye"), R1: "Wait, maybe "cubitus valgus" – no, too long. Wait, three letters each. Let me think again. Maybe "hay fever" is two words but not three letters each. Maybe "dry eye"? "Dry" and "eye" – both three letters. "Dry eye" is a condition. Do they rhyme? "Dry" (d-rye…

The other day I watched Gemini reasoning go in circles for ~30 seconds trying to determine if 9.11 was greater than 9.9. ---- "Realize a direct comparison of tenths is not enough: Wait, this is not right. In 9.9, the tenths digit is 9. In 9.11, the tenths digit is 1. This initial comparison is incorrectly stated. Let's re-examine. Re-examine tenths place comparison (correctly): In 9.9, the tenths digit is 9. In 9.11,…

This and it's struggles with spelling questions are both artifacts of tokenization and not really a failure of reasoning. I think there's probably a simple solution that solves both this and the "how many r's are there in strawberry" problem, though I don't know what it would be.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#80

Post by empath75 »

enum wrote: post #40
rahidz wrote: post #37

What is so interesting to me is that the reasoning traces for these often have the correct answer, but the model fails to realize it. Problem 3 ("Dry Eye"), R1: "Wait, maybe "cubitus valgus" – no, too long. Wait, three letters each. Let me think again. Maybe "hay fever" is two words but not three letters each. Maybe "dry eye"? "Dry" and "eye" – both three letters. "Dry eye" is a condition. Do they rhyme? "Dry" (d-rye…

The nature of the problems makes it relatively easy to follow along with the models' reasoning and reasoning errors. For example, on this problem (answer "New England"): > Think of a place in America. Two words, 10 letters altogether. The first five letters read the same forward and backward. The last five letters spell something found in the body. What place is this? R1 fixates on answers of the form "CITY, STATE" a…

o1 high got this after i gave it the hint that the first five letters are not a single word.
Post reply on HN