Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

arxiv.org

Post reply on HN
Print view View original

21–30 of 85 posts

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#21

Post by enum »

XCabbage wrote: post #19

This doesn't feel like a "reasoning" challenge. The mental skill required to solve most of these seems to be the ability to loop over all known members of a category like "popular brand names" or "well-known actors" and see if they fit the clue. As a human, you'd expect to fail either because you didn't know a category member (e.g. as a non-American I have no idea WTF "Citgo" is; I could never get the answer to the f…

It's definitely U.S. centric.

But, I do think this is reasoning. It requires recall, but anything other than pure logic puzzles do. For example, on a competition math problem or a programming problem, No person or LLM is inventing well-known lemmas and algorithms from first-principles.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#22

Post by XCabbage »

enum wrote: post #21
XCabbage wrote: post #19

This doesn't feel like a "reasoning" challenge. The mental skill required to solve most of these seems to be the ability to loop over all known members of a category like "popular brand names" or "well-known actors" and see if they fit the clue. As a human, you'd expect to fail either because you didn't know a category member (e.g. as a non-American I have no idea WTF "Citgo" is; I could never get the answer to the f…

It's definitely U.S. centric. But, I do think this is reasoning. It requires recall, but anything other than pure logic puzzles do. For example, on a competition math problem or a programming problem, No person or LLM is inventing well-known lemmas and algorithms from first-principles.

It's not just that it requires recall. It's that it requires no non-trivial thought beyond recall.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#23

Post by enum »

XCabbage wrote: post #22
enum wrote: post #21

Earlier quoted context omitted.

It's definitely U.S. centric. But, I do think this is reasoning. It requires recall, but anything other than pure logic puzzles do. For example, on a competition math problem or a programming problem, No person or LLM is inventing well-known lemmas and algorithms from first-principles.

It's not just that it requires recall. It's that it requires no non-trivial thought beyond recall.

I think what you mean is that once you've managed to recall, checking constraints is easy. Remarkably, a few people are much better at this than others. They are able to think fast and execute an explicit mental search over a very small number of plausible candidates. Other people take forever. Seems to be the case for models too.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#25

Post by glial »

mkoubaa wrote: post #24

The term "PhD Knowledge" makes me cringe. PhD is about contribution to a field, not a standardized test score or a weird status-bearing substitute for IQ.

Also - at least in my experience - doing a PhD makes a person more aware of the boundaries of knowledge, and therefore generally pretty skeptical, and capable of being usefully critical, about developments in their field.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#27

Post by ks2048 »

enum wrote: post #2

Results and dataset explorer here: https://huggingface.co/spaces/nuprl/verbal-reasoning-challen...

Is there some standard markup assumed for LLMs? I just looked at a couple and saw markdown-like text (which I expected), but also "$\boxed{foo}$". Looks like LaTeX, but is there some subset of LaTeX LLMs are assumed to know/use?

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#28

Post by enum »

mkoubaa wrote: post #24

The term "PhD Knowledge" makes me cringe. PhD is about contribution to a field, not a standardized test score or a weird status-bearing substitute for IQ.

There is a trend of building hard benchmarks for LLMs, such as GPQA or Humanity's Last Exam, with questions crafted by PhDs. Ergo, most people cannot understand what these benchmarks are testing. If you're an expert in something, and if you're on HN you may be an expert in some sub-field of CS, I recommend trying to solve the questions from your field on Humanity's Last Exam. I took a crack at the Programming Languages questions myself.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#29

Post by levocardia »

mkoubaa wrote: post #24

The term "PhD Knowledge" makes me cringe. PhD is about contribution to a field, not a standardized test score or a weird status-bearing substitute for IQ.

You cannot get a PhD (in the US) without also passing an examination that, nominally at least, demonstrates you have broad and deep knowledge about your field. The qualifying exam is not the most important part, but it is a necessary part.

Re: PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models

#30

Post by krisoft »

mkoubaa wrote: post #24

The term "PhD Knowledge" makes me cringe. PhD is about contribution to a field, not a standardized test score or a weird status-bearing substitute for IQ.

In this case it is a shorthand for specialist level knowledge. Thre kind of information you would expect a person pursuing a PhD in a specific field to know but someone not doing that to not know.

For example how in the “Humanity's Last Exam” AI evaluation dataset they have this question: “Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.”

That’s the kind of “PhD knowledge” which this evaluation does not require.

Post reply on HN