Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

epochai.org

Post reply on HN
Print view View original

41–50 of 110 posts

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#41

Post by youoy »

llm_trw wrote: post #3
agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

These benchmarks are entirely pointless. The people making them are specialists attempting to apply their skills to areas unrelated to LLM performance, a bit like a sprinter making a training regimen for a fighter jet. What matters is the data structures that underlie the problem space - graph traversal. First, finding a path between two nodes; second, identifying the most efficient path; and third, deriving implicit…

Not to mention that math proofs are more than graph trasversals... (Although maybe simple math problems are not) There is the problem of extracting the semantics of math formalisms. This is easier in day to day language, I don't know to what extent LLMs can also extract the semantics and relations of different mathematical abstractions.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#42

Post by olivermuty »

Workaccount2 wrote: post #40
slashdave wrote: post #37

Earlier quoted context omitted.

Except LLM capabilities have already peaked. Scaling has rapidly diminishing returns.

I have yet to see any published evidence of that.

Since you go that route, do you have published evidence that shows they HAVENT entered the top of the S-curve?

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#43

Post by benchmarkist »

agucova wrote: post #30
light_hue_1 wrote: post #12

Earlier quoted context omitted.

If I was going to bet, I would bet yes, they will reach above 85% performance. The problem with all benchmarks, one that we just don't how to solve, is leakage. Systematically, LLMs are much better at benchmarks created before they were trained than after. There are countless papers that show significant leakage between training and test sets for models. This is in part why so many LLMs are so strong according to ben…

This benchmark’s questions and answers will be kept fully private, and the benchmark will only be run by Epoch. Short of the companies fishing out the questions from API logs (which seems quite unlikely), this shouldn’t be a problem.

I looked at the sample questions and even if they get the questions there is no way they will figure out the answers without making significant breakthroughs in understanding mathematics and logic.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#44

Post by benchmarkist »

aurareturn wrote: post #36
mkl wrote: post #18

Earlier quoted context omitted.

Most humans can't solve these problems, so it's certainly possible to imagine a legitimate AGI that can't either.

But humans can solve these problems given enough time and domain knowledge. An LLM would never be able to solve them unless they get smarter. Thats the point. It’s not about whether a random human can solve them. It’s whether AI, in general, can. Humans, in general, have proven to be able to solve them already.

That's correct. Thanks for clarifying for me because I have gotten tired with the comparison to "99% of humans can't do this" as a counter-argument to AI hype criticism.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#45

Post by benchmarkist »

nerdponx wrote: post #16
benchmarkist wrote: post #8

Very cool. It'll be nice to have a benchmark that can be used to validate abstract reasoning capabilities because the hype is really starting to get out of hand.

I wonder if the best benchmark is a Prolog program that generates tests of logical reasoning. You could have a functionally infinite stream of test cases!

You could but most LLMs can't solve sudoku puzzles even though the training corpus already contains books on logic, constraint propagation, and state space exploration with backtracking.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#46

Post by benchmarkist »

sebzim4500 wrote: post #10
benchmarkist wrote: post #8

Very cool. It'll be nice to have a benchmark that can be used to validate abstract reasoning capabilities because the hype is really starting to get out of hand.

I mean, this benchmark is really hard. I don't think it's a requirement that a system claiming to be AGI should be able to solve these problems, 99.99% of humans can't either.

An AGI is often claimed to be a general purpose problem solver and these are exactly the types of problems that a general purpose problem solver would be able to solve if given access to a mathematical library. All existing LLMs have been trained on abstract mathematics and logic but it is obvious that they are incapable of abstract logical reasoning, e.g. solving sudoku puzzles.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#47

Post by edanm »

olivermuty wrote: post #42
Workaccount2 wrote: post #40

Earlier quoted context omitted.

I have yet to see any published evidence of that.

Since you go that route, do you have published evidence that shows they HAVENT entered the top of the S-curve?

What kind of evidence could convince you?

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#48

Post by llm_trw »

aurareturn wrote: post #36
mkl wrote: post #18

Earlier quoted context omitted.

Most humans can't solve these problems, so it's certainly possible to imagine a legitimate AGI that can't either.

But humans can solve these problems given enough time and domain knowledge. An LLM would never be able to solve them unless they get smarter. Thats the point. It’s not about whether a random human can solve them. It’s whether AI, in general, can. Humans, in general, have proven to be able to solve them already.

It is very much an open question just what an llm can solve when allowed to generate an indefinite number of intermediate tokens and allowed to sample an arbitrary amount of text to ground itself.

There are currently no tools that let llms do this and no one is building the tools for answering open ended questions.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#49

Post by llm_trw »

youoy wrote: post #39
tux3 wrote: post #23

Earlier quoted context omitted.

Although the answer isn't sent, so it would have to be a very deliberate effort to fish those out of the API chatter and find the right domain expert with 4-10 hours to spend on cracking it Just letting the AI train on its own wrong output wouldn't help. The benchmark already gives them lots of time for trial and error.

Why do people still insist that this is unlikely? Like assuming that the company that payed 15M for chat.com does not have some spare change to pay some graduate students/postdocs to solve some math problems. The publicity of solving such benchmark would definitely raise the valuation so it would 100% be worth it for them...

Any benchmark which isn't dynamically generated is useless for that very reason.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#50

Post by TeMPOraL »

ekianjo wrote: post #27
TeMPOraL wrote: post #25

Earlier quoted context omitted.

> Surprisingly, prediction markets [1] are putting 62% on AI achieving > 85% performance on the benchmark before 2028. Why surprisingly? 2028 is twice as long as capable LLMs existed to date. By "capable" here I mean capable enough to even remotely consider the idea of LLMs solving such tasks in the first place. ChatGPT/GPT-3.5 isn't even 2 years old ! 4 years is a lot of time. It's kind of silly to assume LLM capabi…

Sure but it is also reasonable to consider that the pace of progress is not always exponential or even linear at best. Diminishing returns are a thing and we already know that a 405b model is not 5 times better than a 70b model.

Yes, but!

Exponential pace of progress isn't usually just one thing; if you zoom in, any particular thing may plateau, but its impact compounds in enabling growth of successors, variations, and related inventions. Nor is it a smooth curve, if you look closely. I feel statements like "a 405b model is not 5 times better than a 70b model" are zooming in on a specific class of models so much you can see the pixels of the pixel grid. There's plenty of open and promising research in tweaking the current architecture in training or inference (see e.g. other thread from yesterday[0]), on top of changes to architecture, methodology, methods of controlling or running inference on exiting models by lobotomizing them or grafting networks to networks, etc. The field is burning hot right now, we're counting space between incremental improvements and interesting research directions in weeks. The overall exponent of "language models" power may just well continue when you zoom out a little bit further.

--

[0] - https://news.ycombinator.com/item?id=42093112

Post reply on HN