Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

epochai.org

Post reply on HN
Print view View original

21–30 of 110 posts

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#21

Post by westurner »

westurner wrote: post #20

ScholarlyArticle: "FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI" (2024) https://arxiv.org/abs/2411.04872 .. https://epochai.org/frontiermath/the-benchmark : > [Not even 2%] > Abstract: We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of mo…

Additional AI math benchmarks:

- "TheoremQA: A Theorem-driven [STEM] Question Answering dataset" (2023) https://github.com/TIGER-AI-Lab/TheoremQA

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#23

Post by tux3 »

sebzim4500 wrote: post #11
agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

>Surprisingly, prediction markets [1] are putting 62% on AI achieving > 85% performance on the benchmark before 2028. Or they know the ancient technique of training on the test set. I know most of the questions are kept secret, but they are being regularly sent over the API to every LLM provider.

Although the answer isn't sent, so it would have to be a very deliberate effort to fish those out of the API chatter and find the right domain expert with 4-10 hours to spend on cracking it

Just letting the AI train on its own wrong output wouldn't help. The benchmark already gives them lots of time for trial and error.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#24

Post by equestria »

agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

Market size matters. There's a whopping total of 71 bidders on that.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#25

Post by TeMPOraL »

agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

> Surprisingly, prediction markets [1] are putting 62% on AI achieving > 85% performance on the benchmark before 2028.

Why surprisingly?

2028 is twice as long as capable LLMs existed to date. By "capable" here I mean capable enough to even remotely consider the idea of LLMs solving such tasks in the first place. ChatGPT/GPT-3.5 isn't even 2 years old!

4 years is a lot of time. It's kind of silly to assume LLM capabilities have already bottomed out.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#26

Post by bravura »

Regarding keeping the test set private to avoid contamination, the comments about leakage are spot on. The real test set should always be the future.

We should evaluate LLMs on text from beyond their knowledge cutoff date, by computing their per-byte perplexity or per-byte compression ratio. There's a deep theoretical connection between compression and learning.

The intuition here is that being able to predict the future of science (or any topic, really) is indicative of true understanding. Slightly more formally: When ICLR 2025 announces and publishes the accepted papers, Yoshua Bengio is less surprised/perplexed by what's new than a fresh PhD student. And Terence Tao is less surprised/perplexed by what will be proven in math in the next 10 years than a graduate student in a related field.

This work has it right: https://ar5iv.labs.arxiv.org/html//2402.00861

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#27

Post by ekianjo »

TeMPOraL wrote: post #25
agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

> Surprisingly, prediction markets [1] are putting 62% on AI achieving > 85% performance on the benchmark before 2028. Why surprisingly? 2028 is twice as long as capable LLMs existed to date. By "capable" here I mean capable enough to even remotely consider the idea of LLMs solving such tasks in the first place. ChatGPT/GPT-3.5 isn't even 2 years old ! 4 years is a lot of time. It's kind of silly to assume LLM capabi…

Sure but it is also reasonable to consider that the pace of progress is not always exponential or even linear at best. Diminishing returns are a thing and we already know that a 405b model is not 5 times better than a 70b model.

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#28

Post by 3abiton »

bravura wrote: post #26

Regarding keeping the test set private to avoid contamination, the comments about leakage are spot on. The real test set should always be the future . We should evaluate LLMs on text from beyond their knowledge cutoff date, by computing their per-byte perplexity or per-byte compression ratio. There's a deep theoretical connection between compression and learning. The intuition here is that being able to predict the f…

Interesting take sounds like MDL (Minimum description length) for LLMs!

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#29

Post by nopinsight »

llm_trw wrote: post #3
agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

These benchmarks are entirely pointless. The people making them are specialists attempting to apply their skills to areas unrelated to LLM performance, a bit like a sprinter making a training regimen for a fighter jet. What matters is the data structures that underlie the problem space - graph traversal. First, finding a path between two nodes; second, identifying the most efficient path; and third, deriving implicit…

> Currently, all LLMs are so limited that they struggle with journeys longer than four edges, even when given a full itinerary of all edges in the graph.

This is probably not the case for LLMs in the o1 series and possibly Claude 3.5 Sonnet. Have you tested them on this claim?

Re: FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

#30

Post by agucova »

light_hue_1 wrote: post #12
agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

If I was going to bet, I would bet yes, they will reach above 85% performance. The problem with all benchmarks, one that we just don't how to solve, is leakage. Systematically, LLMs are much better at benchmarks created before they were trained than after. There are countless papers that show significant leakage between training and test sets for models. This is in part why so many LLMs are so strong according to ben…

This benchmark’s questions and answers will be kept fully private, and the benchmark will only be run by Epoch. Short of the companies fishing out the questions from API logs (which seems quite unlikely), this shouldn’t be a problem.
Post reply on HN