FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

Post by rl3 » Sun, Nov 10, 2024, 9:37 AM UTC

youoy wrote: post #39

tux3 wrote: post #23

Earlier quoted context omitted.

Although the answer isn't sent, so it would have to be a very deliberate effort to fish those out of the API chatter and find the right domain expert with 4-10 hours to spend on cracking it Just letting the AI train on its own wrong output wouldn't help. The benchmark already gives them lots of time for trial and error.

Why do people still insist that this is unlikely? Like assuming that the company that payed 15M for chat.com does not have some spare change to pay some graduate students/postdocs to solve some math problems. The publicity of solving such benchmark would definitely raise the valuation so it would 100% be worth it for them...

Post by aiono » Sun, Nov 10, 2024, 9:51 AM UTC

rl3 wrote: post #51

youoy wrote: post #39

Earlier quoted context omitted.

Why do people still insist that this is unlikely? Like assuming that the company that payed 15M for chat.com does not have some spare change to pay some graduate students/postdocs to solve some math problems. The publicity of solving such benchmark would definitely raise the valuation so it would 100% be worth it for them...

Simple: I highly doubt they're willing to risk a scandal that would further tarnish their brand. It's still reeling from last year's drama, in addition to a spate of high-profile departures this year. Not to mention a few articles with insider sources that aren't exactly flattering.

Post by mkl » Sun, Nov 10, 2024, 9:59 AM UTC

llm_trw wrote: post #35

nopinsight wrote: post #34

Earlier quoted context omitted.

What kinds of problems in what domains did you test o1 models with? I found that they are good at logic and math problems but still hallucinate. I didn’t try to stretch test them with hard problems though.

Finding a path between two vertices when given an itinerary of all the edges in a general graph, exactly what I said in the OP.

Post by ak_111 » Sun, Nov 10, 2024, 10:09 AM UTC

TeMPOraL wrote: post #25

agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

> Surprisingly, prediction markets [1] are putting 62% on AI achieving > 85% performance on the benchmark before 2028. Why surprisingly? 2028 is twice as long as capable LLMs existed to date. By "capable" here I mean capable enough to even remotely consider the idea of LLMs solving such tasks in the first place. ChatGPT/GPT-3.5 isn't even 2 years old ! 4 years is a lot of time. It's kind of silly to assume LLM capabi…

Post by andrepd » Sun, Nov 10, 2024, 10:25 AM UTC

sebzim4500 wrote: post #11

agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

>Surprisingly, prediction markets [1] are putting 62% on AI achieving > 85% performance on the benchmark before 2028. Or they know the ancient technique of training on the test set. I know most of the questions are kept secret, but they are being regularly sent over the API to every LLM provider.