FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

Post by llm_trw » Sun, Nov 10, 2024, 4:55 AM UTC

nopinsight wrote: post #29

llm_trw wrote: post #3

Earlier quoted context omitted.

These benchmarks are entirely pointless. The people making them are specialists attempting to apply their skills to areas unrelated to LLM performance, a bit like a sprinter making a training regimen for a fighter jet. What matters is the data structures that underlie the problem space - graph traversal. First, finding a path between two nodes; second, identifying the most efficient path; and third, deriving implicit…

> Currently, all LLMs are so limited that they struggle with journeys longer than four edges, even when given a full itinerary of all edges in the graph. This is probably not the case for LLMs in the o1 series and possibly Claude 3.5 Sonnet. Have you tested them on this claim?

Post by llm_trw » Sun, Nov 10, 2024, 5:37 AM UTC

NitpickLawyer wrote: post #32

llm_trw wrote: post #19

Earlier quoted context omitted.

If a model can't inately reason over 5 steps in a simple task but produces a flawless 500 step proof you either have divine intervention or memorisation.

AlphaGeometry has entered the chat. Also, AIMOv2 is doing stage 2 of their math challenge, they are now at "national olympics" level of difficulty. They have a new set of questions. Last year's winner (27/50 points) got 2/50 on the new set. In the first 3 weeks of the competition the top score is 10/50 on the new set, mostly with Qwen2.5-math. Given that this is a purposefully made new set of problems, and according…

Post by llm_trw » Sun, Nov 10, 2024, 5:46 AM UTC

nopinsight wrote: post #34

llm_trw wrote: post #31

Earlier quoted context omitted.

Yes, they also fail. I've found the original gpt4 to be the most consistent. One of these days I'll spend the couple of thousands needed to benchmark all the top models and see how they actually perform on a task which can't be gamed.

What kinds of problems in what domains did you test o1 models with? I found that they are good at logic and math problems but still hallucinate. I didn’t try to stretch test them with hard problems though.

Post by slashdave » Sun, Nov 10, 2024, 6:12 AM UTC

TeMPOraL wrote: post #25

agucova wrote: post #2

For some context on why this is important: this benchmark was designed to be extremely challenging for LLMs, with problems requiring several hours or days of work by expert mathematicians. Currently, LLMs solve 2% of problems in the set (which is kept private to prevent contamination). They even provide a quote from Terence Tao, which helped create the benchmark (alongside other Field medalists and IMO question write…

> Surprisingly, prediction markets [1] are putting 62% on AI achieving > 85% performance on the benchmark before 2028. Why surprisingly? 2028 is twice as long as capable LLMs existed to date. By "capable" here I mean capable enough to even remotely consider the idea of LLMs solving such tasks in the first place. ChatGPT/GPT-3.5 isn't even 2 years old ! 4 years is a lot of time. It's kind of silly to assume LLM capabi…

Post by youoy » Sun, Nov 10, 2024, 6:38 AM UTC

tux3 wrote: post #23

sebzim4500 wrote: post #11

Earlier quoted context omitted.

>Surprisingly, prediction markets [1] are putting 62% on AI achieving > 85% performance on the benchmark before 2028. Or they know the ancient technique of training on the test set. I know most of the questions are kept secret, but they are being regularly sent over the API to every LLM provider.

Although the answer isn't sent, so it would have to be a very deliberate effort to fish those out of the API chatter and find the right domain expert with 4-10 hours to spend on cracking it Just letting the AI train on its own wrong output wouldn't help. The benchmark already gives them lots of time for trial and error.