The Long Multiplication Benchmark: A Serious Challenge for Modern LLMs

The Long Multiplication Benchmark evaluates Large Language Models (LLMs) on their ability to handle and utilize long contexts to solve multiplication problems. Despite long multiplication requiring only 2500 tokens for two seven-digit numbers, no modern LLM can solve even two five-digit numbers, revealing a significant gap in their context utilization capabilities compared to humans.