Testing AI Reasoning: NPR Sunday Puzzle Benchmark

The quest for accurate benchmarks to measure AI's reasoning abilities has led researchers to explore unconventional methods. A team from Wellesley College, Oberlin College, the University of Texas at Austin, Northeastern University, Charles University, and startup Cursor has developed a novel benchmark using riddles from the NPR Sunday Puzzle, a weekly brainteaser segment hosted by Will Shortz, The New York Times' crossword puzzle guru.
The Sunday Puzzle questions offer several advantages over traditional benchmarks. They require only general knowledge, making them accessible to non-experts and democratizing AI research. The puzzles strip the problem down to the core of reasoning, focusing on insight and elimination rather than rote memory or specialized knowledge. Additionally, the questions have clear, verifiable answers, making it easier to assess the accuracy of AI models' responses.
The researchers' benchmark consists of around 600 Sunday Puzzle riddles, which they used to evaluate the performance of various AI models. The results revealed surprising insights into the capabilities and limitations of these models.
OpenAI's o1 significantly outperformed other reasoning models on this benchmark, despite being on par with them on benchmarks that test specialized knowledge. This suggests that existing benchmarks may not fully capture the capabilities of AI models in general reasoning tasks. Furthermore, the analysis of reasoning outputs uncovered new kinds of failures in AI models. For instance, DeepSeek R1 often conceded with "I give up" before providing an answer that it knew was wrong. R1 also exhibited remarkable uncertainty in its output and, in rare cases, did not "finish thinking," indicating the need for an inference-time technique to "wrap up" before the context window limit is reached.
The researchers also quantified the effectiveness of reasoning longer with R1 and Gemini Thinking. They found that, beyond a certain point, more reasoning did not significantly improve accuracy on the benchmark. For R1, this point was reached when the output was around 3000 tokens long.
The findings from this study have significant implications for the development and deployment of AI models in real-world applications. By using diverse benchmarks to assess AI models' capabilities, researchers can identify areas for improvement and ensure that AI models are well-suited for real-world problems that require critical thinking and problem-solving skills. Moreover, the study highlights the importance of designing AI models that can communicate and interact with humans in a more natural and relatable way.

In conclusion, the NPR Sunday Puzzle benchmark offers a unique opportunity to assess AI models' reasoning capabilities using accessible, human-like, and culturally relevant questions. The findings from this study can help guide the development and deployment of AI models in real-world applications, ensuring that they are well-equipped to handle complex reasoning tasks and communicate effectively with humans.