OpenAI-o1 AB Testing: Does the o1 model really do good reasoning in math problem solving?
원문 초록
The Orion-1 model by OpenAI is claimed to have more robust logical reasoning capabilities than previous large language models. However, some suggest the excellence might be partially due to the model "memorizing" solutions, resulting in less satisfactory performance when prompted with problems not in the training data. We conduct a comparison experiment using two datasets: one consisting of International Mathematics Olympiad (IMO) problems, which is easily accessible; the other one consisting of Chinese National Team Training camp (CNT) problems, which have similar difficulty but not as publically accessible. We label the response for each problem and compare the performance between the two datasets. We conclude that there is no significant evidence to show that the model relies on memorizing problems and solutions. Also, we perform case studies to analyze some features of the model's response.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games
Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models
Interleaved Reasoning for Large Language Models via Reinforcement Learning
Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
Personalized Exercise Recommendation with Semantically-Grounded Knowledge Tracing
Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
토론
이 논문과 연결된 포럼 글이 아직 없습니다.