Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism
원문 초록
Large language models (LLMs) take advantage of step-by-step reasoning instructions, e.g., chain-of-thought (CoT) prompting. Building on this, their ability to perform CoT-style reasoning robustly is of interest from a probing perspective. In this study, we inspect the step-by-step reasoning ability of LLMs with a focus on negation, which is a core linguistic phenomenon that is difficult to process. In particular, we introduce several controlled settings (e.g., reasoning in case of fictional entities) to evaluate the logical reasoning abilities of the models. We observed that dozens of modern LLMs were not robust against lexical negation (e.g., plausible ->implausible) when performing CoT-style reasoning, and the results highlight unique limitations in each LLM family.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Negation: A Pink Elephant in the Large Language Models' Room?
Empowering LLMs with Logical Reasoning: A Comprehensive Survey
Logical Consistency of Large Language Models in Fact-checking
Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic
SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models
EviNet: Evidential Reasoning Network for Resilient Graph Learning in the Open and Noisy Environments
토론
이 논문과 연결된 포럼 글이 아직 없습니다.