Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness

원문 초록

Chain-of-thought (CoT) reasoning enables large language models (LLMs) to solve complex problems by generating intermediate reasoning steps. While much attention has been paid to the length and content of these reasoning chains, far less is known about their internal geometry. We study the geometry of CoT trajectories in the hidden state space of transformer models, formalizing each reasoning chain as a discrete curve in $\mathbb{R}^d$ and characterizing it through spectral, positional, and kinematic geometric functionals. We introduce the effective dimension $d_\rho$ as a measure of trajectory complexity and empirically show that harder tasks exhibit greater effective dimension. Lastly, we explore how kinematic features of the trajectory, mean position, positional dispersion, initial and final hidden states, mean velocity, mean speed, and speed dispersion, can be used to predict solution correctness before generation is complete, and may inform future early-stopping strategies. Experimentally, on 86 mathematical reasoning problems from the MATH500 dataset and three Qwen2.5 model scales (0.5B, 1.5B, and 7B), $d_\rho$ achieves difficulty-prediction AUROC of $0.90$, $0.94$, and $0.96$, respectively. Seven kinematic and positional features predict correctness from only the first $20\%$ of generated tokens, with AUROC between $0.77$ and $0.88$ across scales. These correctness signatures transfer across held-out questions, establishing that the shape of a model's internal reasoning trajectory is a principled window into both task hardness and solution quality.