State space models (SSMs) are an emerging alternative to Transformers. SSMs offer linear-time inference and parallel training, but most implementations make the parallel scan efficient by making the recurrence diagonal: each channel of the hidden state is multiplied by an input-dependent real or complex number. These numbers commute, so the state records which inputs occurred but not their order. We formalize this with representation theory, which explains and fixes the state-tracking failures previously reported for SSMs. The state update defines a linear representation of the monoid of input sequences, and we show that an SSM tracks a group at every sequence length exactly when that group is a quotient of the group generated by its transition matrices. Two consequences follow. Diagonal real- and complex-valued SSMs can track only commutative groups, while the transition matrices of any SSM that can track a group must split into independent blocks over the real, complex, and quaternion numbers. We then introduce HyperSSM, which adds quaternion channels while keeping linear-time inference and parallel training. Trained only on final answers, HyperSSM solves the group tasks of recent state-tracking benchmarks exactly with at most five state numbers. This includes $S_5$ with 32 generators, where previous SSMs are at chance, and HyperSSM generalizes to sequences 25 times longer than those seen in training. Because the theory says what the transitions must encode, the learned weights of the linear recurrence can be interpreted directly. We demonstrate this by recovering the calibration of a real gyroscope, the rotations of imperfect qubit gates and the geometry of protein backbones.
The field of mechanistic interpretability is well-suited to collaborative projects between mathematicians and AI researchers. This work describes one such project. We trained a small transformer to learn the language of the Coxeter group affine A2, best known as the symmetries of the tiling of the Euclidean plane by equilateral triangles. We then formulated and proved a word-theoretic conjecture about affine A2 after observing the attention patterns of the trained model.
Neural Sheaf Diffusion (NSD) learns a task-adapted sheaf geometry for node classification. We connect the study of these learned geometries with Geometric Invariant Theory (GIT) to analyse their representation-theoretic structure and develop mechanisms for geometric control during training. Identifying cellular sheaves with representations of the incidence quiver connects direct-sum decompositions to the harmonic space and motivates the study of trivial or low-complexity summands as potential degeneration modes. We make this perspective concrete by focusing on the trivial representation: on an embedded copy, the sheaf Dirichlet energy reduces to weighted scalar graph energy. The GIT stability framework provides a criterion for excluding such copies within the semistable class. Using its moment-map characterisation of polystability, we construct differentiable surrogates that encourage the corresponding balance conditions. We identify an equal-stalk obstruction to assigning a destabilising weight to the trivial representation and show how nonuniform stalk dimensions remove it. Controlled energy-minimisation experiments reproduce trivial-component formation, while native feature-conditioned NSD experiments on heterophilic graphs show how moment-map regularisation changes the balance and Dirichlet-energy trajectories of learned geometries during training, alongside their classification behaviour.
Geometric and Topological Deep Learning enhance machine learning by exploiting geometric and topological structures in data. In this context, Group Equivariant Non-Expansive Operators (GENEOs) provide a principled framework for encoding symmetries and designing efficient, interpretable neural architectures. Originally introduced in Topological Data Analysis and later adopted in Deep Learning, GENEOs unify geometric and topological approaches and include, as a special case, the operator computing persistence diagrams. Their theoretical foundations rely on group actions, equivariance, and compactness properties of operator spaces. GENEOs are defined between data spaces endowed with transformation groups, referred to as perception pairs. While previous representation results characterized linear GENEOs acting on identical perception pairs, many applications require operators between heterogeneous ones. In this work, we introduce and prove a new algebraic representation theorem for linear GENEOs acting between different perception pairs. Under mild assumptions on data domains and group actions, this result provides a complete characterization of such operators and implicitly yields a computational method to generate all linear GENEOs between two perception pairs. The proof combines tools from the theory of stochastic matrices, group actions, and invariant measures. In our paper, we also show that the space of linear GENEOs is compact and convex, making easier their use for learning and optimization. Our new theorem enlarges the applicability of GENEOs, and allows for their composition with a broader class of current architectures. We show this with an application in image reconstruction.
Equivariant linear maps cannot mix non-isomorphic irreducible representations, whereas nonlinear activations can. We develop a degree-resolved version of this nonlinear interaction picture and use the resulting block inventory to construct Specht Triple, a residual on unordered graph triples. On a synthetic structural benchmark with a frozen GIN backbone, Specht Triple reduces aggregate standardised test MSE by 11.05% relative to a parameter-matched Projector Control using the same irreducible projectors and complete linear equivariant space. The results support irreducible decomposition as an inductive bias for nonlinear higher-order equivariant processing.