Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures

원문 초록

Do language models encode the same prompted concepts in structurally compatible ways, beyond generic similarity induced by shared questions? We study contrastive-difference centered kernel alignment ($\mathrm{CKA}_{\Delta}$), which compares per-question activation differences between opposing concept instructions. A matched same-concept versus cross-concept comparison separates the question of concept-conditioned structure from the magnitude of representational similarity. Across four checkpoints and eight personality-related traits, a fixed-final-layer analysis gives mean same- and cross-trait $\mathrm{CKA}_{\Delta}$ of 0.7259 and 0.6828. Trait-label randomization detects correspondence conditional on this model and prompt set ($p=1/10001$, one-sided), but a model-and-trait-weighted 95% interval for the difference is $[-0.0330,0.1017]$, limiting broader generalization. Strict question-group holdouts retain high same-trait linear recoverability. More stringently, maps and preprocessing fitted on seven traits transfer a source polarity classifier for the eighth trait to handwritten test questions: mean affine and orthogonal accuracies are 88.10% and 88.57%, versus 47.70% for shuffled pairing. Performance varies substantially across traits and model directions. Synthetic controls show why moderate CKA and high affine accuracy alone do not establish functional equivalence. The results support a scoped diagnostic of concept-specific structure and held-out linear recoverability; they do not establish universal invariance or a robust advantage over raw CKA across architectures.