Self-Modeling Interventions can Modulate Emergent Misalignment

원문 초록

Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits, suggesting that EM operates through disruption of the model's aligned character rather than direct learning of harmful content. Motivated by this connection, we study self-modeling interventions: specifically self-recognition and self-reports as character-targeted interventions that are distinct from existing in-training defenses. We conduct two-stage finetuning experiments across three models (GPT-4.1, Qwen2.5-32B-Instruct, Seed-OSS-36B-Instruct) and multiple EM datasets to compare self-recognition finetuning against benign finetuning baselines (correct domain-specific data, general knowledge, and word counting), finding it an effective defense in both reversal and prevention settings. For prevention, only self-recognition finetuning consistently reduces misalignment without exacerbating any individual metric, suggesting that character fortification specifically drives prevention. We also conduct in-training experiments across these models to compare the effectiveness of self-reports with benign interleaving and inoculation prompting. We provide further evidence for EM's relation to the LLM's default character by showing that EM finetuning leads to fragmentation of LLM's identity self-reports, reduces self-recognition performance to about chance, artificially corrupting self-recognition exacerbates misalignment caused by EM finetuning, and finetuning on self-reports from EM finetuned models can transfer misalignment. Together, these findings reframe EM not as the adoption of a coherent misaligned persona but as the destabilization of aligned character.