X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

원문 초록

Cross-tokenizer knowledge distillation allows a student to learn from teachers with incompatible vocabularies while preserving its tokenizer and inference architecture. Existing logit-based methods either supervise only the observed token or match full distributions through strict token partitions and heuristic ranking. We identify two limitations: an uncommon-token failure, where task-critical tokens outside the shared vocabulary are suppressed, and over-conservative matching, where exact string equality discards useful correspondences. We propose X-Token, which addresses both through a sparse projection initialized from tokenizer-level string rules and optionally refined during distillation. P-KL removes the partition and projects the student distribution into the teacher vocabulary; H-KL retains the hybrid loss while relaxing matching through the projection. A token-coverage audit guides the choice between them, and both extend naturally to heterogeneous teachers. Across Llama, OLMo, and Gemma students, X-Token achieves the highest average accuracy in all four teacher-student settings among all evaluated cross-tokenizer methods. Complementary teachers provide a further 1.30-point gain, and both variants are faster than the evaluated cross-tokenizer baselines in matched profiling.