arXiv:2603.11749v4 cs.CL

Truth as a Compression Artifact in Language Model Training

Konstantin Krestnikov

제출 2026-11-04

원문 초록

When a transformer is trained from scratch on corpora that solve the same mathematical problem both correctly and incorrectly, which solution does it prefer? The usual test is pairwise, and it is not neutral: when the error can take several forms, a model that merely reproduces training frequencies wins it without preferring anything. We instead score the correct completion against every string the error generator can produce, a share on which frequency matching scores exactly 0.500. At 86M parameters the correct completion holds 59.0% +/- 0.8% of the mass against random errors (59.4% on an independent corpus), where the single realized error gives 77.6% and pair accuracy 87.4%, a three- to fourfold overstatement of the excess above chance; below 86M the primary corpus shows no preference. The coherent false rules of the main corpora are not filtered at any size (43.5--48.8%; the shortfall below 0.5 is a completion-length effect). Two experiments built to identify a compression mechanism raise the pairwise measures while the summed share stays flat or falls: both are negative controls, and compression remains motivation, not a measured mechanism. With the scored false strings held fixed, a false rule whose selector the model can learn moves the correct share from 5 points above the null to 7 below (418M--3.08B), though the correct completion still outweighs the selected string. What these models leave unfiltered is consistent or learnable false structure (one single-rule corruption excepted), not falsehood as such.

제출 이력

확인된 버전만 표시합니다. 전체 제출 이력은 arXiv 원문에서 확인하세요.

  1. arXiv:2603.11749v4제출 시각 미확인

함께 살펴볼 논문

공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.

공통 태그로 연결된 공개 논문이 아직 없습니다.

토론

이 논문과 연결된 포럼 글이 아직 없습니다.