ZeroPoint: Lightweight Energy-Based Decision Model

원문 초록

Energy-based models (EBMs) assign a scalar energy to every joint value of the observed and predicted variables. In decision models, the prediction is chosen from a finite list of options provided with each input. Inference then reduces to comparing the energies of the options, and training lowers the energy of the correct option relative to the others. A complete decision must also account for the possibility that no option is acceptable. However, the energies learned by a standard EBM define only relative compatibility: adding the same input-dependent quantity to all energies leaves the predictions, the Gibbs distribution, and the usual training losses unchanged. The model therefore has no learned reference level against which to decide whether any option fits. We propose ZeroPoint, an EBM that makes the empty answer a state of the model with energy zero for every input. This fixed state provides an explicit reference for the energies of all listed options, whose non-negative depths measure their compatibility with the input relative to the empty answer. The options and the empty answer share one distribution, in which an option at zero depth receives no probability. The decision to select an option or to abstain is thus part of ordinary inference, trained with the negative log-likelihood loss, rather than a separate rejection rule. The energy function is a small multilingual encoder that reads the input together with all the options and is invariant to their order. Ablations show that most of the benefit comes from training with examples in which no option fits. With the same encoder and data, the fixed zero separates inputs with and without an acceptable option slightly better than a learned rejection class and clearly better than a thresholded softmax, while leaving the choice among options unchanged.