OraclePhys: A Systematic Framework for LLM Fine-Tuning on Structural Mechanics

원문 초록

What fine-tuning teaches a model is usually diagnosed after the fact. We make it an experimental variable. OraclePhys has three parts. OraclePhys-Bench is a structural-mechanics benchmark whose finite-element oracle grades every answer and counterfactual edit. OraclePhys-30K holds seven answer forms over byte-identical structure descriptions. A controlled training study compares the forms and verifier roles. Two findings follow. First, within this task family the answer form of the label determines what fine-tuning teaches. Ranking supervision lifts held-out localization of the most-deforming story on taller structures from 0.25 to 0.86, a peak-value label yields at best a partial predictor, and a pass/fail label none. This ranking-versus-peak gap replicates on a second domain, a second model family and a paraphrased surface. Second, the gap between our standard SFT and GRPO runs is mostly a recipe effect. With learning rate and step budget matched, the two reach held-out accuracies within 0.03 of each other. At higher rates, GRPO falls below its start while SFT does not. Both also teach the model to apply edits given as instructions. The trained 8B matches a specialist and exceeds a frontier LLM. Both domains are linear, steady-state systems with synthetic text inputs.