Multilingual large language models often produce inconsistent reasoning and answers for semantically equivalent prompts in different languages. Prior work suggests that intermediate representations can be relatively language-agnostic, but generation becomes increasingly language-specific as models commit to discrete output tokens. This is problematic because language-specific lexical choices can cause semantically equivalent reasoning paths to diverge across languages. These divergences motivate searching for a cross-lingual alignment signal that is less tied to any single vocabulary item or script. We propose SOLAR, an auxiliary objective for supervised fine-tuning that aligns soft-token representations across languages, using English as a pivot. Soft tokens are probability-weighted mixtures over the vocabulary embeddings, yielding continuous representations that can aggregate information from semantically related tokens across languages. We then align each non-English soft-token summary to its English counterpart in the shared embedding space. Across four multilingual reasoning benchmarks, SOLAR improves accuracy by up to +17.7 points over the base model and +3.8 over standard supervised fine-tuning, with the largest gains on low-resource languages. SOLAR also strengthens final-layer cross-lingual similarity and substantially reduces language-cluster separability, suggesting that aligning soft-token representations helps preserve shared semantic structure during multilingual reasoning.
There has been significant recent progress in algorithms for approximation of Nash equilibrium in large two-player zero-sum imperfect-information games and exact computation of Nash equilibrium in multiplayer strategic-form games. While counterfactual regret minimization and fictitious play are scalable to large games and have convergence guarantees in two-player zero-sum games, they do not guarantee convergence to Nash equilibrium in multiplayer games. Recently, an approach has been presented for exact computation of Nash equilibrium in multiplayer imperfect-information games that solves a quadratically constrained program based on a nonlinear complementarity problem formulation derived from the sequence-form game representation. This formulation was solved using Gurobi's nonconvex quadratic solver, which employs spatial branch-and-bound to iteratively refine variable bounds by solving convex relaxations of bilinear terms. During presolve, Gurobi introduces auxiliary variables and, in some cases, binary variables, leading to an internal mixed-integer quadratically constrained program. This approach was demonstrated to outperform prior algorithms from the Gambit software suite and quickly solve three-player Kuhn poker after removal of dominated actions; however, the algorithm was not able to solve the full version of the game within 24 hours. In this paper, we derive finite bounds on slack and multiplier variables in the nonlinear complementarity formulation. These bounds can strengthen the convex relaxations used within spatial branch-and-bound. We compare uniform corollary bounds with finer proposition-based bounds on full three-player Kuhn poker. All eight bounded configurations return numerically verified equilibria in under 17 seconds, while the unbounded formulation finds no feasible solution within 300 seconds. The finest bounds do not consistently yield the fastest computation.
Quantum low-density parity-check codes promise low-overhead error correction, but high-performing constructions remain hard to find. Here we introduce structured concept evolution, in which a large language model (LLM) evolves construction programs and their internal search strategies by modifying the finite group, protograph shapes or entry-assignment rule. We search for codes with fewer than $600$ data qubits and check weight at most $12$, optimizing $kd^2/n$ or the circuit-level logical error rate. Our method improves on published lifted-product benchmarks in selected parameter regimes, with exactly certified distances for the leading results. Circuit-level optimization typically yields lower logical error rates than $kd^2/n$ optimization, often despite smaller code distances. With the same evolution framework and step budget, the best LLM outperforms all random-mutation and hill-climbing runs on both objectives. Our results point to LLM-driven program search as an approach to designing general quantum LDPC codes.
Multi-zone HVAC control is a spatial decision problem in which indoor thermal evolution and control decisions depend not only on outdoor conditions and internal heat gains but also on zone layout, physical adjacency, and delayed thermal interactions across the building. Recent LLM-based HVAC controllers have shown that prompt-based control is feasible. However, these methods produce inconsistent results, are difficult to generalize, and experience control-latency issues. Their performance also relies heavily on numeric reward values, textual feedback, or unstructured retrieval, which limits their ability to make control decisions that account for zone dynamics and thermodynamics. No existing LLM-based methods have used building semantics and dynamics, together with rolling history, as the centre of the LLM context while testing and comparing HVAC control performance with existing LLM-based HVAC approaches. This paper presents ThermoLLM, an LLM-based HVAC control framework tested in a multi-zone building environment. Spatiotemporal context grounding is a main component of ThermoLLM, which uses a spatial knowledge graph derived from Brick-style building semantics and linked with recent observations from a rolling history. We evaluate ThermoLLM against standard control methods and recent state-of-the-art baselines in a five-zone building environment across six climate-season scenarios. ThermoLLM consistently achieves the best energy-comfort trade-offs and outperforms all evaluated baselines in the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) rankings across all six scenarios.
In many domains, practitioners seek models that produce accurate forecasts while faithfully capturing latent system dynamics. Existing approaches typically sacrifice one of these goals: deep state space models often assume Gaussian latent transitions, limiting fit and forecasting, while diffusion models are highly expressive but lack principled inference for the underlying dynamics. To combine the strengths of both, we introduce the Diffusion-Driven State Space Model (DDSSM), which replaces the conventional Gaussian transition distribution with a diffusion model. Our DDSSM resolves the open problem of how to jointly train an autoencoder and a diffusion model on sequential data, thereby extending the literature on latent diffusion models for time series. Moreover, we find that the DDSSM empirically outperforms a state-of-the-art deep SSM at fitting and forecasting a simulated time series with multimodal transitions.
Diffusion-based generative models usually perturb data through Markov processes driven by Brownian motion. Volterra processes allow more flexible temporal dependence, but their non-Markovian structure complicates score learning and reverse-time simulation. We develop Volterra generative models and provide Markovian approximations. Gaussian conditioning then reduces the denoising problem to the original data dimension. We present the connections with variance-exploding and variance-preserving representations. Our framework also permits reuse of pretrained denoisers such as EDM2 networks. With one auxiliary factor, our models trained from scratch achieve FIDs of $0.547$ on MNIST and $1.860$ on conditional CIFAR-10. With autoguidance, ImageNet64 FID reaches $1.912$ after fine-tuning, while ImageNet512 FID reaches $1.903$ using frozen pretrained networks.
Existing low-light image enhancement methods often suffer from local exposure imbalance and color distortion under complex, non-uniform illumination, while mainstream Vision Transformers lack explicit modeling of physics-based priors for illumination degradation. To address these limitations, we propose GLFS, a Gaussian Light Field Splatting-based Vision Transformer that incorporates continuous light-field priors from 2D Gaussian Splatting (2DGS) as an explicit physical bias into the Transformer self-attention mechanism to guide spatial gain estimation under complex non-uniform illumination. Specifically, GLFS represents scene illumination as a superposition of anisotropic Gaussian basis functions and introduces a physics-informed bias into the self-attention mechanism to adaptively infer a spatial gain field, enabling accurate and spatially uniform restoration under complex illumination conditions. To mitigate color shifts and structural degradation during enhancement, we further introduce a color vector angular loss and a luminance edge loss, with the former enforcing hue consistency and the latter improving the structural fidelity of local details. Extensive ablation studies and quantitative evaluations demonstrate that GLFS achieves competitive overall performance in illumination correction, structure preservation, and perceptual quality, validating the effectiveness of the continuous Gaussian light field prior for low-light image enhancement.
Realistic patient simulation is a key requirement for developing and testing clinical LLMs at scale without costly user studies. Existing simulators lack realism and controllability: they overshare information and miss the variability of real patient behavior. Here, we introduce PatientsWithPersonality (PWP), a patient simulation framework that generates realistic yet diverse patient responses through explicit personality parametrization over a latent patient state. Grounded in the six-dimensional HEXACO personality model, it gives fine-grained control over conversational style, cooperativeness, and information disclosure. We evaluate PWP in the first direct comparison of different patient simulators under a shared protocol, anchored in transcripts of real physician-patient encounters. Clinicians judge PWP conversations as realistic at least as often as the recorded encounters themselves and far more often than prior simulators, and almost never flag them as "too informative". Our simulator also reproduces the disclosure pacing of real patients, and its HEXACO-conditioned personality traits can be recovered by clinicians and an autorater. PWP further shows a substantially wider behavioral range than the closest baseline. In a downstream benchmark, LLMs interviewing patients simulated by PWP perform closest to how they perform on the real encounters, whereas prior simulators make the task easier. PWP thus provides a realistic, steerable, and diverse patient population for improving and benchmarking clinical LLMs.
Analyzing tennis technique from monocular RGB video could make coaching more accessible, but prior methods usually rely on hand-trimmed strokes, test on random splits that leak player identity, or fail on players with different body proportions. We present an end-to-end, player-disjoint framework with an autonomous kinetic stroke event detector, a 78-dimensional causal velocity representation ([X, V]), a 569,095-parameter multi-task Transformer Encoder (E7), exploratory 9-dimensional kinematic deviation profiling, and a heuristic feedback prototype. All models are evaluated with Clean LOPO v2: Leave-One-Player-Out cross-validation over 7 players and 3 seeds, giving 21 outer evaluations and 147 training runs per configuration, with nested inner cross-validation. Against 206 frozen consensus impact events, the detector reaches an F1-score of 85.65% (+3.24% over baseline thresholding) with a mean absolute timing error of 80.6 ms. Under heavy class imbalance, Stroke Type reaches 74.27% accuracy, 65.37% balanced accuracy, and Macro-F1 = 0.6453. Direction reaches 43.90% accuracy, 37.75% balanced accuracy, and Macro-F1 = 0.3141, a 17.9% relative Macro-F1 gain over the static baseline E0, though balanced accuracy is only a little above the 33.3% chance level. Posture Quality reaches 52.13% accuracy, 50.56% balanced accuracy (essentially chance), and Macro-F1 = 0.4232. Single-task specialists did not differ significantly from the joint model on Stroke Type (p = 0.4062) or Posture Quality (p = 0.1406). On Direction the joint model did somewhat better (exact permutation p = 0.0469), but this is suggestive only and does not pass a Bonferroni threshold. Deviation profiling shows a nominal, unadjusted separation between good and bad posture (|d| = 0.341, p = 4.07 x 10^-8), which drives the feedback prototype. We report negative ablations, seeds, and limits as a baseline for pose-based coaching.
Multimodal large language models (MLLMs) have advanced affective computing by enabling nuanced reasoning over visual, acoustic, and linguistic evidence. A key approach to improving their generalization is outcome-based reinforcement learning, which converts existing label-level annotations into rewards for final-answer correctness. However, its sparse sequence-level supervision provides limited fine-grained guidance for interpreting multimodal affective cues and may reward correct labels even when those cues are misinterpreted. To address this challenge, we introduce OmniOPSD, the first on-policy self-distillation framework for omni-modal affective reasoning. At its core, OmniOPSD transforms label-level supervision into evidence-rich, teacher-only rationales by leveraging reference affective judgments as semantic priors for offline multimodal evidence elicitation from frontier MLLMs. Through rationale-privileged self-distillation, it couples on-policy student rollouts with dense, evidence-informed token-level supervision from a local self-teacher, allowing the student to learn along its own trajectories rather than imitate fixed reasoning traces. This decoupling of offline evidence acquisition from local policy learning enables training without additional human annotation, frontier-model logits, or online frontier-model calls, while preserving student-only inference from the original task input. At both 3B and 7B scales, OmniOPSD outperforms supervised fine-tuning (SFT) and group relative policy optimization (GRPO) in mean macro-F1 on both in-domain and out-of-domain evaluations. Among the compared models, it ranks first by MER-UniBench mean across all three input-modality settings, reaching 84.13 with audio--video--text input, and achieves the highest MME-Emotion recognition score (Rec-S) of 42.4.
We envision continually learning agentic systems that become more useful over time: as they encounter sequences of related tasks, they should infer the hidden structure shared across those tasks and use it to improve future decisions. This cross-task experiential learning capability is pivotal in domains such as personalization and interactive assistance, but existing training/evaluation frameworks do not provide shared, controllable latent structures and cannot measure whether or why agents improve. We introduce LatentGym: a controllable suite in which each environment is organized around a ground-truth latent variable governing the structure across tasks. Our construction yields metrics that separate exploration (whether the agent's actions gather information about the latent) from exploitation (whether the agent uses what it has gathered). We demonstrate our suite on empirical studies addressing three questions: how and why frontier models fail to adapt across related tasks; whether post-training on related task sequences improves general cross-task adaptation, and where those gains come from; and how design choices such as inter-task feedback shape training dynamics and generalization. Together, these results establish a controlled foundation for studying how LLM agents learn from experience across tasks, and for designing agents that adapt more reliably in sequential, personalized, and interactive settings.
Recursive training of generative models on their own outputs can lead to model collapse, a compounding drift away from the true data distribution. Existing theoretical works bound finite-round error accumulation in the context of diffusion models, but two questions remain open:~what distribution does the recursion converge to, and how fast? We answer both, isolating a mechanism distinct from imperfect learning: even with perfect score estimation and exact sampling, the early stopping of the reverse diffusion (required for numerical stability) drives a progressive drift away from the data distribution. We prove that this recursion converges geometrically to a unique limiting distribution, which admits a closed-form characterization as an infinite mixture of increasingly Gaussian-smoothed versions of the data distribution. A Hermite spectral decomposition of this limit reveals that recursive training acts as a low-pass filter: higher-order modes, which encode fine non-Gaussian structure, are attenuated much more strongly than coarse modes. This spectral picture motivates annealed truncation schedules that progressively shrink truncation times across retraining rounds; we prove that any schedule converging to $0$ asymptotically eliminates recursive compounding. Finally, we show our idealized characterization is robust: in the presence of discretization and score estimation errors, the learned distribution remains in a Wasserstein-2 ball around the ideal limit, with mode-dependent contraction rates that contract high-order errors faster than low-order ones. We validate the theory on synthetic Gaussian mixtures and CIFAR-10.
With the growth of LLMs' (Large Language Models) capabilities, there has been an increasing push to curate high quality datasets by filtering samples in the training data. In general, Data Attribution (DA) methods aim to estimate how individual samples in a training dataset can precondition a model to generate certain outputs. As an example, one might be interested in which samples in the data could be the source of toxic behavior after training the LLM. Many methods quantify this conditioning through the paradigm of influence functions. While methods of this family are effective in its function, they lack the necessary processing speed and storage compactness to be practically implemented on large datasets. We propose a method, Influcoder, as a quick and cost-effective approach to influence-based Data Attribution at scale.
On-policy distillation (OPD) has recently become a prominent post-training recipe by combining two desirable ingredients: on-policy student-generated trajectories and dense token-level teacher supervision. Yet how this hybrid training regime shapes a model remains poorly understood. We characterize the sparsity and geometry of OPD parameter updates across several language and vision-language model pairs and application settings. OPD updates are small and coordinate-sparse, while remaining distributed across layers and modules. This sparse support is operationally meaningful: masked training on the discovered subnetwork nearly recovers full-training performance. At the matrix level, the updates are numerically full-rank but spectrally concentrated. Their visible supports avoid coordinates emphasized by the source's principal structure and favor low-magnitude source coordinates, while the source singular-value spectra change little. Together, these findings show that OPD exhibits important weight-space signatures of on-policy post-training despite using dense teacher supervision.
Pretrained vision-language-action (VLA) models offer semantic and spatial priors for robot manipulation, but their parallel-gripper interfaces leave multi-finger contact coordination implicit. Adapting these models requires both timely grasp transitions and embodiment-specific hand trajectories. We introduce InDex, an intent-conditioned framework that connects these two requirements. A normalized aperture signal, derived from retargeted demonstrations, supervises a scalar grasp intent while preserving full hand-action supervision. Stage 1 learns synchronized end-effector-intent chunks; Stage 2 conditions a diffusion hand decoder on these predictions, visual-language context, and proprioception. Sequential optimization establishes the upstream interface before learning dexterous control with low-rate co-adaptation. Across four simulated tasks, InDex with $\pi_{0.5}$ achieves 85.8% mean task success, improving the native policy by 35.5 percentage points, and reaches 69.0% on held-out objects with similar geometry without fine-tuning or recalibration. Improvements also hold across three VLA backbones and separately trained real-world policies, where mean success increases from 71.0% to 83.0%. Component ablations and timing interventions support the value of explicit grasp-intent conditioning and expressive hand generation. These results identify arm-hand coordination as an actionable bottleneck in adapting parallel-gripper priors to dexterous manipulation.
Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications.
A central goal of mechanistic interpretability is to identify which internal components causally drive a language model's behavior. Because these importance estimates serve as the evidence for identifying circuits, systematic errors can lead to the misidentification of the underlying mechanisms. While activation patching provides a gold-standard causal metric, its computational cost is prohibitive at scale. Practitioners instead rely on attribution patching, a gradient-based, first-order approximation whose reliability remains poorly understood. In this work, we characterize the source of this unreliability, demonstrating that the dominant error stems from the non-linearities in the downstream network rather than local curvature at the patched component. This insight yields three practical tools: (i) a reliability score to detect untrustworthy estimates, (ii) error bounds quantifying potential attribution mis-specifications, and (iii) a Hessian-vector-product (HVP) correction that eliminates the leading-order error with only one additional backward pass. In evaluations across five model families (124M--9B parameters) and both random-token and naturalistic (name-swap) perturbations, HVP is the only second-order correction feasible at larger scale, where standard baselines like Integrated Gradients become computationally prohibitive. In comparative experiments, a multi-step HVP variant matches or exceeds the accuracy of Integrated Gradients at significantly lower compute, outperforming prior second-order baselines. These improvements lead to higher-fidelity circuit recovery on standard benchmarks and support a Screen-Flag-Fix workflow that targets computational effort only toward the components flagged as unreliable.
Key-value (KV) cache quantization is widely used to reduce LLM inference memory, yet existing evaluations focus only on perplexity and accuracy without assessing the safety impact. In this study, we explore alignment preservation under KV cache quantization. Across eleven instruction-tuned models (3.8B-72B) and five benchmarks (1,894 prompts), we find that low-bit quantization can silently destroy safety alignment: Mistral-7B loses 15.2% of its refusals at only 1.03x perplexity, and no universal safe bit-width exists, with sharp model-specific phase transitions invisible to standard metrics. We identify that the root cause is geometric: safety features occupy a low-dimensional activation subspace $10^2$-$10^3\times$ more vulnerable to quantization noise than the full representation space perplexity averages over. We propose Per-Channel Reduction (PCR), a diagnostic that classifies each model into one of three mechanistic failure modes: outlier-crushes-safety, where safety lives in non-outlier channels collaterally damaged by outlier-driven scale factors; outlier-as-safety, where safety overlaps outlier channels and finer granularity cannot rescue it; and multi-layer dilution, where safety is distributed across many layers and per-layer fixes fail. PCR predicts the correct mitigation direction on all nine primary models and one held-out model from an independent family using 20 calibration prompts. PCR generalizes across unseen prompts, models, and quantizer families, including production quantizers (KIVI: up to 97.2% recovery) and rotation-based quantization (QuaRot-style Hadamard), succeeding where attention-based allocation methods fail. The resulting training-free protocol, requiring ~35 GPU-minutes, recovers up to 97% of lost alignment at under 1% latency and minimal memory overhead, addressing vulnerabilities confirmed in production vLLM serving with FP8 KV cache on NVIDIA GPUs.
Tissue graph counterfactuals ask what a cell would express if its spatial neighborhood were altered, a question central to predicting cell behavior in tissues. Existing methods, however, present only specific instances of such queries. In this work, we first formalize tissue graph counterfactuals as a class of spatial interventions that either rewire connections between cells (edge perturbation) or modify the expression of their neighbors (node perturbation). We then introduce Cellina, a framework that uses supervised disentanglement to decompose a cell's intrinsic state from its spatial context, using the latter as a conditioning input for in-silico perturbation predictions. Across benchmarks spanning over 2.5 million spatially-resolved cells in observational and interventional screens, Cellina outperforms both spatially-informed and non-spatial competitors in in-silico perturbations, while training as fast as non-spatial methods. Moreover, we show that Cellina reveals cancer subdomains with distinct signaling programs in an unsupervised manner and enables targeted neighbor perturbation simulations.
Facial rigging--creating Facial Action Coding System (FACS) blendshapes and inner-mouth geometry (teeth, gums, and tongue)--remains a major bottleneck in 3D character production. We present OmniFaceRig, a fully automatic end-to-end pipeline that converts a static surface-only 3D character mesh, with no pre-modeled oral cavity, into an inner-mouth-aware FACS rig with up to 155 blendshapes, procedurally fitted teeth, gums, and tongue, and re-packed UV/texture. It supports humans, humanoids, and long- and short-muzzled animals with no manual landmarks, user-provided templates, or per-asset setup. OmniFaceRig combines a hybrid vision-language and computer vision riggability checker, a four-model face segmentation ensemble, keypoint-driven template registration, procedural inner-mouth construction, and collision-aware blendshape transfer. For non-human characters, it selects topology-specific templates with a nose-landmark-free design and uses as-rigid-as-possible deformation with signed-distance-field constraints to reduce teeth-face intersections. We also introduce Omni-Bench, an open-source benchmark of more than 1,000 biped 3D characters rigged with FACS blendshapes and inner-mouth geometry, spanning humans, humanoids, cats, dogs, and other animals. Experiments show high rigging success, nearly complete face detection recall, low inner-mouth penetration, and 20-30 s end-to-end latency per asset on a single A100 GPU.