Distributional shift in reinforcement learning (RL) is defined inconsistently across the literature, and it is studied alongside out-of-distribution (OOD) generalization and non-stationarity without a clear account of how the three terms are related. Methods are also often described as robust after evaluation under only a few particular changes, leaving the scope of such claims unclear. We develop a unified taxonomy that brings the three terms under one definition of distributional shift. We factorize the joint distribution of a one-step interaction into five components, each produced by its own generating function: initial-state distribution, observation mechanism, policy implementation (distinct from the policy being optimized), transition dynamics, and reward mechanism. A shift is specified by what changes (the changed components) and when it changes (the interaction regime: OOD, non-stationary, or combined), while changes in interaction data and performance are treated as observed effects. We complement the taxonomy with an evaluation framework that measures immediate generalization and subsequent adaptation, summarizing adaptation in a shift adaptation score (SAS). Published benchmark results, controlled Gridworld experiments with DQN, A2C, and PPO, and a mapping of selected studies show that method responses and rankings vary with the shifted component, and that existing evaluations cover only some components. Robustness claims should therefore be bounded by the components and regimes evaluated, since strong performance under one shift does not establish robustness under others. Identifying an unknown component-level source from observed interaction data remains a direction for future research.
Do language models encode the same prompted concepts in structurally compatible ways, beyond generic similarity induced by shared questions? We study contrastive-difference centered kernel alignment ($\mathrm{CKA}_{\Delta}$), which compares per-question activation differences between opposing concept instructions. A matched same-concept versus cross-concept comparison separates the question of concept-conditioned structure from the magnitude of representational similarity. Across four checkpoints and eight personality-related traits, a fixed-final-layer analysis gives mean same- and cross-trait $\mathrm{CKA}_{\Delta}$ of 0.7259 and 0.6828. Trait-label randomization detects correspondence conditional on this model and prompt set ($p=1/10001$, one-sided), but a model-and-trait-weighted 95% interval for the difference is $[-0.0330,0.1017]$, limiting broader generalization. Strict question-group holdouts retain high same-trait linear recoverability. More stringently, maps and preprocessing fitted on seven traits transfer a source polarity classifier for the eighth trait to handwritten test questions: mean affine and orthogonal accuracies are 88.10% and 88.57%, versus 47.70% for shuffled pairing. Performance varies substantially across traits and model directions. Synthetic controls show why moderate CKA and high affine accuracy alone do not establish functional equivalence. The results support a scoped diagnostic of concept-specific structure and held-out linear recoverability; they do not establish universal invariance or a robust advantage over raw CKA across architectures.
Recovering dynamical systems from noisy observations is a recurring challenge across scientific domains, including neuroscience and physics. Latent stochastic differential equations (SDEs) address this by modeling the system as an unobserved state that evolves according to a learnable SDE and generates the observations. Variational inference (VI) provides a tractable objective for fitting latent SDEs. Traditional VI algorithms evaluate this objective by numerical simulation over a time discretization, trading fidelity for computational cost. A recent class of algorithms, simulation-free VI, sidesteps this tradeoff by parameterizing the posterior through its instantaneous marginals rather than its drift. In this work, we show that the efficiency of existing simulation-free VI algorithms comes at a price: their parameterizations restrict the approximate posterior to a subset of the SDEs available to simulation-based methods, degrading posterior inference and parameter learning. We propose Helmholtz-SDE, a simulation-free VI algorithm that closes this gap by optimizing over path laws compatible with a prescribed collection of marginals. Helmholtz-SDE recovers dynamics more faithfully than prior simulation-free methods, with the largest gains under high posterior uncertainty. It further matches the performance of simulation-based VI at a fraction of the runtime. Code is available at https://github.com/lindermanlab/helmholtz-sde .
A risk threshold calibrated on old prompts can stop being valid when prompts to a foundation model drift. We study PromptShift-CRC, a rule that weights past prompts by feature similarity, falls back to recent prompts when the current prompt is unlike the past, and updates its level online. We prove that the update controls the long-run violation rate for any weights, so marginal coverage cannot separate weighting schemes. We give a conditional bound at a fixed level whose cost is a weighted total variation distance, bounded by a computable prompt-drift term up to an unknown Lipschitz constant. We propose a comparison protocol: a matched short-window baseline, a validity and sharpness frontier, 30-seed synthetic streams, and real-output streams ordered without using the model's output. On MMLU, TruthfulQA (Qwen2.5-1.5B) and RealToxicityPrompts (GPT-2, replicated with a second generator and classifier), every online method reaches the target. PromptShift-CRC stays within 0.0055 of the Adaptive CRC frontier and avoids the worst window, but DtACI matches or beats it in most settings. The rule performs like a mid-ranked window without having to choose one. Tests fixed in advance to favor the weights show no consistent gain over the frontier. Code, outputs and streams are released.
Conformal prediction intervals lose their guarantee when a series is not exchangeable. We study weighting calibration residuals by the similarity of local Fourier power features, combined with the adaptive level update of ACI. A two-term account of the coverage error, mismatch that falls and sampling error that rises as the bandwidth shrinks, holds with exogenous features and conditionally independent residuals, but not for features computed from the series itself, which is how the method is used. In four simulations, spectral weighting lowered the mean regime-level deviation of coverage by 17% to 19% when the features came from the response and by 75% when they were exogenous, and not clearly under gradual drift. In each recurring-regime design a scaled-error baseline or Conformal PID control calibrated better. On nine public series with 36 forecast origins, spectral weighting with the plain weighted quantile did not differ from projected ACI in coverage error, and Conformal PID and scaled ACI were the best calibrated methods. The features are blind to seasonal phase, so the weights act like a recency kernel. A pure recency kernel did at least as well, especially with a finite-sample rank correction (mean absolute coverage error 0.039 against 0.090 for ACI; a post hoc check). Calendar phase weights narrowed German electricity intervals by 23% at equal coverage, but were not better than ACI across the series. We suggest trying adaptive baselines first, and similarity weights only where regimes differ in spectral shape or season and the effective sample size stays large.
Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice. On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students, but we find that it suffers a characteristic failure mode in this setting: small student errors compound across turns and push the trajectory out of the teacher's familiar state distribution, so the teacher's supervision becomes least reliable precisely where the student needs it most. We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout and schedules the teacher's intervention probability along a curriculum that decays to zero. Strong guidance keeps early trajectories close to the teacher distribution and is then gradually withdrawn to recover the purely on-policy regime used at inference. On ALFWorld, ScienceWorld, and WebShop, distilling Qwen3 students from a Qwen3-30B-A3B teacher, Guided-OPD yields average relative gains of 21.1\% in Score and 25.5\% in Success Rate over vanilla OPD, with larger gains on smaller students.
Every major LLM agent framework gives the LLM the role of orchestrator; the model decides what to do next, when to call tools, and when to stop. We argue that token explosion, control-flow hallucination, and unreliable completion are not implementation bugs but architectural consequences of assigning the deterministic work of looping, branching, and sequencing to a probabilistic system. A better prompt or a stronger model cannot guarantee the reliability of the LLM agent. We therefore propose Agentic Programming, in which the program governs all control flow, and the LLM is itself part of it, an adaptive component we call LLM-as-Code and invoke only where a task calls for reasoning or generation. Within each call the model keeps full flexibility, but it cannot alter the program's execution path. With control in the program, the LLM's context is built from the execution history's call tree and forms a directed acyclic graph (DAG). Each call's context length is then determined by its call depth rather than by accumulation over steps. A case study of computer-use agents shows that the design is practical, not just a theoretical stance, substantially improving the stability of long visual operation sequences.Code is available at https://github.com/FuzzyKuji/OpenProgram.
Selecting an energy-efficient model for periodic sensing requires accounting for time between inferences. Hardware-aware model search typically measures inference alone, even when the application cares about energy per release. We present CREST (Cross-platform Runtime Evaluation and Search Tool), an open-source hardware-in-the-loop experimentation system that evaluates model and hardware configurations under a specified release schedule. Researchers can vary the objective, schedule, target and candidate proposer while reusing evaluation and replay. Records preserve comparison conditions and separate valid measurements from deployable candidates. Three case studies demonstrate the system. First, searches and controlled replay on STM32 N657 show how waiting changes model and clock selection. For one architecture, continuous inference favors 800 MHz over 300 MHz at 8.0 versus 17.1 mJ per inference, but additional waiting offsets 97% of this saving in a fitted decomposition. With STOP between releases, 300 MHz uses about 0.3-0.4% less board energy per release at 100 and 500 ms. At a fixed clock, the preferred architecture changes between release periods, which a model with shared waiting parameters cannot produce. Latency predicts continuous energy within a median 1.7% on held-out observations, yet most release-energy error remains in scheduled terms. In paired replay, one of six randomly selected near-latency pairs reverses its continuous-energy order by 5.8 mJ per release. Second, replay on three Arduino targets shows proxy-selected inertial-odometry candidates can fail to fit or use more energy than measured selections, with only 9 of 16 FLOPs-selected candidates fitting the most memory-constrained board. Third, an audio classification study reuses the evaluator with LLM-based candidate proposers, and cross-target replay moves a selected configuration from third to last in energy rank.
Vision-Language Models (VLMs) such as CLIP have become a standard backbone for open-vocabulary recognition, yet their zero-shot predictions remain vulnerable to distribution shifts encountered at deployment. Test-Time Adaptation (TTA) has recently been extended to CLIP as a lightweight solution, leading to a rapidly growing body of TTA4CLIP methods. However, empirical progress in this area has largely outpaced our understanding of what truly drives adaptation, where their gains originate, and under which shifts they remain reliable. In this paper, we take a step back from the pursuit of state-of-the-art accuracy and conduct a systematic controlled study of TTA4CLIP. We first organize existing methods into three unified paradigms according to what is updated at test time. We then introduce TTABC, an open-source TTA Benchmark for CLIP, which standardizes evaluation protocols and integrates more than 20 representative methods. Our controlled empirical analysis focuses on three key areas. First, we determine the driving factors in parameter-based methods, revealing that adaptation gains are primarily driven by test-time evidence and reliable proxies rather than heavy optimization. Second, we explore evidence utilization beyond heavy parameter tuning, showing that competitive and efficient performance can be achieved through cross- or current-sample evidence and lightweight prototype updates. Finally, we demonstrate that there is no silver bullet for TTA: no single adaptation paradigm is universally optimal, and the preferred paradigm depends on the nature of shift. We hope our benchmark and study provide a clearer understanding of the current TTA4CLIP landscape and establish a foundation for further research. The project page is available at https://walawalagoose.github.io/TTABC/
We study the sample complexity of learning in average-reward weakly-coupled Markov decision processes (WCMDPs) and Restless Bandits (RBs) under a generative model. Naive reduction to a tabular MDP leads to high complexity bounds as the state-action space is exponentially large in the number of arms $N$. By exploiting the weakly coupled structure, we show that near-optimal policies can be learned with sample and computational complexities that are polynomial in $N$. Specifically, we analyze the plug-in approach, which applies an efficient planning algorithm to an empirical model estimated from data. For fully heterogeneous WCMDPs, we establish the first finite-sample PAC guarantee with polynomial complexity and an $O(1/\sqrt{N})$ optimality gap. For homogeneous RBs, we further prove that a smaller optimality gap is achievable under mild structural assumptions. A primary technical contribution is a novel Lyapunov-based analysis framework. Unlike classical approaches that rely on the difficult-to-control bias function, our framework uses an explicitly constructed Lyapunov function along with a drift transfer technique between the true and empirical models. A key step of independent interest in our framework is a fine-grained perturbation analysis for the underlying linear programming (LP) relaxation, which provides a general tool for analyzing LP-based policies and weakly-coupled systems.
Constrained MDPs (CMDPs) are a widely adopted framework for incorporating safety into RL agents; however, the framework does not support risk-sensitive constraints. This can be problematic: For example, CMDPs allow for optimal solutions that, in order to satisfy the risk-neutral constraints, mix infrequent catastrophic behaviors and frequent, overly conservative ones. Moreover, prior empirical results suggest that enforcing stricter, risk-sensitive constraints can improve performance even under risk-neutral evaluation. The natural framework to incorporate risk-sensitive constraints is utility-constrained MDPs (UCMDPs), but no practical solutions for this problem existed. In this work, we introduce a simple yet powerful methodology for UCMDPs and constrained RL. Besides allowing for risk-sensitive constraints, our framework does not require us to fix constraint limits in advance of training the agent, provided that a sensible range is known. This increases policy flexibility and, in practice, allows for adjustments to these limits at no extra training cost. Besides benefiting from the generality of the framework, our agent shows strong performance in practice, consistently matching or outperforming existing baselines in several Safety Gymnasium benchmark tasks.
Creative image editing tools, such as Adobe Photoshop's Remove and Generative Fill buttons, are central to everyday customer use and account for a major share of traffic. However, current generative AI models face significant latency challenges, which become even more pronounced as the field transitions from convolution-based U-Nets to Diffusion Transformers (DiTs). In our evaluation on hundreds of representative image editing samples spanning a wide range of mask ratios, the DiT module alone accounts for an average of 73% of the total model latency, even after being distilled from 50 timesteps down to 8 timesteps. To tackle this challenge, we propose $\textbf{HiLo-Token}$, an input-adaptive token compression framework that allocates a larger token budget to high-frequency, rich-context regions while assigning fewer tokens to low-frequency areas. Specifically, for the editing region specified by the user mask, we retain all tokens within a dilated mask to preserve strong locality and contextual relevance. Outside the editing region, we introduce a simple yet effective high-frequency token selection strategy based on spatial frequency to capture important local details, while using tokens from a heavily downsampled image to represent low-frequency components and preserve the coarse global structure. Extensive experiments on production-level evaluation data validate the effectiveness of the proposed method, which achieves 3.13$\times$, 2.59$\times$, and 1.67$\times$ DiT speedups on A100-80GB for image editing tasks across small, medium, and large mask ratio categories with average ratios of 6.38%, 15.92%, and 35.36%, respectively, without any regression in generation quality. In production, where the editing pipeline is served as disaggregated, independently autoscaled GPU workers, these per-request savings translate into a 33% reduction in the GPUs required to serve the models.
Deep unfolding (DU) accelerates iterative optimizers by training learnable components through unrolled iterations, but extending DU to the large-scale semidefinite programs (SDPs) common in robotics has remained limited. Unrolling a full-update conic solver such as COSMO exposes two obstacles: backpropagating through the per-iteration linear-system solve incurs memory quadratic in the problem size when the coefficient matrix is formed explicitly, and backpropagating through the positive semidefinite (PSD) cone projection is numerically unstable under repeated eigenvalues. We address the first with a matrix-free implicit differentiation rule that reduces memory from $O(n^2)$ to $O(n + \mathrm{nnz}(A))$, and the second with a stable backward rule based on the Dale\v{c}kii--Krein representation of the Fr\'echet derivative. Together, these enable learning lightweight hyperparameter policies and warm starts for COSMO. On standalone SDPs and second-order cone programs, ranging from max-cut and Lov\'asz $\vartheta$ to robust estimation and control, the learned policies outperform tuned COSMO on every problem class, reducing iterations to tolerance by up to 60$\times$, and transfer without retraining to instances up to 10$\times$ larger in the size parameter while retaining 2--5$\times$ speedups. Our matrix-free solver implementation also batches naturally on the GPU, solving 64 Lov\'asz $\vartheta$ SDPs 14$\times$ faster than sequential COSMO.jl. When embedded in sequential convex programming for nonlinear covariance steering, our learned optimizers outperform the interior-point solver Clarabel in terms of both success rate and solve time.
Sensor-based human activity recognition runs on edge devices, trading performance against latency, memory, and energy. Yet new architectures are usually compared with few, often outdated baselines on few datasets with study-specific data preparation and hardware, leaving open which perform best at what cost. We present HARENA, a living benchmark with a public leaderboard, to which new architectures and datasets can be added under one protocol. Its initial suite evaluates 16 neural architectures and three classical baselines on 29 datasets, measuring performance on unseen subjects and deployment cost on a smartphone, smartwatch, and microcontroller. CNN-HAR, the suite's oldest deep architecture, leads overall, but 8 architectures lie within 2.5 points of its mean test macro-F1, while their cost differs by orders of magnitude. On the microcontroller, Random Forest leads on latency and peak RAM, LITEWAY-Light on flash, and TinyHAR is most accurate within the real-time budget. Rankings shift with dataset and channel count, so general claims require many datasets, which a growing benchmark can provide. We find no sign of benchmark overfitting, but performance drops by a median of 15.0 points on unseen subjects, about three times the typical spread between architectures, making generalization and adaptation the most promising future directions.
Are large language models (LLMs) bad at capturing human judgment? Two commonly stated limitations are that LLMs fail to capture full distributions of responses, and that their judgments are unstable across wording variations. We demonstrate simple prompting strategies that mitigate these limitations. Across two datasets--a U.S.--representative set of 144 moral scenarios and 38 moral beliefs from the International Social Survey Programme's Family and Changing Gender Roles module covering 32 countries--we show how simple elicitation techniques help improve AI-human alignment. First, prompting models to report standard deviations and response proportions recovers the full range of human responses better than common strategies. Second, ensuring scenarios are clear to human participants--as reflected in human confusion ratings--is associated with stronger model alignment, and LLMs can track human confusion ratings. At the same time, we find that model confidence ratings do not consistently track errors in models' judgments, though they can predict human variability relatively well. These results suggest that asking better questions to LLMs can yield better answers.
Closed-loop driving simulators mostly fill their environments with traffic agents that behave alike, whether they are rule-based or learned. Recent methods that control driving style learn it from labels inferred from existing driving recordings, from synthetic data generated to reflect each style, or from reward weights estimated by a large language model (LLM). All of these are indirect estimates of a style, not recordings of people who were instructed to drive in it. We introduce PersonaDrive, a vision-language-action (VLA) driving agent that takes its style from retrieved human demonstrations. The demonstrations come from a new dataset in which 15 participants drove CARLA routes on a driving simulator under conservative, neutral, and aggressive instructions. PersonaDrive trains a lightweight retrieval head on features from a frozen image encoder and on control signals over per-style databases, and fine-tunes one VLA backbone over the retrieved demonstrations. Switching style only means searching a different FAISS index, with no retraining. On the Bench2Drive benchmark without style conditioning, PersonaDrive improves the Driving Score by 4.6% over SimLingo and by 2.5% over HiP-AD, a gain that fine-tuning without retrieval does not reproduce. With style conditioning, PersonaDrive outperforms the strongest baseline, Drive My Way, in every style, with a Driving Score 6.7% to 11.6% higher. Its driving also follows the instruction, since average speed rises by 19.8% from the conservative to the aggressive style. Each traffic agent is evaluated individually as a driving policy. The dataset and model weights are available on the project page: https://pervasiveautonomylab.github.io/personadrive-landpage/.
Multisensory perception is essential for robotic manipulation, with tactile sensing providing critical information for fine-grained, contact-rich interactions. Although prior work integrates tactile and spatial signals to improve perception, vision-tactile learning commonly relies on object-level alignment. Addressing this limitation requires systematically aligned 3D vision-tactile data. We develop a data collection system and introduce Touch3D, a dataset comprising 505 real-world objects and 20,025 tactile contacts. Building on this dataset, we establish the Contact-Centric Multimodal Tactile Representation Benchmark (CoMT-Bench), which expands evaluation from object-level retrieval and tactile properties to include patch-level contact evaluation. We further propose Geometry-Relation Distilled Tactile-Centric Cross-Modal Learning (GeoTAC), which explores tactile learning across all multisensory data. In experiments on the Touch3D test set, patch-level alignment improves the average score by 5.32 over object-level alignment, with GeoTAC's geometric supervision improving it by a further 4.48. In robotic manipulation, GeoTAC improves TP by 19.14 and 11.34 over DINOv2 (Vision-only) and DINOv2 (Vision-Tactile), respectively. Project page: https://lihongcs.github.io/tacdino.
Online learning with delayed feedback typically assumes that the learner can track all pending rounds until their feedback arrives. In practice, tracking resources are finite, and feedback from rounds that cannot be tracked is permanently lost. In this paper, we study delayed online convex optimization (OCO) under a hard capacity constraint, where at most $C$ pending rounds can be tracked at any time. To model delay information, we introduce a semi-clairvoyant model that refines the clairvoyant assumption from prior work: rather than requiring delays to be known at prediction time, the learner observes delay expirations online, consistent with the classical unconstrained delayed setting. Our approach proceeds via a reduction to a novel ``delayed and weighted'' OCO problem, using a scheduler that randomizes which rounds are tracked and importance-weights the resulting observations. For the base problem, we propose and analyze Delayed-Weighted FTRL and its bandit analogue, obtaining regret bounds that characterize how weights interact with delayed feedback in both settings. Combining these bounds with our schedulers yields regret guarantees for capacity-constrained OCO under convex and strongly convex losses, for both first-order and bandit feedback. For first-order feedback, capacity $C = \Omega(\log T)$ suffices to recover the standard delayed OCO rates up to logarithmic factors. For bandit feedback, the standard delayed BCO rates are instead modulated by $(1 + \sigma_{\text{max}}/C)$ factors, where $\sigma_{\text{max}}$ is the maximum number of outstanding observations. This allows the regret bound to degrade gracefully when $C < \sigma_{\text{max}}$. We complement our analysis with numerical experiments.
In reinforcement learning for multi-turn agents, a sparse outcome reward makes credit assignment across intermediate tool calls difficult. On-policy self-distillation eases credit assignment by having the model teach itself with dense supervision: the model with privileged information (the teacher) scores each token generated by the model without it (the student). However, it remains unclear what forms of feedback are most effective in multi-turn agentic settings. We find that the natural multi-turn variant fails: when the teacher is given a complete successful demonstration at every turn, training makes the agent worse than the untrained model. The teacher ends up rewarding the student for copying the demonstration, even when a copied action is not right in the student's context. We propose HINT, in which a reflector, by default the policy itself, reads the completed trajectory and writes short turn-level feedback. The teacher then sees what went wrong at each turn, not the actions that followed. On Tau-Bench and WebShop, HINT matches GRPO with one-eighth as many rollouts.
Autoregressive video generation has emerged as a powerful paradigm for World Action Models (WAMs). However, existing approaches suffer from slow training convergence and limited converged accuracy, particularly at high frame rates, as the training supervision is confined to the current chunk without explicit signals about future dynamics; they also suffer from slow inference due to iterative video denoising. In this paper, we present Next Forcing, a multi-chunk prediction (MCP) framework for world action modeling that enables faster training, higher accuracy, and accelerated inference. Inspired by multi-token prediction in large language models, Next Forcing introduces an MCP training objective that augments the main model with lightweight auxiliary MCP modules to simultaneously denoise video chunks at multiple future temporal horizons (next$^1$, next$^2$, next$^3$ chunks). These MCP modules form a causal chain across prediction depths, where intermediate features fused from multiple layers of the main model are leveraged to predict future dynamics, allowing near-future predictions to inform farther-future ones and providing dense multi-scale temporal supervision back to the main model. During training, the MCP modules significantly accelerate convergence and improve converged accuracy, especially at high frame rates: at 50 fps, Next Forcing achieves a 93.1% relative improvement over LingBot-VA at 5k training steps and 2.3$\times$ faster convergence, and establishes new state-of-the-art results on the RoboTwin benchmark (94.1/93.5% on Clean/Random). At inference, the MCP modules can be retained to predict the next video chunk in parallel with the current one, achieving 2$\times$ inference acceleration. Beyond World Action Models, Next Forcing also improves physical-law adherence on PhyWorld and reduces FVD by over 50\% in action-free general video pretraining.