This paper describes Observatorio L\'azaro, a language resource that monitors unassimilated lexical borrowings (predominantly English lexical borrowings or anglicisms) in the Spanish digital press. Since April 2020 the system has automatically processed the daily output of a collection of news outlets, detected borrowings with a neural sequence-labeling model, and made the results available through a public web interface and API. The result is a continuously updated diachronic database which, at the time of writing, records more than two million borrowings across 1.88 million articles and 993 million running tokens of text (2020-2026). The paper documents the resource: we describe the end-to-end pipeline (acquisition, detection, post-processing, storage and access), the data model and the terms of availability; we evaluate the resource through the detector's held-out performance (span-level F1=0.86 for the borrowing class), inter-annotator agreement on the training corpus (Cohen's kappa=0.91) and a manual precision audit of 1,000 spans from the deployed data; and we situate it with respect to Spanish borrowing lexicography, annotated borrowing corpora and neology-monitoring observatories. The data shows that unassimilated anglicisms are used in the Spanish press at a frequency of approximately two anglicisms per thousand tokens, and that this rate remains stable. Our statistical analysis over six years reveals that the anglicism vocabulary in Spanish behaves as an open and growing class, with 58.7% of its types attested only once (53.6% after correcting for detection precision), and that its density is highest in the fashion, technology and lifestyle sections and lowest in political and institutional news. The resource is intended to complement static borrowing dictionaries and one-off annotated corpora by providing a continuously updated record of borrowing in the Spanish press.
Human personality theories characterize traits not as isolated attributes captured by a single score, but as stable individual tendencies expressed through the interplay among persons, situations, and behaviors. Existing studies of personality-related behavior in LLMs have primarily focused on outputs elicited under personality conditioning, characterizing observable trait-related expressions while lacking mechanistic evidence for the existence of internal personality-related representations, their cross-situational expression, and how these representations shape specific behaviors. Building on Funder's personality triad framework, we adapt its three components for LLM analysis: Person as personality-related internal representations, Situation as contexts that afford trait-relevant responses, and Behavior as response patterns on broader social tasks. We introduce a framework for discovering, controlling, and validating trait-like representations in LLMs. First, using contrastive behavior pairs grounded in shared situations, we identify sparse internal features associated with opposing poles of personality traits through SAE decomposition. We validate their trait relevance through effects on behavior to situation, token-level activation patterns, and robustness to paraphrasing. Second, feature-level interventions induce bidirectional trait-related shifts across a separate, diverse set of situations while preserving response validity, demonstrating consistent expression across contexts. Third, applying the same interventions to social intelligence tasks reveals behavioral changes with benefit-tradeoff patterns consistent with findings from human personality research, providing behavioral-level validation beyond personality scores. Our findings provide evidence that LLMs contain controllable trait-like representations linking internal states, situational expression, and behavioral outcomes.
Differential privacy provides formal privacy guarantees for training neural networks on sensitive data, while Bayesian deep learning offers a principled framework for uncertainty-aware prediction. Combining these two objectives remains challenging, as privacy noise can interact with the stochasticity introduced by Bayesian posterior sampling. In this work, we investigate differentially private variational Bayesian learning through the Improved Variational Online Newton (IVON) optimizer. We introduce DP-IVON-Gradsq, a private variant of IVON. The proposed method constructs its curvature estimate from the privatized gradient using a noise-corrected squared-gradient estimator, reducing the direct interaction between posterior-sampling noise and privacy noise while preserving the Adam-like computational efficiency of IVON. We evaluate DP-IVON-Gradsq on CIFAR-10 against the standard private optimizers DP-SGD and DP-Adam over a range of privacy budgets. The results show that DP-IVON-Gradsq is competitive under weak-to-moderate privacy constraints, i.e., large-to-moderate values of $\varepsilon$, while degrading under strong privacy. Code is available at https://github.com/NourJamoussi/DP-IVON-Gradsq.git.
Despite recent advances, self-supervised learning (SSL) models and Joint-Embedding Predictive Architectures (JEPAs) remain susceptible to spurious correlations. Many anti-collapse regularizers constrain the global representation distribution to match a prescribed target, such as a multivariate Gaussian or the uniform distribution on the sphere. However, global matching does not prevent groups defined by a bias attribute from occupying distinct regions of the representation space. We show that existing encoder-only debiasing regularizers, including EnD and FSCL, can be interpreted as restricted forms of conditional distribution matching. Building on this interpretation, we introduce Unbiased Open World (UOW), which matches the representation distribution within each observed bias group to a common target distribution. At the population optimum, using a discriminative conditional discrepancy guarantees independence between the representation and the targeted attribute. We instantiate UOW in Gaussian and spherical spaces and evaluate it across controlled synthetic settings and real-world fairness benchmarks. Across Synthetic Engraving, Colored MNIST, CelebA, and Waterbirds, conditional matching substantially reduces bias compared with global regularization while preserving useful downstream representations. Jointly conditioning on multiple bias attributes further reduces intersectional disparities missed by separate marginal constraints. Finally, experiments on CelebA show that models can be trained from random initialization using UOW as a pretraining objective, provided checkpoints are selected according to fairness-aware validation criteria.
Transformers are known to have internal continuous symmetries that leave outputs invariant while altering how activations are distributed across coordinates, and hence how well they quantize. GaugeQuant leverages this in-training by introducing a LogSumExp term to the loss that breaks the symmetries, thus selecting a basis that minimizes activation outliers. A stop-gradient operator ensures that only rotation matrices are updated, yielding the language modeling objective completely unaltered. Our quantization strategy requires no specific calibration data, no quantization simulation, and adds a small inference overhead. As a proof of concept, we evaluate on WikiText-2 perplexity and zero-shot reasoning. On Qwen-2.5 0.5B, GaugeQuant reduces perplexity (e.g. $25.62 \to 16.76$ under W4A16) and improves zero-shot accuracy. On LLaMA-2 7B, perplexity improves but downstream accuracy does not. The code is available at: https://github.com/MPedraBento/gauge-quant.
Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.
Data-driven pre-fault dynamic security assessment (DSA) rapidly evaluates the dynamic risk of credible contingencies on a power system using machine learning (ML). Existing approaches face two limitations. First, they require a large labelled database for training, with a separate model trained, tuned, and maintained for each contingency in a potentially long list of credible contingencies. Second, the trained models generalize poorly to unseen contingencies. This work addresses these limitations by using a tabular foundation model (TFM) that assesses stability through in-context learning, and requires no retraining or hyperparameter optimization. We use a single TFM to assess many contingencies at once, removing the need to train one classifier per contingency. We also characterize when using electrical distance coordinates (EDC) as continuous features enables the TFM to generalize to unseen contingencies and when it does not, showing how a few labelled samples can reliably improve generalization. Through comprehensive case studies on the IEEE 68-bus system, we show that a single TFM attains an average Macro F1 score of about 90\% with only 120 labelled samples per contingency, outperforming both classical ML and deep learning baselines, without any model retraining or hyperparameter tuning. For held-out contingencies, we show that using just 10 labelled samples of the new contingency with a modified EDC encoding matches the balanced accuracy of the best achievable transfer learning oracle model, which requires fully labelled data and is not deployable in practice. Overall, this initial study paves the way for developing and deploying foundation models for power system operations, with possible applications across multiple operational tasks.
We study causal inference under outcome interference for sequential, observational settings. Specifically, we consider settings where the binary outcomes over N units are Markovian across T time steps. At each time step, the outcomes of N units have dependencies captured through an Ising model; each outcome is also impacted through an external field capturing the effects of its treatment as well as latent confounders. Similar to panel data literature, these latent confounders are modeled to have a low-rank factor structure. Our data is a single sample from this high-dimensional distribution. To estimate causal quantities of interest, we provide a computationally efficient method based on Maximum Pseudo-Likelihood Estimation (MPLE) for learning the model parameters. Under mild assumptions, we establish non-asymptotic consistency for parameter estimation and show this translates to faithful estimation of causal quantities of interest after sampling from the learned model. We demonstrate the efficacy of the method through synthetic experiments as well as a real-world case-study investigating causal effects of vaccine rates on COVID-19 death rates within US counties nationwide.
Sign Language Production (SLP) generates continuous signing from a linguistic input and drives accessibility technologies such as signing avatars. Current generators, diffusion models included, regress joint coordinates as an unstructured vector, so nothing ensures that the generated body respects anatomy or the mechanics of movement. Physics has entered SLP only through geometric loss terms, whose benefit for intelligibility has not been verified, and the dynamics of the signing body has not been modelled. We present PIDiffSign, a diffusion model informed by physics at two levels. For kinematics, a forward-kinematics head outputs bone directions and lengths and composes them into an anatomically exact skeleton. For dynamics, a rigid-body model of the upper body with anthropometric parameters supports two training priors. A physical-plausibility suite that includes two laws of motor control evaluates the generated signing. Under a controlled protocol, the head lowers back-translation word error rate by about five points on PHOENIX14T and CSL-Daily, whereas geometric penalties have no measurable effect; a control experiment traces the gain to the kinematic-chain representation and not to bone-length conservation. The dynamics priors gain about one further point over matched non-physical controls on PHOENIX14T, but not on CSL-Daily. Generated signing follows the laws of motor control as closely as real signing but carries markedly less kinetic energy. The exact skeletons can drive rigged avatars directly, and the suite offers a physical benchmark for future SLP systems.
In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others. Continuous space modeling helps TTCD avoid the parallel sampling of multiple tokens, which is a key source of inaccuracy at high speedups for models that iterate purely in discrete space. The notion of per-token times helps TTCD to better model conditional generation, allows for more sure tokens to proceed at a faster rate, and allows for differentiated inter-token influences during refinement. TTCD outperforms discrete models at high speedups. We train a 160M parameter TTCD model on OpenWebText, and then self-distill it; we find that at high speedups we are comparable in unconditional generation quality, and outperform in conditional generation, several existing models of similar size trained, on the same data, and self-distilled. We achieve similar gains in Sudoku solving as well.
We study the contextual combinatorial semi-bandit (CCSB) problem, where at each round, the learner observes a context, selects a combinatorial action consisting of a subset of basic arms, and receives the reward of each selected arm; its goal is to maximize its cumulative reward over time. While many previous works focus on CCSB problems with linear reward function approximation, the design of efficient algorithms for the CCSB problem with general reward function approximation is limited. In this work, we study the realizable setting where the learner has access to a reward function class $\mathcal{F}$ that contains the true reward function, and propose the $\mathrm{SquareCB{.}Comb}$ algorithm. We prove that $\mathrm{SquareCB{.}Comb}$ achieves a minimax optimal regret bound of $O(\sqrt{m A T \log |\mathcal{F}|})$, where $A$ is the number of arms, $m$ is the maximum number of arms in a combinatorial action, and $T$ is the time horizon. In addition, $\mathrm{SquareCB{.}Comb}$ is computationally efficient when we have an online regression oracle that can efficiently predict rewards of all arms given contexts, and the combinatorial action set admits either efficient linear optimization or efficient sampling given any target marginal probability on arms. We empirically verify the effectiveness of our algorithm in webpage recommendation and contextual shortest path problems.
The behavior of an LLM agent is determined not only by the underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures. Existing approaches typically optimize harnesses on training or development data and freeze the resulting programs at test time, limiting adaptation to shifts in task distributions, failure modes, and tool interactions. We ask whether the harness can instead be optimized during evaluation using unlabeled execution traces. We introduce Test-Time Harness Evolution (TTHE), which treats the executable harness as the state of test-time adaptation. On each unlabeled test batch, TTHE iteratively evolves a population of candidate harnesses through agentic proposers that reason over execution traces and explore diverse edit directions. A judge selects a candidate using execution traces and execution-derived proxy signals. The selected harness determines the current batch's outputs and initializes adaptation on the next. Crucially, the solver, proposers, and judge operate through different harnesses around the same frozen LLM, without weight updates, access to final-answer labels, or a separately trained adaptation model. Across six execution-grounded benchmarks spanning text-to-SQL, competitive programming, software engineering, data-science coding, terminal-based tasks, and agentic tool use, TTHE improves fixed ReAct-style baseline harnesses through persistent, inspectable program adaptation. These results recast test-time adaptation for LLM agents as evolution over executable control programs and identify execution-derived proxy reliability as a central challenge for robust unsupervised agent improvement.
This paper studies a scalable approach for ordinary least squares (OLS) regression based on distributed sketching. It distributes small sketches of a large data set across multiple workers, has each worker construct a local OLS estimator from its sketch, and averages the local estimators into a global one. Prior studies assume each sketch is generated from all data, but this sketching step has high computation cost, sometimes even higher than standard OLS. Instead, we propose to generate each sketch from a partitioned subset of the data, which allows the sketching cost to decrease with more workers. Under the fixed design setting, we characterize the exact excess loss of the global estimator. This loss is higher than the loss established in prior studies, but remains close to it when the average Burg divergence between subset covariances is small. Since more workers tend to increase the divergence under our partition design, this reveals a tradeoff between the computational and statistical efficiency of our estimator.
Large language models have made rapid progress in generating machine-checked proofs for well-defined mathematical problems. Extending this progress to research mathematics requires more than solving isolated theorems. Research involves choosing questions, developing definitions and conjectures, connecting results, and combining formal proofs with expert judgment. This survey examines how LLM-driven systems can support these broader activities. We review formal-mathematics tasks, datasets, and autoformalization, and organize proof-generation methods into three overlapping areas: training strategies, inference-time search and adaptation, and agentic workflows. We then examine benchmark results, selected research projects, and platforms for collaborative mathematical work. A central distinction runs throughout the survey: checking a formal proof does not by itself establish that it captures the intended problem or makes a new mathematical contribution. We therefore distinguish theorem benchmarks, repository tasks, formalization projects, and research claims, and discuss the evidence needed to assess each. Finally, we identify five priorities for progress: better data and evaluation, representations of mathematical relationships, conjecture and concept generation, reliable tool integration, and human--AI collaboration. Together, these perspectives connect advances in proof generation to the practical demands of mathematical research.
Latent reasoning performs multi-step inference in continuous hidden states, promising more compact and efficient reasoning. However, these opaque states raise a question of faithfulness: whether the latent reasoning steps drive the final answer. Prior work reports unfaithful behaviors at selected checkpoints, leaving their development during training unexamined. We track behavioral and activation-based evidence across training using verified counterfactual edits and interventions on the latent reasoning states. We find that accuracy can improve while models become less responsive to edits that change the required reasoning path and the correct answer. This counterfactual responsiveness follows distinct trajectories across latent reasoning approaches. On the graph reachability task ProsQA, output sensitivity to noise replacement declines alongside counterfactual responsiveness. Across separately trained binary-choice and open-ended versions of GSM8K math problems, interventions on all latent states show opposite sensitivity trajectories. Guided by these trajectories, we intervene on the training procedure and improve counterfactual responsiveness. These results show that evaluating only a final checkpoint can obscure both when counterfactual responsiveness changes and what the latent states contribute.
Uncertainty estimation is essential for the trustworthy deployment of LLMs: it underpins self-refinement and serves as a decision signal for downstream systems, from verifiers to collaborating agents. However, existing approaches operate at suboptimal granularities: token-level scores lack semantic coherence, while sequence-level scores fail to localize errors. We formalize Span-Level Uncertainty Estimation (SLUE), a new task that targets the natural granularity for uncertainty: semantically coherent text spans, each conveying a single assessable unit of meaning. To address this task, we introduce SPANUQ, a lightweight probe that distills the uncertainty knowledge from expensive multi-sample inference into a single forward pass over LLM hidden states. SPANUQ employs a DETR-style span decoder to simultaneously detect spans and estimate their uncertainty via a Mixture of Beta distribution, trained with a principled combination of Beta NLL regression and contrastive ranking objectives. We construct SPANUQ-BENCH, the first span-level uncertainty benchmark comprising 20K prompts, 293K annotated spans, and continuous soft labels derived from multi-sample claim verification. Experiments on five LLM backbones show that SPANUQ consistently achieves the best span-level uncertainty quality , outperforming the strongest probe baseline and all sampling-based methods while being 10~20x faster. Its DETR-based span detector attains 0.910 F1, surpassing the best heuristic by 39.4%, enabling precise error localization that sequence-level methods cannot provide. The same architecture ports to five LLMs spanning two model families, with one probe trained per backbone, and we additionally observe that sequence-level uncertainty is partially decomposable, suggesting that span-level estimation subsumes sequence-level as a special case. The project page is available damon-demon.github.io/SpanUQ.
The performance of deep learning models crucially depends on the settings of hyperparameters like learning rate, initialization scale, and weight decay. Hyperparameter transfer aims to make near-optimal hyperparameter settings consistent across model scale, so that large models can be optimized by proxy tuning their smaller, cheaper-to-optimize counterparts. While transfer principles are well-studied in the context of dense neural networks in language and vision tasks, they remain comparatively under-explored for graph neural networks (GNNs). We develop and validate a transfer parameterization for GNNs trained with SGD, Adam, and AdamW. Through theoretical scaling analyses and controlled experiments, we show that the proposed parameterization yields stable feature updates, learning rate transfer, and improved performance as width and depth increase. For SGD, we identify graph-dependent first-layer correction factors and show that their use can accelerate early training in graphs with sparse bag-of-words inputs. For Adam, we explore how different message passing normalizations affect early- and late-training transfer behavior, illustrating the importance of message passing normalization and advocating for an associated hyperparameter. For AdamW, we adapt a parameterization that allows for the joint transfer of weight decay and learning rate. Together, these results provide a practical recipe for scaling GNNs across a variety of learning tasks and training scenarios.
Can a simple labeled-pixel contrastive branch improve semi-supervised segmentation with a strong pretrained encoder? We study this question with PixCon, which adds a class-wise memory bank to a DINOv2-Base implementation of UniMatch V2. The bank stores ground-truth-labeled features only where the student prediction agrees with the label. A multi-positive contrastive loss uses these features to train the shared encoder-decoder; the head and bank are discarded at inference. This simplifies earlier labeled-pixel banks by removing confidence thresholds and learned ranking. On Pascal VOC with 183 labels, PixCon improves all three paired seeds by 0.20, 0.64 and 1.83 mIoU points, raising the mean from 87.01 to 87.90. With 92 labels the mean gain is 0.40 points; single-seed differences on Cityscapes and ADE20K range from -0.08 to +0.14 points. A predicted-label contrastive control reaches nearly the same Pascal accuracy, so these results support the added branch rather than an accuracy benefit from its correctness filter. An exact gradient decomposition distinguishes mislabeled-positive counts from their similarity-weighted influence. Separate teacher diagnostics find retained-pixel error rates of 1.8% on Pascal and 10.6% on ADE20K. These diagnostics do not measure the training bank or predict final accuracy. PixCon adds 15.3-19.7% iteration time in the measured runs and no inference computation. We report paired seeds, training cost and the limits of the evidence.
Restoring archival film remains a fundamentally challenging problem due to the absence of paired training data and the lack of standardized evaluation benchmarks. Pristine versions of deteriorated footage are physically unrecoverable, requiring supervised methods to rely on synthetic data that often fail to capture the complex, temporally coherent nature of real film degradation. At the same time, existing real-world datasets are limited in scale, quality, and accessibility, hindering reliable evaluation and fair comparison across methods. We address both limitations with AbsoluteDegradation, a physics-inspired, modular pipeline for synthesizing realistic film degradations, and a new large-scale archival benchmark. The proposed pipeline models the analog-to-digital process as a structured composition of artifact families, incorporating signal-dependent grain, parametric scratches, and temporally coherent camera motion, enabling controlled generation of diverse degradation regimes. In parallel, we introduce a curated dataset of 81,576 high-resolution frames sourced from real archival footage, designed for consistent evaluation under real-world conditions. Together, these contributions provide a unified framework for training and benchmarking restoration models. Experiments across six restoration architectures, evaluated with no-reference image quality metrics, clip-level temporal quality assessment, and a blinded human preference study (93 raters, 1,395 trials), show that models trained with AbsoluteDegradation generalize better to real-world footage: human raters preferred them in 64.7% of comparisons and for every evaluated architecture. The proposed benchmark also reveals systematic failure modes of current methods and of no-reference metrics. We hope this work establishes a foundation for reproducible and domain-authentic evaluation in archival film restoration.
Robot policies that remain broadly competent after deployment can degrade when dynamics, contact, geometry, actuation, or sensing differ from source conditions. We study post-hoc adaptation through structured output-space adapters: compact transformations of samples from a frozen base policy. We introduce Warp, a context-conditioned monotone bijective spline that enables nonlinear sample-dependent reshaping while preserving per-dimension ordering; Translation and Affine provide location-only and location--scale references. Across four ManiSkill tasks with controlled deployment shifts, Warp achieves the strongest recovery in a matched evolution-strategy (ES) adapter study, outperforming controls for spline parameterization, location--scale adaptation, and generic nonlinear capacity. Under PPO, the ordering becomes task dependent, showing that richer adapter structure does not automatically translate into better adaptation under a different optimizer. Finally, we test the practical benefit of this approach on physical SO101 peg insertion. Within an interaction-limited RLPD/SAC pipeline, Warp achieves comparable success to a strong stochastic residual while reducing median successful cycle time by approximately 30%. These results identify structured output-space adaptation as a useful design space for adapting frozen robot policies.