PyTorchKR Papers

최신 논문

총 523,801편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2608.10805v3cs.CV

Fast and Memory-Efficient Wavelet Convolutions via I/O-Aware Reformulation

Wavelet convolution (WTConv) has emerged as an increasingly popular drop-in replacement for standard convolutions, expanding a network's receptive field exponentially with the number of decomposition levels while keeping the parameter count linear. However, its reference implementation is severely memory-bound due to excessive data movement through high-bandwidth memory (HBM). We develop an I/O model of WTConv to characterize this bottleneck and use it to guide three algebraic reformulations: (1) recomputing the inexpensive Haar analysis butterfly on chip, (2) collapsing the multi-level synthesis cascade into a single closed-form pass indexed by output-coordinate bits, and (3) folding learned per-channel scales into the convolution weights. Together, these reformulations enable an I/O-aware fused implementation that substantially reduces HBM traffic. We evaluate the WTConvNeXt configuration across decomposition levels and a broad range of tensor shapes. Despite performing comparable arithmetic, the reference WTConv is substantially slower than the depthwise convolution it replaces. Our reformulation reduces modeled HBM traffic by approximately $2.55\times$, yielding up to a $4.35\times$ training speedup over the reference while roughly halving peak memory usage. Thus, our reformulation preserves the benefits of WTConv while substantially reducing its execution time and memory footprint, removing the systems overhead that previously limited its practical efficiency.

arXiv:2608.10459v2cs.CL

MD-ProTector: Positioning Multiple Data-Driven Prototypes for LLM-Generated Text Detection

As LLM-generated content becomes more sophisticated, detection systems for distinguishing those texts from human-written text must operate at scale while handling diverse writing styles, domains, languages, and generator models. Input-only encoder detectors are suitable for practical deployment setting, but standard binary classification supplies only the class label and does not explicitly organize the substantial variation within either class. We propose MD-ProTector, which represents each class with multiple trainable reference vectors in the encoder embedding space, referred to as prototypes. These prototypes provide separate decision boundaries for different groups of texts within the same class. However, adding multiple prototypes alone does not determine which variation each prototype should represent. MD-ProTector addresses this problem with Prototype Positioning loss, which separates class-level structure from the within-class variation that differentiates individual prototypes. Evaluated across five settings from three large-scale benchmarks covering domain, generator, language, and adversarial variation, MD-ProTector achieves the highest AvgRec on MAGE CDCM and RAID and the highest AUROC and lowest FPR95 on RAID among the compared encoder-based methods.

arXiv:2608.06004v2cs.LG

Do Tabular Foundation Models Agree with Themselves?

Tabular Foundation Models (TFMs) are the state of the art for many tabular prediction tasks. They are transformers that approximate the Bayesian posterior predictive distribution under a pretraining prior. TFMs become multivariate predictors autoregressively, by sampling one target and appending it to the features. This paper is the first to evaluate whether this setup induces a valid stochastic process. Informally, we ask: *Do we get the same answer to the same question asked in two different ways?* The standard Kolmogorov consistency conditions cannot be checked on TFMs directly, since TFMs are unable to model certain conditional distributions. We therefore propose two adapted, testable requirements. Marginalization consistency demands that marginalized conditionals equal directly predicted marginals, and factorization consistency that both factorization orders yield the same joint. Since the data-generating distribution satisfies both, a perfect TFM would as well, and every violation lower-bounds how far the model's predictions are from the ground truth. Every TFM we evaluate violates both requirements, for classification and regression, on all datasets.

arXiv:2608.04968v3cs.LG

EvolveNet: Collaborative Harness Evolution for Agent Self-Improvement

An LLM agent can acquire persistent improvements by evolving its harness---the executable program governing model calls, tool use, verification, and recovery---without updating model weights. When the same harness serves users with different workloads, isolated evolution limits the sharing of useful discoveries, while centralized adaptation can introduce conflicting task-specific changes. We introduce EvolveNet, a framework for collaborative harness evolution that connects local specialization with shared capability accumulation. Clients evolve the same broadcast harness on their local workloads in parallel, and a server composes their adaptations into an updated shared program. Redistribution makes accepted discoveries available to all clients, allowing subsequent local evolution to build on improvements accumulated across the collaboration. EvolveNet supports this process through evidence-guided, scope-typed program aggregation and behavioral validation. Our theoretical analysis provides sufficient conditions for retaining and combining complementary improvements despite program interactions. Experiments across five settings show improved shared-harness performance, with ablation and behavioral analyses supporting the contribution of composition beyond specialist selection alone.

arXiv:2608.02946v2cs.LG

Sedentary Behavior Classification for Wearable Sensors with a CNN-BiLSTM Model

Accurate detection of sedentary behavior is important for studying health risks related to prolonged sitting, but posture-based classification remains challenging with wearable sensors, especially at the wrist. We study whether a deep learning model trained on hip-worn accelerometer data can transfer to wrist-worn accelerometer data for sitting versus non-sitting classification. We use CHAP, a CNN-BiLSTM model originally developed for hip accelerometers, and evaluate its zero-shot performance on wrist data as well as its adaptation through finetuning with varying amounts of labeled wrist data. Experiments are conducted on the iWatch dataset with ground-truth posture labels derived from wearable cameras. The hip-trained model performs strongly on hip data without retraining, but accuracy drops on wrist data due to sensor placement shift. Finetuning CHAP provides consistent advantages over transformer models trained from scratch. These findings suggest that hip-based pretraining provides a useful starting point for wrist deployment, while highlighting the need for wrist-specific adaptation to handle higher signal variability.

arXiv:2608.02602v2cs.CL

AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling

Language remains an outlier in generative modeling: while images, video, and audio are increasingly modeled in continuous latent spaces, text generation still relies predominantly on discrete tokens. Existing continuous language models either inherit embedding spaces not designed for joint generation and decoding, or compress autoencoded latents to ease diffusion, sacrificing token-level fidelity. Instead of simplifying the representation to suit the generative model, we preserve a high-capacity, decodable text latent and design the diffusion model to learn its distribution directly. We introduce AURORA-LM, a continuous-latent diffusion language model that separates the construction of a decodable text representation from the modeling of its distribution. A Query-based Encoder-Decoder organizes text into a high-capacity, prefix-aligned latent sequence, and a Block-causal Diffusion Transformer learns its distribution through flow matching, generating blocks left to right while denoising positions within each block in parallel. Because such a latent is harder for diffusion to model, AURORA-LM restricts only the noisy-input pathway while retaining the full clean-latent prediction target, accommodating full-width latents without reducing decoder-facing capacity. We further calibrate the noise-level distribution to the latent width, and introduce self-trajectory consistency to bridge independently sampled training noise and iterative denoising at inference. AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. Scaling to 1B parameters with about 1500 EFLOPs of total compute yields further gains, surpassing a larger publicly released latent-diffusion language model under a matched evaluation protocol. All experiments are conducted on Ascend NPUs.

arXiv:2608.00713v2cs.CL

Observatorio Lazaro: A self-populating database of anglicism usage in the Spanish press

This paper describes Observatorio L\'azaro, a language resource that monitors unassimilated lexical borrowings (predominantly English lexical borrowings or anglicisms) in the Spanish digital press. Since April 2020 the system has automatically processed the daily output of a collection of news outlets, detected borrowings with a neural sequence-labeling model, and made the results available through a public web interface and API. The result is a continuously updated diachronic database which, at the time of writing, records more than two million borrowings across 1.88 million articles and 993 million running tokens of text (2020-2026). The paper documents the resource: we describe the end-to-end pipeline (acquisition, detection, post-processing, storage and access), the data model and the terms of availability; we evaluate the resource through the detector's held-out performance (span-level F1=0.86 for the borrowing class), inter-annotator agreement on the training corpus (Cohen's kappa=0.91) and a manual precision audit of 1,000 spans from the deployed data; and we situate it with respect to Spanish borrowing lexicography, annotated borrowing corpora and neology-monitoring observatories. The data shows that unassimilated anglicisms are used in the Spanish press at a frequency of approximately two anglicisms per thousand tokens, and that this rate remains stable. Our statistical analysis over six years reveals that the anglicism vocabulary in Spanish behaves as an open and growing class, with 58.7% of its types attested only once (53.6% after correcting for detection precision), and that its density is highest in the fashion, technology and lifestyle sections and lowest in political and institutional news. The resource is intended to complement static borrowing dictionaries and one-off annotated corpora by providing a continuously updated record of borrowing in the Spanish press.

arXiv:2607.26853v2cs.CL

From Representations to Behaviors: Exploring the Person-Situation-Behavior Triad in LLMs

Human personality theories characterize traits not as isolated attributes captured by a single score, but as stable individual tendencies expressed through the interplay among persons, situations, and behaviors. Existing studies of personality-related behavior in LLMs have primarily focused on outputs elicited under personality conditioning, characterizing observable trait-related expressions while lacking mechanistic evidence for the existence of internal personality-related representations, their cross-situational expression, and how these representations shape specific behaviors. Building on Funder's personality triad framework, we adapt its three components for LLM analysis: Person as personality-related internal representations, Situation as contexts that afford trait-relevant responses, and Behavior as response patterns on broader social tasks. We introduce a framework for discovering, controlling, and validating trait-like representations in LLMs. First, using contrastive behavior pairs grounded in shared situations, we identify sparse internal features associated with opposing poles of personality traits through SAE decomposition. We validate their trait relevance through effects on behavior to situation, token-level activation patterns, and robustness to paraphrasing. Second, feature-level interventions induce bidirectional trait-related shifts across a separate, diverse set of situations while preserving response validity, demonstrating consistent expression across contexts. Third, applying the same interventions to social intelligence tasks reveals behavioral changes with benefit-tradeoff patterns consistent with findings from human personality research, providing behavioral-level validation beyond personality scores. Our findings provide evidence that LLMs contain controllable trait-like representations linking internal states, situational expression, and behavioral outcomes.

arXiv:2607.23649v2cs.LG

DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton

Differential privacy provides formal privacy guarantees for training neural networks on sensitive data, while Bayesian deep learning offers a principled framework for uncertainty-aware prediction. Combining these two objectives remains challenging, as privacy noise can interact with the stochasticity introduced by Bayesian posterior sampling. In this work, we investigate differentially private variational Bayesian learning through the Improved Variational Online Newton (IVON) optimizer. We introduce DP-IVON-Gradsq, a private variant of IVON. The proposed method constructs its curvature estimate from the privatized gradient using a noise-corrected squared-gradient estimator, reducing the direct interaction between posterior-sampling noise and privacy noise while preserving the Adam-like computational efficiency of IVON. We evaluate DP-IVON-Gradsq on CIFAR-10 against the standard private optimizers DP-SGD and DP-Adam over a range of privacy budgets. The results show that DP-IVON-Gradsq is competitive under weak-to-moderate privacy constraints, i.e., large-to-moderate values of $\varepsilon$, while degrading under strong privacy. Code is available at https://github.com/NourJamoussi/DP-IVON-Gradsq.git.

arXiv:2607.22149v2cs.LG

Unbiased Open World Regularization for Fair Self-Supervised Learning

Despite recent advances, self-supervised learning (SSL) models and Joint-Embedding Predictive Architectures (JEPAs) remain susceptible to spurious correlations. Many anti-collapse regularizers constrain the global representation distribution to match a prescribed target, such as a multivariate Gaussian or the uniform distribution on the sphere. However, global matching does not prevent groups defined by a bias attribute from occupying distinct regions of the representation space. We show that existing encoder-only debiasing regularizers, including EnD and FSCL, can be interpreted as restricted forms of conditional distribution matching. Building on this interpretation, we introduce Unbiased Open World (UOW), which matches the representation distribution within each observed bias group to a common target distribution. At the population optimum, using a discriminative conditional discrepancy guarantees independence between the representation and the targeted attribute. We instantiate UOW in Gaussian and spherical spaces and evaluate it across controlled synthetic settings and real-world fairness benchmarks. Across Synthetic Engraving, Colored MNIST, CelebA, and Waterbirds, conditional matching substantially reduces bias compared with global regularization while preserving useful downstream representations. Jointly conditioning on multiple bias attributes further reduces intersectional disparities missed by separate marginal constraints. Finally, experiments on CelebA show that models can be trained from random initialization using UOW as a pretraining objective, provided checkpoints are selected according to fairness-aware validation criteria.

arXiv:2607.20757v3cs.LG

GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries

Transformers are known to have internal continuous symmetries that leave outputs invariant while altering how activations are distributed across coordinates, and hence how well they quantize. GaugeQuant leverages this in-training by introducing a LogSumExp term to the loss that breaks the symmetries, thus selecting a basis that minimizes activation outliers. A stop-gradient operator ensures that only rotation matrices are updated, yielding the language modeling objective completely unaltered. Our quantization strategy requires no specific calibration data, no quantization simulation, and adds a small inference overhead. As a proof of concept, we evaluate on WikiText-2 perplexity and zero-shot reasoning. On Qwen-2.5 0.5B, GaugeQuant reduces perplexity (e.g. $25.62 \to 16.76$ under W4A16) and improves zero-shot accuracy. On LLaMA-2 7B, perplexity improves but downstream accuracy does not. The code is available at: https://github.com/MPedraBento/gauge-quant.

arXiv:2607.16097v3cs.LG

Understanding Reasoning from Pretraining to Post-Training

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.

arXiv:2607.16031v2cs.LG

Revisiting data-driven dynamic security assessment with a tabular foundation model

Data-driven pre-fault dynamic security assessment (DSA) rapidly evaluates the dynamic risk of credible contingencies on a power system using machine learning (ML). Existing approaches face two limitations. First, they require a large labelled database for training, with a separate model trained, tuned, and maintained for each contingency in a potentially long list of credible contingencies. Second, the trained models generalize poorly to unseen contingencies. This work addresses these limitations by using a tabular foundation model (TFM) that assesses stability through in-context learning, and requires no retraining or hyperparameter optimization. We use a single TFM to assess many contingencies at once, removing the need to train one classifier per contingency. We also characterize when using electrical distance coordinates (EDC) as continuous features enables the TFM to generalize to unseen contingencies and when it does not, showing how a few labelled samples can reliably improve generalization. Through comprehensive case studies on the IEEE 68-bus system, we show that a single TFM attains an average Macro F1 score of about 90\% with only 120 labelled samples per contingency, outperforming both classical ML and deep learning baselines, without any model retraining or hyperparameter tuning. For held-out contingencies, we show that using just 10 labelled samples of the new contingency with a modified EDC encoding matches the balanced accuracy of the best achievable transfer learning oracle model, which requires fully labelled data and is not deployable in practice. Overall, this initial study paves the way for developing and deploying foundation models for power system operations, with possible applications across multiple operational tasks.

arXiv:2607.14940v2cs.LG

Causal Inference for Sequential Settings under Interference and Latent Confounding

We study causal inference under outcome interference for sequential, observational settings. Specifically, we consider settings where the binary outcomes over N units are Markovian across T time steps. At each time step, the outcomes of N units have dependencies captured through an Ising model; each outcome is also impacted through an external field capturing the effects of its treatment as well as latent confounders. Similar to panel data literature, these latent confounders are modeled to have a low-rank factor structure. Our data is a single sample from this high-dimensional distribution. To estimate causal quantities of interest, we provide a computationally efficient method based on Maximum Pseudo-Likelihood Estimation (MPLE) for learning the model parameters. Under mild assumptions, we establish non-asymptotic consistency for parameter estimation and show this translates to faithful estimation of causal quantities of interest after sampling from the learned model. We demonstrate the efficacy of the method through synthetic experiments as well as a real-world case-study investigating causal effects of vaccine rates on COVID-19 death rates within US counties nationwide.

arXiv:2607.14836v2cs.CV

A Diffusion-Based Kinematics and Dynamics Model for 3D Sign Language Production

Sign Language Production (SLP) generates continuous signing from a linguistic input and drives accessibility technologies such as signing avatars. Current generators, diffusion models included, regress joint coordinates as an unstructured vector, so nothing ensures that the generated body respects anatomy or the mechanics of movement. Physics has entered SLP only through geometric loss terms, whose benefit for intelligibility has not been verified, and the dynamics of the signing body has not been modelled. We present PIDiffSign, a diffusion model informed by physics at two levels. For kinematics, a forward-kinematics head outputs bone directions and lengths and composes them into an anatomically exact skeleton. For dynamics, a rigid-body model of the upper body with anthropometric parameters supports two training priors. A physical-plausibility suite that includes two laws of motor control evaluates the generated signing. Under a controlled protocol, the head lowers back-translation word error rate by about five points on PHOENIX14T and CSL-Daily, whereas geometric penalties have no measurable effect; a control experiment traces the gain to the kinematic-chain representation and not to bone-length conservation. The dynamics priors gain about one further point over matched non-physical controls on PHOENIX14T, but not on CSL-Daily. Generated signing follows the laws of motor control as closely as real signing but carries markedly less kinetic energy. The exact skeletons can drive rigged avatars directly, and the suite offers a physical benchmark for future SLP systems.

arXiv:2607.14106v2cs.CL

Token Time Continuous Diffusion for Language Modeling

In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others. Continuous space modeling helps TTCD avoid the parallel sampling of multiple tokens, which is a key source of inaccuracy at high speedups for models that iterate purely in discrete space. The notion of per-token times helps TTCD to better model conditional generation, allows for more sure tokens to proceed at a faster rate, and allows for differentiated inter-token influences during refinement. TTCD outperforms discrete models at high speedups. We train a 160M parameter TTCD model on OpenWebText, and then self-distill it; we find that at high speedups we are comparable in unconditional generation quality, and outperform in conditional generation, several existing models of similar size trained, on the same data, and self-distilled. We achieve similar gains in Sudoku solving as well.

arXiv:2607.13686v2cs.LG

Optimal and Efficient Contextual Combinatorial Semi-bandits with General Function Approximation

We study the contextual combinatorial semi-bandit (CCSB) problem, where at each round, the learner observes a context, selects a combinatorial action consisting of a subset of basic arms, and receives the reward of each selected arm; its goal is to maximize its cumulative reward over time. While many previous works focus on CCSB problems with linear reward function approximation, the design of efficient algorithms for the CCSB problem with general reward function approximation is limited. In this work, we study the realizable setting where the learner has access to a reward function class $\mathcal{F}$ that contains the true reward function, and propose the $\mathrm{SquareCB{.}Comb}$ algorithm. We prove that $\mathrm{SquareCB{.}Comb}$ achieves a minimax optimal regret bound of $O(\sqrt{m A T \log |\mathcal{F}|})$, where $A$ is the number of arms, $m$ is the maximum number of arms in a combinatorial action, and $T$ is the time horizon. In addition, $\mathrm{SquareCB{.}Comb}$ is computationally efficient when we have an online regression oracle that can efficiently predict rewards of all arms given contexts, and the combinatorial action set admits either efficient linear optimization or efficient sampling given any target marginal probability on arms. We empirically verify the effectiveness of our algorithm in webpage recommendation and contextual shortest path problems.

arXiv:2607.08124v3cs.SE

TTHE: Test-Time Harness Evolution

The behavior of an LLM agent is determined not only by the underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures. Existing approaches typically optimize harnesses on training or development data and freeze the resulting programs at test time, limiting adaptation to shifts in task distributions, failure modes, and tool interactions. We ask whether the harness can instead be optimized during evaluation using unlabeled execution traces. We introduce Test-Time Harness Evolution (TTHE), which treats the executable harness as the state of test-time adaptation. On each unlabeled test batch, TTHE iteratively evolves a population of candidate harnesses through agentic proposers that reason over execution traces and explore diverse edit directions. A judge selects a candidate using execution traces and execution-derived proxy signals. The selected harness determines the current batch's outputs and initializes adaptation on the next. Crucially, the solver, proposers, and judge operate through different harnesses around the same frozen LLM, without weight updates, access to final-answer labels, or a separately trained adaptation model. Across six execution-grounded benchmarks spanning text-to-SQL, competitive programming, software engineering, data-science coding, terminal-based tasks, and agentic tool use, TTHE improves fixed ReAct-style baseline harnesses through persistent, inspectable program adaptation. These results recast test-time adaptation for LLM agents as evolution over executable control programs and identify execution-derived proxy reliability as a central challenge for robust unsupervised agent improvement.

arXiv:2607.07888v2cs.LG

Distributed Sketching on Data Partitions for OLS Regression

This paper studies a scalable approach for ordinary least squares (OLS) regression based on distributed sketching. It distributes small sketches of a large data set across multiple workers, has each worker construct a local OLS estimator from its sketch, and averages the local estimators into a global one. Prior studies assume each sketch is generated from all data, but this sketching step has high computation cost, sometimes even higher than standard OLS. Instead, we propose to generate each sketch from a partitioned subset of the data, which allows the sketching cost to decrease with more workers. Under the fixed design setting, we characterize the exact excess loss of the global estimator. This loss is higher than the loss established in prior studies, but remains close to it when the average Burg divergence between subset covariances is small. Since more workers tend to increase the divergence under our partition design, this reveals a tradeoff between the computational and statistical efficiency of our estimator.

arXiv:2607.07779v4cs.CL

From Solvers to Research: Large Language Model-Driven Formal Mathematics at the Research Frontier

Large language models have made rapid progress in generating machine-checked proofs for well-defined mathematical problems. Extending this progress to research mathematics requires more than solving isolated theorems. Research involves choosing questions, developing definitions and conjectures, connecting results, and combining formal proofs with expert judgment. This survey examines how LLM-driven systems can support these broader activities. We review formal-mathematics tasks, datasets, and autoformalization, and organize proof-generation methods into three overlapping areas: training strategies, inference-time search and adaptation, and agentic workflows. We then examine benchmark results, selected research projects, and platforms for collaborative mathematical work. A central distinction runs throughout the survey: checking a formal proof does not by itself establish that it captures the intended problem or makes a new mathematical contribution. We therefore distinguish theorem benchmarks, repository tasks, formalization projects, and research claims, and discuss the evidence needed to assess each. Finally, we identify five priorities for progress: better data and evaluation, representations of mathematical relationships, conjecture and concept generation, reliable tool integration, and human--AI collaboration. Together, these perspectives connect advances in proof generation to the practical demands of mathematical research.