기존의 언어 모델 지식 증류는 고정된 교사 생성 데이터에 의존하는 경우가 많은데, 이는 진화하는 학생 정책이 마주치는 상태를 충분히 포괄하지 못할 수 있다. 온-정책 지식 증류(OPD)는 대신 학생이 생성한 롤아웃에 대해 교사 또는 평가자의 감독을 수집한다. 그러나 기존 OPD 방법들은 감독 형식, 토크나이저 호환성, 교사 접근 방식, 감독 세분성 면에서 크게 상이하여, 재현 및 확장이 어려운 파편화된 구현들로 이어졌다. 우리는 대규모 언어 모델을 위한 분산 강화학습 프레임워크인 verl 위에 구축된 온-정책 지식 증류 프레임워크 EasyOPD를 제시한다. EasyOPD는 사용자 측 설정, 방법별 감독 로직, verl 기반 실행을 분리한다. 방법 모듈은 손실 구성, 롤아웃 메타데이터, 보상 처리, 토크나이저 정렬, 교사 측 연산을 위한 확장 경계를 통해 공유 백엔드에 연결된다. 우리는 교차-토크나이저 OPD, 온-정책 자기 지식 증류, 단계별 OPD의 세 가지 OPD 설정에 대한 대표 방법을 구현한다. 추론, 코드 생성, 과학 지식, 도구 사용 벤치마크에서의 실험 결과, 이 구현들이 방법별 목표와 과제 의존적 성능 프로파일을 유지하면서 동일한 verl 기반 백엔드를 통해 실행될 수 있음을 보여준다. 우리는 실행 가능한 YAML 설정, 문서, 설치 가능한 시연 패키지 및 영상과 함께 EasyOPD를 공개한다.
대규모 멀티모달 모델의 최근 발전은 비전-언어 과제에서 눈부신 성과를 이끌었지만, 풍부하고 명시적으로 정렬된 청각 단서를 포함한 데이터셋의 부족으로 인해 포괄적인 옴니모달 이해는 여전히 충분히 탐구되지 않은 상태이다. 이 격차를 해소하기 위해, 우리는 시각 및 청각 의미론을 분리하도록 설계된 대규모 데이터셋인 AVDC(Audio-Visual Decoupled Captions)를 제시한다. 구체적으로, 기성 모델을 활용하여 영상에 세 가지 캡션—시각 전용(V), 오디오 전용(A), 오디오-시각 결합(AV)—을 자동으로 주석 처리하는 파이프라인을 제안한다. 이 분리된 구조는 각 모달리티 고유의 미묘한 특성과 복잡한 교차 모달 상호작용을 명시적으로 포착한다. 이를 바탕으로, 오디오-시각 추론을 촉진하기 위한 사고의 연쇄(Chain-of-Thought) 기반 질의응답 데이터셋인 AVDC-QA-CoT를 소개한다. 이러한 자원을 충분히 활용하기 위해, AVDC에 대한 옴니모달 캡션 생성 사전학습에 이어 AVDC-QA-CoT에 대한 지시 미세조정을 수행하는 2단계 학습 패러다임을 채택한다. 영상 캡션 생성, 오디오 중심 분석, 옴니모달 벤치마크를 아우르는 다양한 다운스트림 과제에 걸친 광범위한 실험을 통해 일관되고 유의미한 성능 향상이 확인되었으며, 이는 제안된 데이터셋과 학습 전략이 옴니모달 인식 발전에 기여함을 보여준다. 코드와 데이터셋은 https://radiant0726.github.io/AVDC-web/ 에서 공개된다.
대부분의 이상 탐지(anomaly detection) 시스템은 보정된 의사결정 대신 점수를 출력하므로, 실무자는 명확한 통계적 해석 없이 휴리스틱하게 임계값을 선택해야 하는 한계가 있습니다. 컨포멀 이상 탐지(conformal anomaly detection)는 데이터 교환가능성(exchangeability)이라는 통계적 가정 하에 유효한 보정된 p-값(calibrated p-value)으로 이상치 점수를 변환하여 이러한 한계를 해결하며, 최근 연구들을 통해 이 개념은 해당 가정을 넘어선 환경으로 확장되고 있습니다. 본 논문에서는 기존 머신러닝 워크플로 내에서 컨포멀 이상 탐지를 적용할 수 있는 Python 패키지인 nonconform을 소개하고, 이를 바탕으로 구현 기반의 분야 입문을 제공합니다. 이 패키지는 scikit-learn, PyOD 및 커스텀 이상 탐지기와 연동되며, 보정(calibration), p-값 생성, 위발견율(FDR) 제어를 위한 통합 인터페이스를 제공합니다. 또한 단순한 분할-컨포멀(split-conformal) 보정부터 데이터 효율성이 높고 분포 변화를 고려하는(shift-aware) 확장에 이르기까지 다양한 컨포멀화(conformalization) 전략을 지원합니다. 기초 개념에서 고급 컨포멀화 전략으로 이어지는 구성과 코드 예제를 통해, 본 논문은 컨포멀 이상 탐지 이면의 통계적 개념과 nonconform에서의 실질적인 활용을 연결합니다. 실증적 결과는 구현된 방법들이 통계적으로 원칙에 입각한 이상 탐지를 가능하게 함을 보여줍니다. 이 패키지와 설명은 실험 및 프로덕션 지향 환경에서 핵심 컨포멀 이상 탐지 워크플로의 접근성과 재현성을 높이는 것을 목표로 합니다.
AI 연구소들이 연산 용량이 새로운 고품질 텍스트 생성 속도를 앞서는 데이터 상한에 접근하면서, 언어 모델 사전학습은 고정 코퍼스에 대한 생산적인 다중 에포크 학습을 요구하는 데이터 제약·연산 풍부 체제로 이동하고 있다. 표준 자기회귀(autoregressive, AR) 사전학습은 이 설정에서 심각하게 과적합하며, 일찍 최적점에 도달한 뒤 지속적으로 악화된다. 우리는 이 과적합을 완화하고 동일 데이터에서 수백 에포크에 걸친 생산적 학습을 가능하게 하기 위해 학습 시점 데이터 증강을 정규화(regularizer)로 조사한다. AR 사전학습을 위한 세 가지 직교적 증강 범주를 도입한다: 토큰 수준 잡음(마스킹, 무작위 치환), 시퀀스 순열(우→좌 예측, Fill-in-the-Middle), 목표 오프셋 예측($x_{t+i}$, $i > 1$). 체계적 ablation을 통해 개별 증강이 기준선 대비 과적합을 지연시키고 검증 손실을 낮추며, 무작위 토큰 치환이 개별 방법 중 최소 손실을 가장 잘 달성함을 확인했다. 증강 범주를 결합하면 최소 검증 손실이 더 낮아진다. 실험 결과, 데이터 증강은 AR 사전학습의 데이터 비효율을 완화하며 데이터 제약 체제에 유망한 해결책을 제공함을 보여준다~\footnote{All code and data are available at https://github.com/ michaelchen-lab/ data-augmentations-for-pretraining.
참조 다중 객체 추적(RMOT)은 주어진 언어 표현과 일치하는 비디오 내의 모든 인스턴스를 추적하는 것을 목표로 합니다. 일반화 성능을 향상하기 위해 최근 멀티모달 대규모 언어 모델(MLLM)을 통합하고 있음에도 불구하고, 기존 방법들은 주로 MLLM을 캡션 생성기의 역할로 격하시켜 최종 의사결정을 위한 외부 모듈을 필요로 합니다. 이러한 패러다임은 추가적인 지연 시간을 유발할 뿐만 아니라 MLLM의 고유한 비전-언어 정렬(alignment) 능력을 심각하게 제대로 활용하지 못합니다. 이러한 한계를 해결하기 위해, 본 연구에서는 명시적인 텍스트 생성 없이 Yes/No 검증에 MLLM을 직접 활용하여 참조를 판별(discriminative) 태스크로 재구성하는 새로운 2단계 RMOT 방법인 YesTrack을 제안합니다. 이러한 MLLM 기반 검증의 신뢰성과 효율성을 더욱 향상하기 위해 시간적 신뢰도 사전확률(Temporal Confidence Prior, TCP)과 시간적 참조 전파(Temporal Reference Propagation, TRP)라는 두 가지 경량 시간 일관성 제약 조건을 도입합니다. 또한 일반 다중 객체 추적(MOT)을 위한 직관적이면서도 매우 효과적인 인스턴스화인 YesTrack-MOT를 제안하여 이 판별 패러다임의 범용성을 추가로 검증합니다. Refer-KITTI 및 Refer-KITTI-V2에 대한 실험 결과, YesTrack은 가장 작은 Qwen3-VL 변형으로 구현되었을 때도 높은 효율성을 유지하면서 기존 최첨단(SOTA) 방법들보다 우수한 성능을 크게 보여줍니다. 코드는 https://github.com/ggbondrighthere24/YesTrack 에 공개되어 있습니다.
대규모 언어 모델(LLM) 기반의 원격 탐사(Remote Sensing, RS) 에이전트는 지리공간 분석을 자동화하는 유망한 접근 방식을 제공합니다. 그러나 소형 언어 모델 기반의 경량 RS 에이전트는 장기 상태(state) 손실, 비효율적인 환경 피드백 활용, 희소한 최적화 신호로 인해 다단계 상호작용 작업에서 어려움을 겪습니다. 본 논문에서는 3단계를 거쳐 경량 에이전트를 점진적으로 향상시키는 환경 피드백 기반 프레임워크인 RS-Claw-Evolution을 제안합니다. 상호작용 진화(Interaction evolution)는 실행 가능한 코드를 활용해 관측을 제어하고 중간 상태를 유지하며 문맥 중복을 줄입니다. 경험 진화(Experience evolution)는 오류 인식 궤적 생성과 오류 턴 마스킹을 결합하여, 잘못된 동작을 모방하지 않고 유익한 실패 복구 경험으로부터 학습합니다. 의사결정 진화(Decision evolution)는 다차원 환경 보상과 턴 단위 어드밴티지 보호를 결합한 강화학습(RL)을 활용하여 도구 사용 행동을 최적화하고 긴 시퀀스에서의 신용 할당(credit assignment)을 개선합니다. Earth-Bench에서 최적화된 Qwen3-4B 기반 에이전트는 자율 계획(Autonomous Planning) 모드에서 65.9%의 정확도를 달성하여, 미학습 Qwen3-32B 기준선(43.8%) 및 DeepSeek-V3.1(60.8%)을 능가하고 GPT-5(71.6%)에 근접하는 성능을 보였습니다. 이러한 결과는 환경 피드백으로부터 학습함으로써 경량 에이전트를 개선하고 장기 RS 작업에서 더 큰 모델과의 성능 격차를 좁힐 수 있음을 보여줍니다.
This paper presents HoughNet, a one-stage, anchor-free, voting-based, bottom-up object detection method. Inspired by the Generalized Hough Transform, HoughNet determines the presence of an object at a certain location by the sum of the votes cast on that location. Votes are collected from both near and long-distance locations based on a log-polar vote field. Thanks to this voting mechanism, HoughNet is able to integrate both near and long-range, class-conditional evidence for visual recognition, thereby generalizing and enhancing current object detection methodology, which typically relies on only local evidence. On the COCO dataset, HoughNet's best model achieves $46.4$ $AP$ (and $65.1$ $AP_{50}$), performing on par with the state-of-the-art in bottom-up object detection and outperforming most major one-stage and two-stage methods. We further validate the effectiveness of our proposal in other visual detection tasks, namely, video object detection, instance segmentation, 3D object detection and keypoint detection for human pose estimation, and an additional "labels to photo" image generation task, where the integration of our voting module consistently improves performance in all cases. Code is available at https://github.com/nerminsamet/houghnet.
Understanding movies and their structural patterns is a crucial task in decoding the craft of video editing. While previous works have developed tools for general analysis, such as detecting characters or recognizing cinematography properties at the shot level, less effort has been devoted to understanding the most basic video edit, the Cut. This paper introduces the Cut type recognition task, which requires modeling multi-modal information. To ignite research in this new task, we construct a large-scale dataset called MovieCuts, which contains 173,967 video clips labeled with ten cut types defined by professionals in the movie industry. We benchmark a set of audio-visual approaches, including some dealing with the problem's multi-modal nature. Our best model achieves 47.7% mAP, which suggests that the task is challenging and that attaining highly accurate Cut type recognition is an open research problem. Advances in automatic Cut-type recognition can unleash new experiences in the video editing industry, such as movie analysis for education, video re-editing, virtual cinematography, machine-assisted trailer generation, machine-assisted video editing, among others. Our data and code are publicly available: https://github.com/PardoAlejo/MovieCuts}{https://github.com/PardoAlejo/MovieCuts.
Structured and grounded representation of text is typically formalized by closed information extraction, the problem of extracting an exhaustive set of (subject, relation, object) triplets that are consistent with a predefined set of entities and relations from a knowledge base schema. Most existing works are pipelines prone to error accumulation, and all approaches are only applicable to unrealistically small numbers of entities and relations. We introduce GenIE (generative information extraction), the first end-to-end autoregressive formulation of closed information extraction. GenIE naturally exploits the language knowledge from the pre-trained transformer by autoregressively generating relations and entities in textual form. Thanks to a new bi-level constrained generation strategy, only triplets consistent with the predefined knowledge base schema are produced. Our experiments show that GenIE is state-of-the-art on closed information extraction, generalizes from fewer training data points than baselines, and scales to a previously unmanageable number of entities and relations. With this work, closed information extraction becomes practical in realistic scenarios, providing new opportunities for downstream tasks. Finally, this work paves the way towards a unified end-to-end approach to the core tasks of information extraction. Code, data and models available at https://github.com/epfl-dlab/GenIE.
This article presents our generative model for rhythm action games together with applications in business operations. Rhythm action games are video games in which the player is challenged to issue commands at the right timings during a music session. The timings are rendered in the chart, which consists of visual symbols, called notes, flying through the screen. We introduce our deep generative model, Gen\'eLive!, which outperforms the state-of-the-art model by taking into account musical structures through beats and temporal scales. Thanks to its favorable performance, Gen\'eLive! was put into operation at KLab Inc., a Japan-based video game developer, and reduced the business cost of chart generation by as much as half. The application target included the phenomenal "Love Live!," which has more than 10 million users across Asia and beyond, and is one of the few rhythm action franchises that has led the online era of the genre. In this article, we evaluate the generative performance of Gen\'eLive! using production datasets at KLab as well as open datasets for reproducibility, while the model continues to operate in their business. Our code and the model, tuned and trained using a supercomputer, are publicly available.
Text-to-image generative models have achieved unprecedented success in generating high-quality images based on natural language descriptions. However, it is shown that these models tend to favor specific social groups when prompted with neutral text descriptions (e.g., 'a photo of a lawyer'). Following Zhao et al. (2021), we study the effect on the diversity of the generated images when adding ethical intervention that supports equitable judgment (e.g., 'if all individuals can be a lawyer irrespective of their gender') in the input prompts. To this end, we introduce an Ethical NaTural Language Interventions in Text-to-Image GENeration (ENTIGEN) benchmark dataset to evaluate the change in image generations conditional on ethical interventions across three social axes -- gender, skin color, and culture. Through ENTIGEN framework, we find that the generations from minDALL.E, DALL.E-mini and Stable Diffusion cover diverse social groups while preserving the image quality. Preliminary studies indicate that a large change in the model predictions is triggered by certain phrases such as 'irrespective of gender' in the context of gender bias in the ethical interventions. We release code and annotated data at https://github.com/Hritikbansal/entigen_emnlp.
Stroke extraction of Chinese characters plays an important role in the field of character recognition and generation. The most existing character stroke extraction methods focus on image morphological features. These methods usually lead to errors of cross strokes extraction and stroke matching due to rarely using stroke semantics and prior information. In this paper, we propose a deep learning-based character stroke extraction method that takes semantic features and prior information of strokes into consideration. This method consists of three parts: image registration-based stroke registration that establishes the rough registration of the reference strokes and the target as prior information; image semantic segmentation-based stroke segmentation that preliminarily separates target strokes into seven categories; and high-precision extraction of single strokes. In the stroke registration, we propose a structure deformable image registration network to achieve structure-deformable transformation while maintaining the stable morphology of single strokes for character images with complex structures. In order to verify the effectiveness of the method, we construct two datasets respectively for calligraphy characters and regular handwriting characters. The experimental results show that our method strongly outperforms the baselines. Code is available at https://github.com/MengLi-l1/StrokeExtraction.
A powerful way to adapt a visual recognition model to a new domain is through image translation. However, common image translation approaches only focus on generating data from the same distribution as the target domain. Given a cross-modal application, such as pedestrian detection from aerial images, with a considerable shift in data distribution between infrared (IR) to visible (RGB) images, a translation focused on generation might lead to poor performance as the loss focuses on irrelevant details for the task. In this paper, we propose HalluciDet, an IR-RGB image translation model for object detection. Instead of focusing on reconstructing the original image on the IR modality, it seeks to reduce the detection loss of an RGB detector, and therefore avoids the need to access RGB data. This model produces a new image representation that enhances objects of interest in the scene and greatly improves detection performance. We empirically compare our approach against state-of-the-art methods for image translation and for fine-tuning on IR, and show that our HalluciDet improves detection accuracy in most cases by exploiting the privileged information encoded in a pre-trained RGB detector. Code: https://github.com/heitorrapela/HalluciDet
Speech synthesis technology has witnessed significant advancements in recent years, enabling the creation of natural and expressive synthetic speech. One area of particular interest is the generation of synthetic child speech, which presents unique challenges due to children's distinct vocal characteristics and developmental stages. This paper presents a novel approach that leverages the Fastpitch text-to-speech (TTS) model for generating high-quality synthetic child speech. This study uses the transfer learning training pipeline. The approach involved finetuning a multi-speaker TTS model to work with child speech. We use the cleaned version of the publicly available MyST dataset (55 hours) for our finetuning experiments. We also release a prototype dataset of synthetic speech samples generated from this research together with model code to support further research. By using a pretrained MOSNet, we conducted an objective assessment that showed a significant correlation between real and synthetic child voices. Additionally, to validate the intelligibility of the generated speech, we employed an automatic speech recognition (ASR) model to compare the word error rates (WER) of real and synthetic child voices. The speaker similarity between the real and generated speech is also measured using a pretrained speaker encoder.
Prompt tuning represents a valuable technique for adapting pre-trained visual-language models (VLM) to various downstream tasks. Recent advancements in CoOp-based methods propose a set of learnable domain-shared or image-conditional textual tokens to facilitate the generation of task-specific textual classifiers. However, those textual tokens have a limited generalization ability regarding unseen domains, as they cannot dynamically adjust to the distribution of testing classes. To tackle this issue, we present a novel Textual-based Class-aware Prompt tuning(TCP) that explicitly incorporates prior knowledge about classes to enhance their discriminability. The critical concept of TCP involves leveraging Textual Knowledge Embedding (TKE) to map the high generalizability of class-level textual knowledge into class-aware textual tokens. By seamlessly integrating these class-aware prompts into the Text Encoder, a dynamic class-aware classifier is generated to enhance discriminability for unseen domains. During inference, TKE dynamically generates class-aware prompts related to the unseen classes. Comprehensive evaluations demonstrate that TKE serves as a plug-and-play module effortlessly combinable with existing methods. Furthermore, TCP consistently achieves superior performance while demanding less training time. Code:https://github.com/htyao89/Textual-based_Class-aware_prompt_tuning/
We present X-MDPT ($\underline{Cross}$-view $\underline{M}$asked $\underline{D}$iffusion $\underline{P}$rediction $\underline{T}$ransformers), a novel diffusion model designed for pose-guided human image generation. X-MDPT distinguishes itself by employing masked diffusion transformers that operate on latent patches, a departure from the commonly-used Unet structures in existing works. The model comprises three key modules: 1) a denoising diffusion Transformer, 2) an aggregation network that consolidates conditions into a single vector for the diffusion process, and 3) a mask cross-prediction module that enhances representation learning with semantic information from the reference image. X-MDPT demonstrates scalability, improving FID, SSIM, and LPIPS with larger models. Despite its simple design, our model outperforms state-of-the-art approaches on the DeepFashion dataset while exhibiting efficiency in terms of training parameters, training time, and inference speed. Our compact 33MB model achieves an FID of 7.42, surpassing a prior Unet latent diffusion approach (FID 8.07) using only $11\times$ fewer parameters. Our best model surpasses the pixel-based diffusion with $\frac{2}{3}$ of the parameters and achieves $5.43 \times$ faster inference. The code is available at https://github.com/trungpx/xmdpt.
Automatic prompt optimization is an important approach to improving the performance of large language models (LLMs). Recent research demonstrates the potential of using LLMs as prompt optimizers, which can generate improved task prompts via iterative refinement. In this paper, we propose a novel perspective to investigate the design of LLM-based prompt optimizers, by drawing an analogy with gradient-based model optimizers. To connect these two approaches, we identify two pivotal factors in model parameter learning: update direction and update method. By systematically analyzing a rich set of improvement strategies on the two aspects, we further develop a capable Gradient-inspired LLM-based Prompt Optimizer called GPO. At each step, it first retrieves relevant prompts from the optimization trajectory as the update direction. Then, it utilizes the generation-based refinement strategy to perform the update, while controlling the edit distance through a cosine-based decay strategy. Extensive experiments demonstrate the effectiveness and efficiency of GPO. In particular, GPO brings an additional improvement of up to 56.8% on Big-Bench Hard and 62.6% on MMLU compared to baseline methods. The code is available at https://github.com/RUCAIBox/GPO.
Speech-driven 3D talking heads generation has emerged as a significant area of interest among researchers, presenting numerous challenges. Existing methods are constrained by animating faces with fixed topologies, wherein point-wise correspondence is established, and the number and order of points remains consistent across all identities the model can animate. In this work, we present \textbf{ScanTalk}, a novel framework capable of animating 3D faces in arbitrary topologies including scanned data. Our approach relies on the DiffusionNet architecture to overcome the fixed topology constraint, offering promising avenues for more flexible and realistic 3D animations. By leveraging the power of DiffusionNet, ScanTalk not only adapts to diverse facial structures but also maintains fidelity when dealing with scanned data, thereby enhancing the authenticity and versatility of generated 3D talking heads. Through comprehensive comparisons with state-of-the-art methods, we validate the efficacy of our approach, demonstrating its capacity to generate realistic talking heads comparable to existing techniques. While our primary objective is to develop a generic method free from topological constraints, all state-of-the-art methodologies are bound by such limitations. Code for reproducing our results, and the pre-trained model are available at https://github.com/miccunifi/ScanTalk .
We introduce a new benchmark, TAPVid-3D, for evaluating the task of long-range Tracking Any Point in 3D (TAP-3D). While point tracking in two dimensions (TAP) has many benchmarks measuring performance on real-world videos, such as TAPVid-DAVIS, three-dimensional point tracking has none. To this end, leveraging existing footage, we build a new benchmark for 3D point tracking featuring 4,000+ real-world videos, composed of three different data sources spanning a variety of object types, motion patterns, and indoor and outdoor environments. To measure performance on the TAP-3D task, we formulate a collection of metrics that extend the Jaccard-based metric used in TAP to handle the complexities of ambiguous depth scales across models, occlusions, and multi-track spatio-temporal smoothness. We manually verify a large sample of trajectories to ensure correct video annotations, and assess the current state of the TAP-3D task by constructing competitive baselines using existing tracking models. We anticipate this benchmark will serve as a guidepost to improve our ability to understand precise 3D motion and surface deformation from monocular video. Code for dataset download, generation, and model evaluation is available at https://tapvid3d.github.io
Existing music captioning methods are limited to generating concise global descriptions of short music clips, which fail to capture fine-grained musical characteristics and time-aware musical changes. To address these limitations, we propose FUTGA, a model equipped with fined-grained music understanding capabilities through learning from generative augmentation with temporal compositions. We leverage existing music caption datasets and large language models (LLMs) to synthesize fine-grained music captions with structural descriptions and time boundaries for full-length songs. Augmented by the proposed synthetic dataset, FUTGA is enabled to identify the music's temporal changes at key transition points and their musical functions, as well as generate detailed descriptions for each music segment. We further introduce a full-length music caption dataset generated by FUTGA, as the augmentation of the MusicCaps and the Song Describer datasets. We evaluate the automatically generated captions on several downstream tasks, including music generation and retrieval. The experiments demonstrate the quality of the generated captions and the better performance in various downstream tasks achieved by the proposed music captioning approach. Our code and datasets can be found in \href{https://huggingface.co/JoshuaW1997/FUTGA}{\textcolor{blue}{https://huggingface.co/JoshuaW1997/FUTGA}}.