PyTorchKR Papers

“Code Generation”

총 10,140편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2508.02038v5cs.CL

Marco-Voice 기술 보고서

Marco-Voice Technical Report

본 논문은 음성 복제(voice cloning)와 감정 제어 음성 합성을 단일 프레임워크 안에서 통합한 다기능 음성 합성 시스템을 제시한다. 본 연구의 목표는 다양한 언어적·감정적 맥락에서 화자 정체성을 충실히 보존하면서, 높은 표현력·제어 가능성·자연스러움을 갖춘 음성 생성을 달성하는 데 있어 오랫동안 남아 온 과제를 해결하는 것이다. 우리 접근법은 배치 내 대조 학습(contrastive learning)을 활용한 효과적인 화자-감정 분리(disentanglement) 메커니즘을 도입하여 화자 정체성과 감정적 스타일을 독립적으로 조작할 수 있게 하며, 부드러운 감정 제어를 위한 회전 감정 임베딩 통합 방법도 제안한다. 포괄적인 학습과 평가를 지원하기 위해, 일곱 가지 감정 범주에 걸쳐 여섯 명의 전문 화자가 녹음한 10시간 분량의 표준 중국어(Mandarin) 음성을 담은 고품질 감정 음성 데이터셋 CSEMOTIONS를 구축하였다. 광범위한 실험을 통해 우리 시스템 Marco-Voice가 객관적·주관적 지표 모두에서 큰 폭의 성능 향상을 달성함을 보였다. 포괄적인 평가와 분석을 수행한 결과, MarcoVoice는 음성 선명도와 감정적 풍부함 측면에서 경쟁력 있는 성능을 보이며, 표현력 있는 신경 음성 합성 분야에서 실질적인 진전을 나타낸다. 코드와 데이터셋은 각각 https://github.com/AIDC-AI/Marco-Voice 및 https://huggingface.co/datasets/AIDC-AI/CSEMOTIONS 에서 공개되어 있다.

arXiv:2609.23697v1cs.CL

신뢰하는 것을 증류하라: 신뢰도 인식 다중 교사 온-정책 증류

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

다중 교사 온-정책(on-policy) 증류(distillation)는 학생 모델이 자체 궤적(trajectory) 상에서 상호보완적인 전문가 모델들로부터 학습(training)할 수 있도록 합니다. 그러나 도메인 라우팅 방식은 예시당 하나의 교사를 선택하고 응답 전체에 걸쳐 이를 고정합니다. 이러한 설계는 혼합 학습 말뭉치에 종종 결여되어 있는 레이블에 의존할 뿐만 아니라, 궤적 내에서 필요한 전문성이 변할 때 교사 선택을 적응적으로 조정하지 못합니다. 본 논문에서는 예시 수준의 교사 선택을 레이블이 필요 없는 토큰(token) 수준의 지도 할당으로 대체하는 \textbf{TrustMOPD}를 제안합니다. 학생 모델이 생성한 각 접두 문맥(prefix)에서, TrustMOPD는 공유된 사전 강화학습(pre-RL) 참조 모델 대비 각 전문가의 강화학습 유도 변위를 국소 신뢰도의 대리 지표로 사용하고, 교사 간에 이 점수들을 보정하여 가중 증류 목표를 구성합니다. 수학, 코드 및 인스트럭션 따르기 전반에 걸쳐 TrustMOPD는 가장 강력한 무라벨 기준선(baseline)을 능가하여, \textsc{SingleCap}에서는 복구율을 $54.4\%$에서 $91.5\%$로, \textsc{MultiCap}에서는 $54.5\%$에서 $98.0\%$로 높이는 동시에 \textsc{SingleCap}에서 레이블 기반 MOPD에 근접합니다. 학생 모델이 생성한 접두 문맥과 무관하게 토큰 수준 가중치를 무작위화하는 것은 균일 가중 방식보다 나을 바가 없었으며, 이는 점진적으로 전개되는 생성 문맥에 따라 지도를 조건화하는 것의 중요성을 뒷받침합니다.

arXiv:2609.24855v1cs.AI

논증을 분류만이 아니라 추출하기: 생성형 구성요소 탐지를 위한 지시 미세조정 LLM

Extracting Arguments, Not Just Classifying Them: Instruction-Tuned LLMs for Generative Component Detection

논증적 구성요소 탐지(ACD)는 논증 마이닝(AM)의 핵심 하위 과제이며, 주장(claim)과 전제(premise) 등의 구성요소로 논증 구간을 동시에 한정하고 분류해야 하므로 그중에서도 가장 어려운 측면 중 하나이다. 이 하위 과제에 대한 연구는 다른 AM 과제에 비해 상대적으로 제한적이지만, 기존 접근법의 대부분은 이를 단순화된 시퀀스 라벨링 문제, 구성요소 분류, 또는 구성요소 분할 뒤 분류하는 파이프라인으로 형식화한다. 본 논문에서는 간결한 지시 기반 프롬프트를 사용하는 지시 미세조정 대규모 언어 모델(LLM)에 기반한 새로운 접근법 ITFACD를 제안하고, ACD를 언어 생성 태스크로 재구성하여 사전 분할된 구성요소에 의존하지 않고 평문에서 논증을 직접 식별할 수 있게 한다. 표준 벤치마크에서의 실험 결과, 본 접근법이 최신(state-of-the-art) 시스템보다 더 높은 성능을 달성함을 보여준다. 우리가 아는 한, 이는 ACD를 생성 태스크로 완전히 모델링한 최초의 시도 중 하나이며, 복잡한 AM 문제에 대한 지시 미세조정의 잠재력을 부각한다. 코드와 사용된 데이터셋은 다음 GitHub 저장소에서 공개적으로 이용할 수 있다.

arXiv:2108.06682

ST3D++: Denoised Self-training for Unsupervised Domain Adaptation on 3D Object Detection

In this paper, we present a self-training method, named ST3D++, with a holistic pseudo label denoising pipeline for unsupervised domain adaptation on 3D object detection. ST3D++ aims at reducing noise in pseudo label generation as well as alleviating the negative impacts of noisy pseudo labels on model training. First, ST3D++ pre-trains the 3D object detector on the labeled source domain with random object scaling (ROS) which is designed to reduce target domain pseudo label noise arising from object scale bias of the source domain. Then, the detector is progressively improved through alternating between generating pseudo labels and training the object detector with pseudo-labeled target domain data. Here, we equip the pseudo label generation process with a hybrid quality-aware triplet memory to improve the quality and stability of generated pseudo labels. Meanwhile, in the model training stage, we propose a source data assisted training strategy and a curriculum data augmentation policy to effectively rectify noisy gradient directions and avoid model over-fitting to noisy pseudo labeled data. These specific designs enable the detector to be trained on meticulously refined pseudo labeled target data with denoised training signals, and thus effectively facilitate adapting an object detector to a target domain without requiring annotations. Finally, our method is assessed on four 3D benchmark datasets (i.e., Waymo, KITTI, Lyft, and nuScenes) for three common categories (i.e., car, pedestrian and bicycle). ST3D++ achieves state-of-the-art performance on all evaluated settings, outperforming the corresponding baseline by a large margin (e.g., 9.6% $\sim$ 38.16% on Waymo $\rightarrow$ KITTI in terms of AP$_{\text{3D}}$), and even surpasses the fully supervised oracle results on the KITTI 3D object detection benchmark with target prior. Code will be available.

arXiv:2301.04545공식 코드 · Papers with Code

AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformers

In this paper, we present a new method that reformulates point cloud completion as a set-to-set translation problem and design a new model, called PoinTr, which adopts a Transformer encoder-decoder architecture for point cloud completion. By representing the point cloud as a set of unordered groups of points with position embeddings, we convert the input data to a sequence of point proxies and employ the Transformers for generation. To facilitate Transformers to better leverage the inductive bias about 3D geometric structures of point clouds, we further devise a geometry-aware block that models the local geometric relationships explicitly. The migration of Transformers enables our model to better learn structural knowledge and preserve detailed information for point cloud completion. Taking a step towards more complicated and diverse situations, we further propose AdaPoinTr by developing an adaptive query generation mechanism and designing a novel denoising task during completing a point cloud. Coupling these two techniques enables us to train the model efficiently and effectively: we reduce training time (by 15x or more) and improve completion performance (over 20%). We also show our method can be extended to the scene-level point cloud completion scenario by designing a new geometry-enhanced semantic scene completion framework. Extensive experiments on the existing and newly-proposed datasets demonstrate the effectiveness of our method, which attains 6.53 CD on PCN, 0.81 CD on ShapeNet-55 and 0.392 MMD on real-world KITTI, surpassing other work by a large margin and establishing new state-of-the-arts on various benchmarks. Most notably, AdaPoinTr can achieve such promising performance with higher throughputs and fewer FLOPs compared with the previous best methods in practice. The code and datasets are available at https://github.com/yuxumin/PoinTr

arXiv:2410.12046

Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings

When a Commit Message Generation (CMG) system is integrated into the IDEs and other products at JetBrains, we perform online evaluation based on user acceptance of the generated messages. However, performing online experiments with every change to a CMG system is troublesome, as each iteration affects users and requires time to collect enough statistics. On the other hand, offline evaluation, a prevalent approach in the research literature, facilitates fast experiments but employs automatic metrics that are not guaranteed to represent the preferences of real users. In this work, we describe a novel way we employed to deal with this problem at JetBrains, by leveraging an online metric - the number of edits users introduce before committing the generated messages to the VCS - to select metrics for offline experiments. To support this new type of evaluation, we develop a novel markup collection tool mimicking the real workflow with a CMG system, collect a dataset with 57 pairs consisting of commit messages generated by GPT-4 and their counterparts edited by human experts, and design and verify a way to synthetically extend such a dataset. Then, we use the final dataset of 656 pairs to study how the widely used similarity metrics correlate with the online metric reflecting the real users' experience. Our results indicate that edit distance exhibits the highest correlation with the online metric, whereas commonly used similarity metrics such as BLEU and METEOR demonstrate low correlation. This contradicts the previous studies on similarity metrics for CMG, suggesting that user interactions with a CMG system in real-world settings differ significantly from the responses by human labelers within controlled environments. We release all the code and the dataset to support future research in the field: https://jb.gg/cmg-evaluation.

arXiv:2604.12002v2cs.CL

Self-Distillation Zero: 자기 수정이 이진 보상을 밀집 감독으로 변환한다

Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

SD-Zero는 외부 교사나 고품질 데모 없이 이진 보상을 토큰 수준의 밀집 감독으로 변환하는 사후학습 방법으로, 단일 모델이 생성자와 수정자 역할을 수행하며 자가 증류를 통해 성능을 향상시킵니다. 수학 및 코드 추론 벤치마크에서 기본 모델 대비 최소 10% 이상의 성능 향상을 달성하여 RFT나 GRPO 같은 강력한 기준선을 능가했습니다.

arXiv:1805.00917공식 코드 · Papers with Code

A Scalable Discrete-Time Survival Model for Neural Networks

There is currently great interest in applying neural networks to prediction tasks in medicine. It is important for predictive models to be able to use survival data, where each patient has a known follow-up time and event/censoring indicator. This avoids information loss when training the model and enables generation of predicted survival curves. In this paper, we describe a discrete-time survival model that is designed to be used with neural networks, which we refer to as Nnet-survival. The model is trained with the maximum likelihood method using minibatch stochastic gradient descent (SGD). The use of SGD enables rapid convergence and application to large datasets that do not fit in memory. The model is flexible, so that the baseline hazard rate and the effect of the input data on hazard probability can vary with follow-up time. It has been implemented in the Keras deep learning framework, and source code for the model and several examples is available online. We demonstrate the performance of the model on both simulated and real data and compare it to existing models Cox-nnet and Deepsurv.

arXiv:2104.00675

In&Out : Diverse Image Outpainting via GAN Inversion

Image outpainting seeks for a semantically consistent extension of the input image beyond its available content. Compared to inpainting -- filling in missing pixels in a way coherent with the neighboring pixels -- outpainting can be achieved in more diverse ways since the problem is less constrained by the surrounding pixels. Existing image outpainting methods pose the problem as a conditional image-to-image translation task, often generating repetitive structures and textures by replicating the content available in the input image. In this work, we formulate the problem from the perspective of inverting generative adversarial networks. Our generator renders micro-patches conditioned on their joint latent code as well as their individual positions in the image. To outpaint an image, we seek for multiple latent codes not only recovering available patches but also synthesizing diverse outpainting by patch-based generation. This leads to richer structure and content in the outpainted regions. Furthermore, our formulation allows for outpainting conditioned on the categorical input, thereby enabling flexible user controls. Extensive experimental results demonstrate the proposed method performs favorably against existing in- and outpainting methods, featuring higher visual quality and diversity.

arXiv:2104.08453공식 코드 · Papers with Code

R&R: Metric-guided Adversarial Sentence Generation

Adversarial examples are helpful for analyzing and improving the robustness of text classifiers. Generating high-quality adversarial examples is a challenging task as it requires generating fluent adversarial sentences that are semantically similar to the original sentences and preserve the original labels, while causing the classifier to misclassify them. Existing methods prioritize misclassification by maximizing each perturbation's effectiveness at misleading a text classifier; thus, the generated adversarial examples fall short in terms of fluency and similarity. In this paper, we propose a rewrite and rollback (R&R) framework for adversarial attack. It improves the quality of adversarial examples by optimizing a critique score which combines the fluency, similarity, and misclassification metrics. R&R generates high-quality adversarial examples by allowing exploration of perturbations that do not have immediate impact on the misclassification metric but can improve fluency and similarity metrics. We evaluate our method on 5 representative datasets and 3 classifier architectures. Our method outperforms current state-of-the-art in attack success rate by +16.2%, +12.8%, and +14.0% on the classifiers respectively. Code is available at https://github.com/DAI-Lab/fibber

arXiv:2109.09163공식 코드 · Papers with Code

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

Task-relevant grasping is critical for industrial assembly, where downstream manipulation tasks constrain the set of valid grasps. Learning how to perform this task, however, is challenging, since task-relevant grasp labels are hard to define and annotate. There is also yet no consensus on proper representations for modeling or off-the-shelf tools for performing task-relevant grasps. This work proposes a framework to learn task-relevant grasping for industrial objects without the need of time-consuming real-world data collection or manual annotation. To achieve this, the entire framework is trained solely in simulation, including supervised training with synthetic label generation and self-supervised, hand-object interaction. In the context of this framework, this paper proposes a novel, object-centric canonical representation at the category level, which allows establishing dense correspondence across object instances and transferring task-relevant grasps to novel instances. Extensive experiments on task-relevant grasping of densely-cluttered industrial objects are conducted in both simulation and real-world setups, demonstrating the effectiveness of the proposed framework. Code and data are available at https://sites.google.com/view/catgrasp.

arXiv:2112.02468공식 코드 · Papers with Code

Anomaly Detection of Wind Turbine Time Series using Variational Recurrent Autoencoders

Ice accumulation in the blades of wind turbines can cause them to describe anomalous rotations or no rotations at all, thus affecting the generation of electricity and power output. In this work, we investigate the problem of ice accumulation in wind turbines by framing it as anomaly detection of multi-variate time series. Our approach focuses on two main parts: first, learning low-dimensional representations of time series using a Variational Recurrent Autoencoder (VRAE), and second, using unsupervised clustering algorithms to classify the learned representations as normal (no ice accumulated) or abnormal (ice accumulated). We have evaluated our approach on a custom wind turbine time series dataset, for the two-classes problem (one normal versus one abnormal class), we obtained a classification accuracy of up to 96$\%$ on test data. For the multiple-class problem (one normal versus multiple abnormal classes), we present a qualitative analysis of the low-dimensional learned latent space, providing insights into the capacities of our approach to tackle such problem. The code to reproduce this work can be found here https://github.com/agrija9/Wind-Turbines-VRAE-Paper.

arXiv:2211.02397

Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration

Diffusion-based generative models have had a high impact on the computer vision and speech processing communities these past years. Besides data generation tasks, they have also been employed for data restoration tasks like speech enhancement and dereverberation. While discriminative models have traditionally been argued to be more powerful e.g. for speech enhancement, generative diffusion approaches have recently been shown to narrow this performance gap considerably. In this paper, we systematically compare the performance of generative diffusion models and discriminative approaches on different speech restoration tasks. For this, we extend our prior contributions on diffusion-based speech enhancement in the complex time-frequency domain to the task of bandwith extension. We then compare it to a discriminatively trained neural network with the same network architecture on three restoration tasks, namely speech denoising, dereverberation and bandwidth extension. We observe that the generative approach performs globally better than its discriminative counterpart on all tasks, with the strongest benefit for non-additive distortion models, like in dereverberation and bandwidth extension. Code and audio examples can be found online at https://uhh.de/inf-sp-sgmsemultitask

arXiv:2212.09741공식 코드 · Papers with Code

One Embedder, Any Task: Instruction-Finetuned Text Embeddings

We introduce INSTRUCTOR, a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). Unlike encoders from prior work that are more specialized, INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. We first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture with a contrastive loss. We evaluate INSTRUCTOR on 70 embedding evaluation tasks (66 of which are unseen during training), ranging from classification and information retrieval to semantic textual similarity and text generation evaluation. INSTRUCTOR, while having an order of magnitude fewer parameters than the previous best model, achieves state-of-the-art performance, with an average improvement of 3.4% compared to the previous best results on the 70 diverse datasets. Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets. Our model, code, and data are available at https://instructor-embedding.github.io.

arXiv:2302.04460공식 코드 · Papers with Code

Bag of Tricks for Training Data Extraction from Language Models

With the advance of language models, privacy protection is receiving more attention. Training data extraction is therefore of great importance, as it can serve as a potential tool to assess privacy leakage. However, due to the difficulty of this task, most of the existing methods are proof-of-concept and still not effective enough. In this paper, we investigate and benchmark tricks for improving training data extraction using a publicly available dataset. Because most existing extraction methods use a pipeline of generating-then-ranking, i.e., generating text candidates as potential training data and then ranking them based on specific criteria, our research focuses on the tricks for both text generation (e.g., sampling strategy) and text ranking (e.g., token-level criteria). The experimental results show that several previously overlooked tricks can be crucial to the success of training data extraction. Based on the GPT-Neo 1.3B evaluation results, our proposed tricks outperform the baseline by a large margin in most cases, providing a much stronger baseline for future research. The code is available at https://github.com/weichen-yu/LM-Extraction.

arXiv:2305.06424공식 코드 · Papers with Code

Bot or Human? Detecting ChatGPT Imposters with A Single Question

Large language models (LLMs) like GPT-4 have recently demonstrated impressive capabilities in natural language understanding and generation. However, there is a concern that they can be misused for malicious purposes, such as fraud or denial-of-service attacks. Therefore, it is crucial to develop methods for detecting whether the party involved in a conversation is a bot or a human. In this paper, we propose a framework named FLAIR, Finding Large Language Model Authenticity via a Single Inquiry and Response, to detect conversational bots in an online manner. Specifically, we target a single question scenario that can effectively differentiate human users from bots. The questions are divided into two categories: those that are easy for humans but difficult for bots (e.g., counting, substitution, searching, and ASCII art reasoning), and those that are easy for bots but difficult for humans (e.g., memorization and computation). Our approach shows different strengths of these questions in their effectiveness, providing a new way for online service providers to protect themselves against nefarious activities. Our code and question set are available at https://github.com/hongwang600/FLAIR.

arXiv:2311.14282공식 코드 · Papers with Code

Image Super-Resolution with Text Prompt Diffusion

Image super-resolution (SR) methods typically model degradation to improve reconstruction accuracy in complex and unknown degradation scenarios. However, extracting degradation information from low-resolution images is challenging, which limits the model performance. To boost image SR performance, one feasible approach is to introduce additional priors. Inspired by advancements in multi-modal methods and text prompt image processing, we introduce text prompts to image SR to provide degradation priors. Specifically, we first design a text-image generation pipeline to integrate text into the SR dataset through the text degradation representation and degradation model. The text representation applies a discretization manner based on the binning method to describe the degradation abstractly. This method maintains the flexibility of the text and is user-friendly. Meanwhile, we propose the PromptSR to realize the text prompt SR. The PromptSR utilizes the pre-trained language model (e.g., T5 or CLIP) to enhance restoration. We train the PromptSR on the generated text-image dataset. Extensive experiments indicate that introducing text prompts into SR, yields excellent results on both synthetic and real-world images. Code is available at: https://github.com/zhengchen1999/PromptSR.

arXiv:2405.13028공식 코드 · Papers with Code

DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues

User Simulators play a pivotal role in training and evaluating task-oriented dialogue systems. Traditional user simulators typically rely on human-engineered agendas, resulting in generated responses that often lack diversity and spontaneity. Although large language models (LLMs) exhibit a remarkable capacity for generating coherent and contextually appropriate utterances, they may fall short when tasked with generating responses that effectively guide users towards their goals, particularly in dialogues with intricate constraints and requirements. This paper introduces DuetSim, a novel framework designed to address the intricate demands of task-oriented dialogues by leveraging LLMs. DuetSim stands apart from conventional approaches by employing two LLMs in tandem: one dedicated to response generation and the other focused on verification. This dual LLM approach empowers DuetSim to produce responses that not only exhibit diversity but also demonstrate accuracy and are preferred by human users. We validate the efficacy of our method through extensive experiments conducted on the MultiWOZ dataset, highlighting improvements in response quality and correctness, largely attributed to the incorporation of the second LLM. Our code is accessible at: https://github.com/suntea233/DuetSim.

arXiv:2405.19783공식 코드 · Papers with Code

Instruction-Guided Visual Masking

Instruction following is crucial in contemporary LLM. However, when extended to multimodal setting, it often suffers from misalignment between specific textual instruction and targeted local region of an image. To achieve more accurate and nuanced multimodal instruction following, we introduce Instruction-guided Visual Masking (IVM), a new versatile visual grounding model that is compatible with diverse multimodal models, such as LMM and robot model. By constructing visual masks for instruction-irrelevant regions, IVM-enhanced multimodal models can effectively focus on task-relevant image regions to better align with complex instructions. Specifically, we design a visual masking data generation pipeline and create an IVM-Mix-1M dataset with 1 million image-instruction pairs. We further introduce a new learning technique, Discriminator Weighted Supervised Learning (DWSL) for preferential IVM training that prioritizes high-quality data samples. Experimental results on generic multimodal tasks such as VQA and embodied robotic control demonstrate the versatility of IVM, which as a plug-and-play tool, significantly boosts the performance of diverse multimodal models, yielding new state-of-the-art results across challenging multimodal benchmarks. Code, model and data are available at https://github.com/2toinf/IVM.

arXiv:2406.08316

Is Programming by Example solved by LLMs?

Programming-by-Examples (PBE) aims to generate an algorithm from input-output examples. Such systems are practically and theoretically important: from an end-user perspective, they are deployed to millions of people, and from an AI perspective, PBE corresponds to a very general form of few-shot inductive inference. Given the success of Large Language Models (LLMs) in code-generation tasks, we investigate here the extent to which LLMs can be said to have "solved" PBE. We experiment on classic domains such as lists and strings, and an uncommon graphics programming domain not well represented in typical pretraining data. We find that pretrained models are not effective at PBE, but that they can be fine-tuned for much higher performance, provided the test problems are in-distribution. We analyze empirically what causes these models to succeed and fail, and take steps toward understanding how to achieve better out-of-distribution generalization. Collectively these results suggest that LLMs make strong progress toward solving the typical suite of PBE tasks, potentially increasing the flexibility and applicability of PBE systems, while also identifying ways in which LLMs still fall short.