PyTorchKR Papers

“Code Generation”

총 10,186편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2503.17309공식 코드 · Papers with Code

LLM+MAP: Bimanual Robot Task Planning using Large Language Models and Planning Domain Definition Language

Bimanual robotic manipulation provides significant versatility, but also presents an inherent challenge due to the complexity involved in the spatial and temporal coordination between two hands. Existing works predominantly focus on attaining human-level manipulation skills for robotic hands, yet little attention has been paid to task planning on long-horizon timescales. With their outstanding in-context learning and zero-shot generation abilities, Large Language Models (LLMs) have been applied and grounded in diverse robotic embodiments to facilitate task planning. However, LLMs still suffer from errors in long-horizon reasoning and from hallucinations in complex robotic tasks, lacking a guarantee of logical correctness when generating the plan. Previous works, such as LLM+P, extended LLMs with symbolic planners. However, none have been successfully applied to bimanual robots. New challenges inevitably arise in bimanual manipulation, necessitating not only effective task decomposition but also efficient task allocation. To address these challenges, this paper introduces LLM+MAP, a bimanual planning framework that integrates LLM reasoning and multi-agent planning, automating effective and efficient bimanual task planning. We conduct simulated experiments on various long-horizon manipulation tasks of differing complexity. Our method is built using GPT-4o as the backend, and we compare its performance against plans generated directly by LLMs, including GPT-4o, V3 and also recent strong reasoning models o1 and R1. By analyzing metrics such as planning time, success rate, group debits, and planning-step reduction rate, we demonstrate the superior performance of LLM+MAP, while also providing insights into robotic reasoning. Code is available at https://github.com/Kchu/LLM-MAP.

arXiv:2503.17489

Judge Anything: MLLM as a Judge Across Any Modality

Evaluating generative foundation models on open-ended multimodal understanding (MMU) and generation (MMG) tasks across diverse modalities (e.g., images, audio, video) poses significant challenges due to the complexity of cross-modal interactions. To this end, the idea of utilizing Multimodal LLMs (MLLMs) as automated judges has emerged, with encouraging results in assessing vision-language understanding tasks. Moving further, this paper extends MLLM-as-a-Judge across modalities to a unified manner by introducing two benchmarks, TaskAnything and JudgeAnything, to respectively evaluate the overall performance and judging capabilities of MLLMs across any-to-any modality tasks. Specifically, TaskAnything evaluates the MMU and MMG capabilities across 15 any-to-any modality categories, employing 1,500 queries curated from well-established benchmarks. Furthermore, JudgeAnything evaluates the judging capabilities of 5 advanced (e.g., GPT-4o and Gemini-2.0-Flash) from the perspectives of Pair Comparison and Score Evaluation, providing a standardized testbed that incorporates human judgments and detailed rubrics. Our extensive experiments reveal that while these MLLMs show promise in assessing MMU (i.e., achieving an average of 66.55% in Pair Comparison setting and 42.79% in Score Evaluation setting), they encounter significant challenges with MMG tasks (i.e., averaging only 53.37% in Pair Comparison setting and 30.05% in Score Evaluation setting), exposing cross-modality biases and hallucination issues. To address this, we present OmniArena, an automated platform for evaluating omni-models and multimodal reward models. Our work highlights the need for fairer evaluation protocols and stronger alignment with human preferences. The source code and dataset are publicly available at: https://urrealhero.github.io/judgeanythingweb/.

arXiv:2507.14111

CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning

The exponential growth in demand for GPU computing resources, driven by the rapid advancement of Large Language Models, has created an urgent need for automated CUDA optimization strategies. While recent advances in LLMs show promise for code generation, current SOTA models (e.g. R1, o1) achieve low success rates in improving CUDA speed. In this paper, we introduce CUDA-L1, an automated reinforcement learning framework for CUDA optimization. CUDA-L1 achieves performance improvements on the CUDA optimization task: trained on NVIDIA A100, it delivers an average speedup of x17.7 across all 250 CUDA kernels of KernelBench, with peak speedups reaching x449. Furthermore, the model also demonstrates excellent portability across GPU architectures, achieving average speedups of x17.8 on H100, x19.0 on RTX 3090, x16.5 on L40, x14.7 on H800, and x13.9 on H20 despite being optimized specifically for A100. Beyond these benchmark results, CUDA-L1 demonstrates several remarkable properties: 1) Discovers a variety of CUDA optimization techniques and learns to combine them strategically to achieve optimal performance; 2) Uncovers fundamental principles of CUDA optimization; 3) Identifies non-obvious performance bottlenecks and rejects seemingly beneficial optimizations that harm performance. The capabilities of CUDA-L1 demonstrate that reinforcement learning can transform an initially poor-performing LLM into an effective CUDA optimizer through speedup-based reward signals alone, without human expertise or domain knowledge. More importantly, the trained RL model extend the acquired reasoning abilities to new kernels. This paradigm opens possibilities for automated optimization of CUDA operations, and holds promise to substantially promote GPU efficiency and alleviate the rising pressure on GPU computing resources.

arXiv:2607.23607v2cs.LG

MS-GPT: MS/MS de novo 구조 규명을 분자-언어 모델의 스펙트럼 유도 사후 분포 쿼리로 재정의

MS-GPT: Rethinking MS/MS De Novo Structure Elucidation as Spectrum-Induced Posterior Querying of a Molecule-Language Model

탠덤 질량 스펙트럼(MS/MS)으로부터 분자 구조를 규명하는 것은 분석화학의 핵심적인 역문제이다. 기존의 MS/MS 식별 접근법 대부분은 참조 라이브러리나 사전 정의된 후보 집합에 의존하는 반면, 드노보(de novo) 방법은 스펙트럼으로부터 직접 구조를 생성하는 것을 목표로 한다. 일반적인 드노보 방식은 스펙트럼으로부터 분자 지문(fingerprint)을 예측한 뒤 이로부터 구조를 디코딩하며, 이를 통해 분자만으로 구성된 대규모 코퍼스에서 디코더를 사전학습할 수 있다. 그러나 이 패러다임은 학습과 추론 간 불일치를 초래한다. 디코더는 분자로부터 계산된 오라클 지문으로 학습되지만, 추론 시에는 잡음이 있는 스펙트럼 유도 지문 사후분포로 질의되며, 이 사후분포는 대개 임계값을 적용한 단일 지문으로 축약된다. 본 논문은 지문 매개 드노보 구조 규명을 조건부 분자 언어 모델에 대한 스펙트럼 유도 사후분포 질의로 재정립한 MS-GPT를 제안한다. MS-GPT는 분자 언어 모델을 지문과 분자식에 조건화하고, 활성 비트 밀도 보정을 통해 스펙트럼 유도 사후분포를 오라클 지문 매니폴드 근방의 지문 질의 대역으로 변환한다. 이 대역 전반에서 샘플링된 후보들을 하나로 모아 생성 빈도와 사후분포 유사도에 따라 순위를 매긴다. 경량 LoRA 어댑터는 사전학습된 분자 사전분포를 보존하면서 도메인별 사후분포 편향을 추가로 완화한다. NPLIB1과 MassSpecGym에서 MS-GPT는 각각 Top-1/Top-10 정확 일치 정확도 17.1%/27.6%와 10.2%/13.2%를 달성하여 새로운 최고 성능을 기록하였으며, 후보 풀을 확장하면 추론 비용이 거의 선형으로 증가하면서 복원 성능이 향상된다. 소스 코드와 모델 체크포인트는 https://github.com/VIKI623/MS-GPT 에서 공개되어 있다.

arXiv:2608.15893v1cs.AI

LLM 기반 소셜 미디어 봇 탐지 시스템의 공격과 방어

Breaking and Defending LLM-Powered Social Media Bot Detection Systems

소셜 미디어 봇의 확산은 허위 정보, 여론 조작, 온라인 플랫폼에 대한 신뢰 침식 등 지속적인 위협을 초래한다. 이에 대응하기 위해 머신러닝 시스템이 봇 활동을 탐지하고 제한하도록 개발되었지만, 공격자는 적대적 학습 및 행동 모방과 같은 기법을 통해 지속적으로 적응하며, 봇과 탐지 도구 사이의 끊임없는 군비 경쟁을 부추긴다. 대규모 언어 모델(LLM)의 최근 발전은 계정과 그 콘텐츠에 대한 더 깊은 의미적·문맥적 분석을 가능하게 하여 봇 탐지를 크게 개선했다. 그러나 이러한 전환은 새로운 공격 표면을 도입하기도 하며, 적대자가 LLM 기반 분류기의 추론 및 생성 메커니즘을 직접 겨냥한 익스플로잇을 제작할 수 있게 한다. Anthropic의 Claude Code Security와 같은 산업용 도구도 보안에 중요한 결정에 LLM을 활용하므로, 이들의 공격 표면에 대한 면밀한 연구가 더욱 필요해진다. 본 연구에서는 LLM 기반 위협 특화 사이버보안 응용의 공격적·방어적 측면을 모두 조사한다. 소셜 미디어 봇 탐지 과제를 중심으로 하지만, 방법론과 통찰은 피싱 탐지, 이메일 분류, 사기 분석 등 LLM 기반 사이버보안 시스템의 넓은 범주로 일반화된다. LLM 기반 분류기의 의미적·문맥적 약점을 체계적으로 악용하는 두 가지 새로운 적대적 공격 전략을 제안하며, 탐지 정확도를 최대 48%까지 저하시킨다. 이러한 위협에 대응하기 위해 적응형 적대적 조건에서도 탐지 신뢰성을 유지하도록 설계된 견고한 다중 LLM 방어 아키텍처를 제안한다. 우리의 해법인 LSABRE(LLM-powered Social Adversarial Bot Recognition Ensemble)는 다중 LLM 프레임워크로, 다양한 공격에 걸쳐 견고성을 크게 향상시키며, 강한 적응형 적대적 압력 하에서도 86%의 탐지 정확도를 유지한다.

arXiv:2610.00332v1cs.LG

가장 취약한 고리: 최악 조건 제약 강화학습을 통한 대규모 언어 모델의 추론 지식 증류

The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning

대규모 언어 모델(LLM)의 추론 능력을 더 작은 학생 모델로 증류(distillation)하는 것은 효율적인 배포를 위한 핵심 과제입니다. 현재의 접근법들은 근본적인 긴장 관계에 직면해 있습니다. 검증 가능한 태스크 보상(예: GRPO를 통한 방식)만을 최적화하면 학생 모델이 결함이 있는 중간 논리를 통해 올바른 최종 답변에 도달하는 보상 해킹(reward hacking)이 발생하며, 교사 모델에 대한 소프트 발산 페널티(예: KL 기반 증류)로 정규화하면 태스크 성능이 희석되고 결정적으로 학생 모델이 한 단계에서의 심각한 논리적 위반을 다른 단계에서의 높은 교사 일치도로 상쇄할 수 있게 됩니다. 우리는 이러한 평균화가 추론의 본질과 근본적으로 어긋난다고 주장합니다. 사고의 연쇄(Chain-of-Thought)는 가장 취약한 고리만큼만 유효하기 때문입니다. 이러한 관찰에 착안하여, 우리는 궤적의 모든 접두 문맥을 따라 교사 로그 가능도(log-likelihood)에 대한 최악 조건 제약 하에서 태스크 보상을 극대화하는 제약 강화학습 문제로 추론 증류를 정형화합니다. 이중 라그랑지안 솔버의 막대한 비용과 Saute와 같은 상태 증강 방식의 테스트 시점 교사 의존성을 피하기 위해, 우리는 하드 제약 의미론을 보존하고 단일 단계 및 장기 항으로의 저분산 정책 그래디언트 분해를 허용하며 페널티 극한에서 거의 확실하게 최악 조건 제약을 증명 가능하게 만족하는 비증강 제약 MDP를 도출합니다. 수학적 추론 및 코드 생성 태스크에 대한 광범위한 실험을 통해, 우리의 방법이 정확도-충실도 파레토 전선을 크게 확장함을 입증합니다. 순수 강화학습의 높은 최종 답변 정확도와 일치하면서도 교사 제약 위반을 대폭 줄임으로써, 궁극적으로 평가된 모든 환경에서 가장 높은 엄밀한 추론 성공률(Reasoning Success Rate)을 달성합니다.

arXiv:1809.06582

Symbolic Tensor Neural Networks for Digital Media - from Tensor Processing via BNF Graph Rules to CREAMS Applications

This tutorial material on Convolutional Neural Networks (CNN) and its applications in digital media research is based on the concept of Symbolic Tensor Neural Networks. The set of STNN expressions is specified in Backus-Naur Form (BNF) which is annotated by constraints typical for labeled acyclic directed graphs (DAG). The BNF induction begins from a collection of neural unit symbols with extra (up to five) decoration fields (including tensor depth and sharing fields). The inductive rules provide not only the general graph structure but also the specific shortcuts for residual blocks of units. A syntactic mechanism for network fragments modularization is introduced via user defined units and their instances. Moreover, the dual BNF rules are specified in order to generate the Dual Symbolic Tensor Neural Network (DSTNN). The joined interpretation of STNN and DSTNN provides the correct flow of gradient tensors, back propagated at the training stage. The proposed symbolic representation of CNNs is illustrated for six generic digital media applications (CREAMS): Compression, Recognition, Embedding, Annotation, 3D Modeling for human-computer interfacing, and data Security based on digital media objects. In order to make the CNN description and its gradient flow complete, for all presented applications, the symbolic representations of mathematically defined loss/gain functions and gradient flow equations for all used core units, are given. The tutorial is to convince the reader that STNN is not only a convenient symbolic notation for public presentations of CNN based solutions for CREAMS problems but also that it is a design blueprint with a potential for automatic generation of application source code.

arXiv:2212.04108공식 코드 · Papers with Code

Shadow Removal by High-Quality Shadow Synthesis

Most shadow removal methods rely on the invasion of training images associated with laborious and lavish shadow region annotations, leading to the increasing popularity of shadow image synthesis. However, the poor performance also stems from these synthesized images since they are often shadow-inauthentic and details-impaired. In this paper, we present a novel generation framework, referred to as HQSS, for high-quality pseudo shadow image synthesis. The given image is first decoupled into a shadow region identity and a non-shadow region identity. HQSS employs a shadow feature encoder and a generator to synthesize pseudo images. Specifically, the encoder extracts the shadow feature of a region identity which is then paired with another region identity to serve as the generator input to synthesize a pseudo image. The pseudo image is expected to have the shadow feature as its input shadow feature and as well as a real-like image detail as its input region identity. To fulfill this goal, we design three learning objectives. When the shadow feature and input region identity are from the same region identity, we propose a self-reconstruction loss that guides the generator to reconstruct an identical pseudo image as its input. When the shadow feature and input region identity are from different identities, we introduce an inter-reconstruction loss and a cycle-reconstruction loss to make sure that shadow characteristics and detail information can be well retained in the synthesized images. Our HQSS is observed to outperform the state-of-the-art methods on ISTD dataset, Video Shadow Removal dataset, and SRD dataset. The code is available at https://github.com/zysxmu/HQSS.

arXiv:2304.05568공식 코드 · Papers with Code

Improving Diffusion Models for Scene Text Editing with Dual Encoders

Scene text editing is a challenging task that involves modifying or inserting specified texts in an image while maintaining its natural and realistic appearance. Most previous approaches to this task rely on style-transfer models that crop out text regions and feed them into image transfer models, such as GANs. However, these methods are limited in their ability to change text style and are unable to insert texts into images. Recent advances in diffusion models have shown promise in overcoming these limitations with text-conditional image editing. However, our empirical analysis reveals that state-of-the-art diffusion models struggle with rendering correct text and controlling text style. To address these problems, we propose DIFFSTE to improve pre-trained diffusion models with a dual encoder design, which includes a character encoder for better text legibility and an instruction encoder for better style control. An instruction tuning framework is introduced to train our model to learn the mapping from the text instruction to the corresponding image with either the specified style or the style of the surrounding texts in the background. Such a training method further brings our method the zero-shot generalization ability to the following three scenarios: generating text with unseen font variation, e.g., italic and bold, mixing different fonts to construct a new font, and using more relaxed forms of natural language as the instructions to guide the generation task. We evaluate our approach on five datasets and demonstrate its superior performance in terms of text correctness, image naturalness, and style controllability. Our code is publicly available. https://github.com/UCSB-NLP-Chang/DiffSTE

arXiv:2306.16741공식 코드 · Papers with Code

Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train

Foundation models have exhibited remarkable success in various applications, such as disease diagnosis and text report generation. To date, a foundation model for endoscopic video analysis is still lacking. In this paper, we propose Endo-FM, a foundation model specifically developed using massive endoscopic video data. First, we build a video transformer, which captures both local and global long-range dependencies across spatial and temporal dimensions. Second, we pre-train our transformer model using global and local views via a self-supervised manner, aiming to make it robust to spatial-temporal variations and discriminative across different scenes. To develop the foundation model, we construct a large-scale endoscopy video dataset by combining 9 publicly available datasets and a privately collected dataset from Baoshan Branch of Renji Hospital in Shanghai, China. Our dataset overall consists of over 33K video clips with up to 5 million frames, encompassing various protocols, target organs, and disease types. Our pre-trained Endo-FM can be easily adopted for a given downstream task via fine-tuning by serving as the backbone. With experiments on 3 different types of downstream tasks, including classification, segmentation, and detection, our Endo-FM surpasses the current state-of-the-art (SOTA) self-supervised pre-training and adapter-based transfer learning methods by a significant margin, such as VCL (3.1% F1, 4.8% Dice, and 5.5% F1 for classification, segmentation, and detection) and ST-Adapter (5.9% F1, 9.6% Dice, and 9.9% F1 for classification, segmentation, and detection). Code, datasets, and models are released at https://github.com/med-air/Endo-FM.

arXiv:2403.16335공식 코드 · Papers with Code

MEDDAP: Medical Dataset Enhancement via Diversified Augmentation Pipeline

The effectiveness of Deep Neural Networks (DNNs) heavily relies on the abundance and accuracy of available training data. However, collecting and annotating data on a large scale is often both costly and time-intensive, particularly in medical cases where practitioners are already occupied with their duties. Moreover, ensuring that the model remains robust across various scenarios of image capture is crucial in medical domains, especially when dealing with ultrasound images that vary based on the settings of different devices and the manual operation of the transducer. To address this challenge, we introduce a novel pipeline called MEDDAP, which leverages Stable Diffusion (SD) models to augment existing small datasets by automatically generating new informative labeled samples. Pretrained checkpoints for SD are typically based on natural images, and training them for medical images requires significant GPU resources due to their heavy parameters. To overcome this challenge, we introduce USLoRA (Ultrasound Low-Rank Adaptation), a novel fine-tuning method tailored specifically for ultrasound applications. USLoRA allows for selective fine-tuning of weights within SD, requiring fewer than 0.1\% of parameters compared to fully fine-tuning only the UNet portion of SD. To enhance dataset diversity, we incorporate different adjectives into the generation process prompts, thereby desensitizing the classifiers to intensity changes across different images. This approach is inspired by clinicians' decision-making processes regarding breast tumors, where tumor shape often plays a more crucial role than intensity. In conclusion, our pipeline not only outperforms classifiers trained on the original dataset but also demonstrates superior performance when encountering unseen datasets. The source code is available at https://github.com/yasamin-med/MEDDAP.

arXiv:2406.12295공식 코드 · Papers with Code

Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding

Large Language Models (LLMs) exhibit impressive capabilities across various applications but encounter substantial challenges such as high inference latency, considerable training costs, and the generation of hallucinations. Collaborative decoding between large and small language models (SLMs) presents a promising strategy to mitigate these issues through methods including speculative decoding, contrastive decoding, and emulator or proxy fine-tuning. However, the specifics of such collaborations, particularly from a unified perspective, remain largely unexplored. Inspired by dual-process cognitive theory, we propose a unified framework in this paper, termed Fast and Slow Generating (FS-GEN). Within this framework, LLMs (sometimes along with SLMs) are categorized as System 2 (slow and deliberate), while independent SLMs are designated as System 1 (fast and intuitive). We provide a comprehensive analysis of these collaborative methodologies, elucidating their common properties and shedding light on the differential knowledge capabilities of System 2 versus System 1 through the FS-GEN framework. Our findings indicate that only a small proportion of collaborative interactions (approximately less than 20\% in most instances) are necessary across various methods. These interactions between System 1 and System 2 conform to a scaling law related to the parameter ratios, enabling predictable collaboration. Furthermore, we explore the specific conditions under which collaboration proves most effective, particularly from an uncertainty perspective, offering novel insights that may guide future optimization efforts. Our research underscores that the fundamental distinction between System 1 and System 2 lies in the uncertainty of next token predictions, where interventions by System 2 are crucial to support System 1. Code for Reproduction: https://github.com/TsinghuaC3I/FS-GEN

arXiv:2408.15205공식 코드 · Papers with Code

Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation

Promptable segmentation typically requires instance-specific manual prompts to guide the segmentation of each desired object. To minimize such a need, task-generic promptable segmentation has been introduced, which employs a single task-generic prompt to segment various images of different objects in the same task. Current methods use Multimodal Large Language Models (MLLMs) to reason detailed instance-specific prompts from a task-generic prompt for improving segmentation accuracy. The effectiveness of this segmentation heavily depends on the precision of these derived prompts. However, MLLMs often suffer hallucinations during reasoning, resulting in inaccurate prompting. While existing methods focus on eliminating hallucinations to improve a model, we argue that MLLM hallucinations can reveal valuable contextual insights when leveraged correctly, as they represent pre-trained large-scale knowledge beyond individual images. In this paper, we utilize hallucinations to mine task-related information from images and verify its accuracy for enhancing precision of the generated prompts. Specifically, we introduce an iterative Prompt-Mask Cycle generation framework (ProMaC) with a prompt generator and a mask generator.The prompt generator uses a multi-scale chain of thought prompting, initially exploring hallucinations for extracting extended contextual knowledge on a test image.These hallucinations are then reduced to formulate precise instance-specific prompts, directing the mask generator to produce masks that are consistent with task semantics by mask semantic alignment. The generated masks iteratively induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. Experiments on 5 benchmarks demonstrate the effectiveness of ProMaC. Code given in https://lwpyh.github.io/ProMaC/.

arXiv:2410.17779공식 코드 · Papers with Code

ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning

Recent advancements in multimodal fusion have witnessed the remarkable success of vision-language (VL) models, which excel in various multimodal applications such as image captioning and visual question answering. However, building VL models requires substantial hardware resources, where efficiency is restricted by two key factors: the extended input sequence of the language model with vision features demands more computational operations, and a large number of additional learnable parameters increase memory complexity. These challenges significantly restrict the broader applicability of such models. To bridge this gap, we propose ADEM-VL, an efficient vision-language method that tunes VL models based on pretrained large language models (LLMs) by adopting a parameter-free cross-attention mechanism for similarity measurements in multimodal fusion. This approach only requires embedding vision features into the language space, significantly reducing the number of trainable parameters and accelerating both training and inference speeds. To enhance representation learning in fusion module, we introduce an efficient multiscale feature generation scheme that requires only a single forward pass through the vision encoder. Moreover, we propose an adaptive fusion scheme that dynamically discards less relevant visual information for each text token based on its attention score. This ensures that the fusion process prioritizes the most pertinent visual features. With experiments on various tasks including visual question answering, image captioning, and instruction-following, we demonstrate that our framework outperforms existing approaches. Specifically, our method surpasses existing methods by an average accuracy of 0.77% on ScienceQA dataset, with reduced training and inference latency, demonstrating the superiority of our framework. The code is available at https://github.com/Hao840/ADEM-VL.

arXiv:2410.18267

Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editing

Large pre-trained models have achieved notable success across a range of downstream tasks. However, recent research shows that a type of adversarial attack ($\textit{i.e.,}$ backdoor attack) can manipulate the behavior of machine learning models through contaminating their training dataset, posing significant threat in the real-world application of large pre-trained model, especially for those customized models. Therefore, addressing the unique challenges for exploring vulnerability of pre-trained models is of paramount importance. Through empirical studies on the capability for performing backdoor attack in large pre-trained models ($\textit{e.g.,}$ ViT), we find the following unique challenges of attacking large pre-trained models: 1) the inability to manipulate or even access large training datasets, and 2) the substantial computational resources required for training or fine-tuning these models. To address these challenges, we establish new standards for an effective and feasible backdoor attack in the context of large pre-trained models. In line with these standards, we introduce our EDT model, an \textbf{E}fficient, \textbf{D}ata-free, \textbf{T}raining-free backdoor attack method. Inspired by model editing techniques, EDT injects an editing-based lightweight codebook into the backdoor of large pre-trained models, which replaces the embedding of the poisoned image with the target image without poisoning the training dataset or training the victim model. Our experiments, conducted across various pre-trained models such as ViT, CLIP, BLIP, and stable diffusion, and on downstream tasks including image classification, image captioning, and image generation, demonstrate the effectiveness of our method. Our code is available in the supplementary material.

arXiv:2412.06674공식 코드 · Papers with Code

EMOv2: Pushing 5M Vision Model Frontier

This work focuses on developing parameter-efficient and lightweight models for dense predictions while trading off parameters, FLOPs, and performance. Our goal is to set up the new frontier of the 5M magnitude lightweight model on various downstream tasks. Inverted Residual Block (IRB) serves as the infrastructure for lightweight CNNs, but no counterparts have been recognized by attention-based design. Our work rethinks the lightweight infrastructure of efficient IRB and practical components in Transformer from a unified perspective, extending CNN-based IRB to attention-based models and abstracting a one-residual Meta Mobile Block (MMBlock) for lightweight model design. Following neat but effective design criterion, we deduce a modern Improved Inverted Residual Mobile Block (i2RMB) and improve a hierarchical Efficient MOdel (EMOv2) with no elaborate complex structures. Considering the imperceptible latency for mobile users when downloading models under 4G/5G bandwidth and ensuring model performance, we investigate the performance upper limit of lightweight models with a magnitude of 5M. Extensive experiments on various vision recognition, dense prediction, and image generation tasks demonstrate the superiority of our EMOv2 over state-of-the-art methods, e.g., EMOv2-1M/2M/5M achieve 72.3, 75.8, and 79.4 Top-1 that surpass equal-order CNN-/Attention-based models significantly. At the same time, EMOv2-5M equipped RetinaNet achieves 41.5 mAP for object detection tasks that surpasses the previous EMO-5M by +2.6. When employing the more robust training recipe, our EMOv2-5M eventually achieves 82.9 Top-1 accuracy, which elevates the performance of 5M magnitude models to a new level. Code is available at https://github.com/zhangzjn/EMOv2.

arXiv:2504.10358

FingER: Content Aware Fine-grained Evaluation with Reasoning for AI-Generated Videos

Recent advances in video generation have posed great challenges in the assessment of AI-generated content, particularly with the emergence of increasingly sophisticated models. The various inconsistencies and defects observed in such videos are inherently complex, making overall scoring notoriously difficult. In this paper, we emphasize the critical importance of integrating fine-grained reasoning into video evaluation, and we propose $\textbf{F}$ing$\textbf{ER}$, a novel entity-level reasoning evaluation framework that first automatically generates $\textbf{F}$ine-grained $\textbf{E}$ntity-level questions, and then answers those questions by a $\textbf{R}$easoning model with scores, which can be subsequently weighted summed to an overall score for different applications. Specifically, we leverage LLMs to derive entity-level questions across five distinct perspectives, which (i) often focus on some specific entities of the content, thereby making answering or scoring much easier by MLLMs, and (ii) are more interpretable. Then we construct a FingER dataset, consisting of approximately 3.3k videos and corresponding 60k fine-grained QA annotations, each with detailed reasons. Based on that, we further investigate various training protocols to best incentivize the reasoning capability of MLLMs for correct answer prediction. Extensive experiments demonstrate that a reasoning model trained using Group Relative Policy Optimization (GRPO) with a cold-start strategy achieves the best performance. Notably, our model surpasses existing methods by a relative margin of $11.8\%$ on GenAI-Bench and $5.5\%$ on MonetBench with only 3.3k training videos, which is at most one-tenth of the training samples utilized by other methods. Our code and dataset will be released soon.

arXiv:2608.01298v2cs.CV

UDT: 데이터 적응형 토큰 축소를 통한 U-Net과 확산 트랜스포머의 조화

UDT: Reconciling U-Nets and Diffusion Transformers with Data-Adaptive Token Reduction

확산 트랜스포머(DiT)는 확장성과 멀티모달 과제에 대한 적응성 덕분에 생성 모델링의 핵심 아키텍처로 부상했다. DiT는 등방성 트랜스포머 블록으로 구성되며 깊이에 따라 표현을 점진적으로 학습하는데, 노이즈 제거 목적함수로 인해 후반부 레이어가 세부 복원에 집중하게 된다. 이는 표현 품질의 저하와 인코더-디코더 거동의 불균형을 초래한다. 표현 정렬(REPA)과 같은 기존 접근법은 학습 정규화를 통해 초기 표현을 강화함으로써 이를 완화한다. 한편 U-Net 방식의 DiT 아키텍처는 수렴 개선을 위해 명시적인 다중 스케일 인코더-디코더 구조를 도입한다. 그러나 이들은 공간 다운샘플링에 학습 가능한 연산자를 쓰는 표준 U-Net의 관행에 기반하는데, 이는 트랜스포머 아키텍처에 적합하지 않아 비효율을 낳고 교차 어텐션 및 표현 정규화와 같은 구성 요소와의 호환성 문제를 일으킨다. 본 연구에서는 DiT의 토큰 차원을 유지하면서 다운샘플링과 업샘플링에 데이터 적응형 토큰 병합을 사용하여, DiT의 표현력과 U-Net의 인코딩-디코딩 이점을 결합한 U-Net 확산 트랜스포머 UDT를 제안한다. 기본 UDT 아키텍처는 기존 U-Net DiT를 능가하며 모든 모델 크기에서 REPA에 필적하는 성능을 달성한다. 나아가 아키텍처 최적화와 REPA를 사용하면, 256x256 ImageNet의 XL 모델 크기에서 UDT는 SiT가 1400 에폭에서 달성한 FID 7.9(CFG 미사용)를 40 에폭 만에 능가한다(약 40배 빠른 수렴). 마지막으로 CFG를 사용하여 SD-VAE로 FID 1.38(320 에폭), VA-VAE로 FID 1.35(500 에폭)를 달성하는 강력한 이미지 생성 성능을 보여, 강력한 경험적 이점을 지닌 DiT의 새로운 백본을 제공한다. 코드는 https://github.com/JN-Yun/UDT 에서 확인할 수 있다.

arXiv:2606.26671v1cs.AI

NebulaExp-8B: 전규모 ablation 연구를 통한 경험적 사후 학습 파이프라인

NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research

Qwen3-8B-base 기반의 투명한 ablation 사후 학습 파이프라인 NebulaExp는 instruct와 추론 특화 두 분기에서 384만 SFT 샘플·20만 RL 후보 풀과 응답 증류·필터링·난이도 등급 등 end-to-end 데이터 처리 스택을 공개하고 SFT, GRPO, OPD(MOPD)를 체계적으로 검증한다. Instruct 분기는 평균 벤치마크를 55.01에서 61.85로, Reasoning 분기는 중간 난이도 GRPO로 추론 점수를 73.88에서 75.17로 올리며, 데이터 정확성 필터링이 1차 최적화 요인이고 소량 샘플의 OPD·MOPD가 RL 기준선을 능가할 수 있음을 보인다.

arXiv:2608.27406v1cs.RO

CLAP: 크로스-엠보디먼트 영상 월드 모델은 제로샷 물리 시뮬레이터

CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators

최신 행동 조건(action-conditioned) 영상 모델은 일반적으로 단일 로봇 엠보디먼트(embodiment)에 제한되어, 일반화 가능한 물리 학습을 위한 풍부한 신호를 담고 있는 이질적인 대규모 영상 데이터를 활용하지 못한다. 이 간극을 메우기 위해, 우리는 인간 및 로봇 에이전트에 걸친 다양한 인터넷 규모의 영상으로 학습할 수 있는 크로스-엠보디먼트 행동 조건 영상 생성 프레임워크인 CLAP를 제안한다. CLAP는 행위자와 관계없이 보편적 물리 법칙이 시공간 역학을 지배한다는 통찰에 그라운딩(grounding)되어 있다. 그러나 로봇 플랫폼마다 행동 표현이 크게 달라지고 인간 영상에는 일반적으로 존재하지 않기 때문에, 크로스-엠보디먼트 학습은 쉽지 않다. CLAP는 다음 핵심 기여를 통해 이 근본적 과제를 해결한다. 첫째, CLAP는 엔드 이펙터(end-effector) 자세, 언어 지시, 잠재 행동(latent actions)을 사용하여 서로 다른 행동 공간을 조화시킨다. 둘째, 각 접근의 개별 한계를 해결하기 위해 CLAP는 잠재 행동으로 라벨 없는 영상 데이터 전반에 걸친 기초 물리 사전(prior)을 먼저 학습한 뒤, 이를 엔드 이펙터 행동 공간에 그라운딩하여 실제 과제에 제로샷(zero-shot) 배포할 수 있도록 하는 커리큘럼 기반 크로스-엠보디먼트 학습 방법을 도입한다. 중요하게도, CLAP는 DROID와 같은 도전적인 환경에서 최신 단일-엠보디먼트 영상 모델에 근접하거나 이를 능가한다. 이러한 성능 이점은 퓨샷(few-shot) 적응을 통해 더해져 단일-엠보디먼트 영상 월드 모델 학습을 위한 새로운 패러다임을 확립한다. 궁극적으로 CLAP는 엔드 이펙터, 언어, 잠재 등 다양한 행동 조건 공간과 크로스-엠보디먼트, DROID, Bridge, 양손 YAM 로봇, G1 휴머노이드 등 다양한 로봇 형태(morphology)를 아우르는, 지금까지 가장 포괄적인 행동 조건 영상 월드 모델 모음을 제공한다. 모든 코드와 모델을 오픈소스로 공개한다. 프로젝트 웹사이트: https://omni-clap.github.io .