PyTorchKR Papers

“Code Generation”

총 10,163편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2403.14066공식 코드 · Papers with Code

LeFusion: Controllable Pathology Synthesis via Lesion-Focused Diffusion Models

Patient data from real-world clinical practice often suffers from data scarcity and long-tail imbalances, leading to biased outcomes or algorithmic unfairness. This study addresses these challenges by generating lesion-containing image-segmentation pairs from lesion-free images. Previous efforts in medical imaging synthesis have struggled with separating lesion information from background, resulting in low-quality backgrounds and limited control over the synthetic output. Inspired by diffusion-based image inpainting, we propose LeFusion, a lesion-focused diffusion model. By redesigning the diffusion learning objectives to focus on lesion areas, we simplify the learning process and improve control over the output while preserving high-fidelity backgrounds by integrating forward-diffused background contexts into the reverse diffusion process. Additionally, we tackle two major challenges in lesion texture synthesis: 1) multi-peak and 2) multi-class lesions. We introduce two effective strategies: histogram-based texture control and multi-channel decomposition, enabling the controlled generation of high-quality lesions in difficult scenarios. Furthermore, we incorporate lesion mask diffusion, allowing control over lesion size, location, and boundary, thus increasing lesion diversity. Validated on 3D cardiac lesion MRI and lung nodule CT datasets, LeFusion-generated data significantly improves the performance of state-of-the-art segmentation models, including nnUNet and SwinUNETR. Code and model are available at https://github.com/M3DV/LeFusion.

arXiv:2403.19046공식 코드 · Papers with Code

LITA: Language Instructed Temporal-Localization Assistant

There has been tremendous progress in multimodal Large Language Models (LLMs). Recent works have extended these models to video input with promising instruction following capabilities. However, an important missing piece is temporal localization. These models cannot accurately answer the "When?" questions. We identify three key aspects that limit their temporal localization capabilities: (i) time representation, (ii) architecture, and (iii) data. We address these shortcomings by proposing Language Instructed Temporal-Localization Assistant (LITA) with the following features: (1) We introduce time tokens that encode timestamps relative to the video length to better represent time in videos. (2) We introduce SlowFast tokens in the architecture to capture temporal information at fine temporal resolution. (3) We emphasize temporal localization data for LITA. In addition to leveraging existing video datasets with timestamps, we propose a new task, Reasoning Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for learning and evaluating this task. Reasoning temporal localization requires both the reasoning and temporal localization of Video LLMs. LITA demonstrates strong performance on this challenging task, nearly doubling the temporal mean intersection-over-union (mIoU) of baselines. In addition, we show that our emphasis on temporal localization also substantially improves video-based text generation compared to existing Video LLMs, including a 36% relative improvement of Temporal Understanding. Code is available at: https://github.com/NVlabs/LITA

arXiv:2404.07031공식 코드 · Papers with Code

ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling

Every day, countless surgeries are performed worldwide, each within the distinct settings of operating rooms (ORs) that vary not only in their setups but also in the personnel, tools, and equipment used. This inherent diversity poses a substantial challenge for achieving a holistic understanding of the OR, as it requires models to generalize beyond their initial training datasets. To reduce this gap, we introduce ORacle, an advanced vision-language model designed for holistic OR domain modeling, which incorporates multi-view and temporal capabilities and can leverage external knowledge during inference, enabling it to adapt to previously unseen surgical scenarios. This capability is further enhanced by our novel data augmentation framework, which significantly diversifies the training dataset, ensuring ORacle's proficiency in applying the provided knowledge effectively. In rigorous testing, in scene graph generation, and downstream tasks on the 4D-OR dataset, ORacle not only demonstrates state-of-the-art performance but does so requiring less data than existing models. Furthermore, its adaptability is displayed through its ability to interpret unseen views, actions, and appearances of tools and equipment. This demonstrates ORacle's potential to significantly enhance the scalability and affordability of OR domain modeling and opens a pathway for future advancements in surgical data science. We will release our code and data upon acceptance.

arXiv:2405.17719공식 코드 · Papers with Code

EgoNCE++: Do Egocentric Video-Language Models Really Understand Hand-Object Interactions?

Egocentric video-language pretraining is a crucial paradigm to advance the learning of egocentric hand-object interactions (EgoHOI). Despite the great success on existing testbeds, these benchmarks focus more on closed-set visual concepts or limited scenarios. Due to the occurrence of diverse EgoHOIs in the real world, we propose an open-vocabulary benchmark named EgoHOIBench to reveal the diminished performance of current egocentric video-language models (EgoVLM) on fined-grained concepts, indicating that these models still lack a full spectrum of egocentric understanding. We attribute this performance gap to insufficient fine-grained supervision and strong bias towards understanding objects rather than temporal dynamics in current methods. To tackle these issues, we introduce a novel asymmetric contrastive objective for EgoHOI named EgoNCE++. For video-to-text loss, we enhance text supervision through the generation of negative captions by leveraging the in-context learning of large language models to perform HOI-related word substitution. For text-to-video loss, we propose an object-centric positive video sampling strategy that aggregates video representations by the same nouns. Our extensive experiments demonstrate that EgoNCE++ significantly boosts open-vocabulary HOI recognition, multi-instance retrieval, and action recognition tasks across various egocentric models, with improvements of up to +26.55%. Our code is available at https://github.com/xuboshen/EgoNCEpp.

arXiv:2406.11816

VideoLLM-online: Online Video Large Language Model for Streaming Video

Recent Large Language Models have been enhanced with vision capabilities, enabling them to comprehend images, videos, and interleaved vision-language content. However, the learning methods of these large multimodal models typically treat videos as predetermined clips, making them less effective and efficient at handling streaming video inputs. In this paper, we propose a novel Learning-In-Video-Stream (LIVE) framework, which enables temporally aligned, long-context, and real-time conversation within a continuous video stream. Our LIVE framework comprises comprehensive approaches to achieve video streaming dialogue, encompassing: (1) a training objective designed to perform language modeling for continuous streaming inputs, (2) a data generation scheme that converts offline temporal annotations into a streaming dialogue format, and (3) an optimized inference pipeline to speed up the model responses in real-world video streams. With our LIVE framework, we built VideoLLM-online model upon Llama-2/Llama-3 and demonstrate its significant advantages in processing streaming videos. For instance, on average, our model can support streaming dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU. Moreover, it also showcases state-of-the-art performance on public offline video benchmarks, such as recognition, captioning, and forecasting. The code, model, data, and demo have been made available at https://showlab.github.io/videollm-online.

arXiv:2409.13431공식 코드 · Papers with Code

Leveraging Text Localization for Scene Text Removal via Text-aware Masked Image Modeling

Existing scene text removal (STR) task suffers from insufficient training data due to the expensive pixel-level labeling. In this paper, we aim to address this issue by introducing a Text-aware Masked Image Modeling algorithm (TMIM), which can pretrain STR models with low-cost text detection labels (e.g., text bounding box). Different from previous pretraining methods that use indirect auxiliary tasks only to enhance the implicit feature extraction ability, our TMIM first enables the STR task to be directly trained in a weakly supervised manner, which explores the STR knowledge explicitly and efficiently. In TMIM, first, a Background Modeling stream is built to learn background generation rules by recovering the masked non-text region. Meanwhile, it provides pseudo STR labels on the masked text region. Second, a Text Erasing stream is proposed to learn from the pseudo labels and equip the model with end-to-end STR ability. Benefiting from the two collaborative streams, our STR model can achieve impressive performance only with the public text detection datasets, which greatly alleviates the limitation of the high-cost STR labels. Experiments demonstrate that our method outperforms other pretrain methods and achieves state-of-the-art performance (37.35 PSNR on SCUT-EnsText). Code will be available at https://github.com/wzx99/TMIM.

arXiv:2411.02083공식 코드 · Papers with Code

Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models

While language models have exceptional capabilities at text generation, they lack a natural inductive bias for emitting numbers and thus struggle in tasks involving quantitative reasoning, especially arithmetic. One fundamental limitation is the nature of the Cross Entropy loss, which assumes a nominal scale and thus cannot convey proximity between generated number tokens. In response, we here present a regression-like loss that operates purely on token level. Our proposed Number Token Loss (NTL) comes in two flavors and minimizes either the Lp norm or the Wasserstein distance between the numerical values of the real and predicted number tokens. NTL can easily be added to any language model and extend the Cross Entropy objective during training without runtime overhead. We evaluate the proposed scheme on various mathematical datasets and find that it consistently improves performance in math-related tasks. In a direct comparison on a regression task, we find that NTL can match the performance of a regression head, despite operating on token level. Finally, we scale NTL up to 3B parameter models and observe improved performance, demonstrating its potential for seamless integration into LLMs. We hope that this work can inspire LLM developers to improve their pretraining objectives. The code is available via: https://tum-ai.github.io/number-token-loss/

arXiv:2411.00626공식 코드 · Papers with Code

ZIM: Zero-Shot Image Matting for Anything

The recent segmentation foundation model, Segment Anything Model (SAM), exhibits strong zero-shot segmentation capabilities, but it falls short in generating fine-grained precise masks. To address this limitation, we propose a novel zero-shot image matting model, called ZIM, with two key contributions: First, we develop a label converter that transforms segmentation labels into detailed matte labels, constructing the new SA1B-Matte dataset without costly manual annotations. Training SAM with this dataset enables it to generate precise matte masks while maintaining its zero-shot capability. Second, we design the zero-shot matting model equipped with a hierarchical pixel decoder to enhance mask representation, along with a prompt-aware masked attention mechanism to improve performance by enabling the model to focus on regions specified by visual prompts. We evaluate ZIM using the newly introduced MicroMat-3K test set, which contains high-quality micro-level matte labels. Experimental results show that ZIM outperforms existing methods in fine-grained mask generation and zero-shot generalization. Furthermore, we demonstrate the versatility of ZIM in various downstream tasks requiring precise masks, such as image inpainting and 3D NeRF. Our contributions provide a robust foundation for advancing zero-shot matting and its downstream applications across a wide range of computer vision tasks. The code is available at \url{https://github.com/naver-ai/ZIM}.

arXiv:2501.04582

Boosting Salient Object Detection with Knowledge Distillated from Large Foundation Models

Salient Object Detection (SOD) aims to identify and segment prominent regions within a scene. Traditional models rely on manually annotated pseudo labels with precise pixel-level accuracy, which is time-consuming. We developed a low-cost, high-precision annotation method by leveraging large foundation models to address the challenges. Specifically, we use a weakly supervised approach to guide large models in generating pseudo-labels through textual prompts. Since large models do not effectively focus on the salient regions of images, we manually annotate a subset of text to fine-tune the model. Based on this approach, which enables precise and rapid generation of pseudo-labels, we introduce a new dataset, BDS-TR. Compared to the previous DUTS-TR dataset, BDS-TR is more prominent in scale and encompasses a wider variety of categories and scenes. This expansion will enhance our model's applicability across a broader range of scenarios and provide a more comprehensive foundational dataset for future SOD research. Additionally, we present an edge decoder based on dynamic upsampling, which focuses on object edges while gradually recovering image feature resolution. Comprehensive experiments on five benchmark datasets demonstrate that our method significantly outperforms state-of-the-art approaches and also surpasses several existing fully-supervised SOD methods. The code and results will be made available.

arXiv:2503.19730공식 코드 · Papers with Code

CamSAM2: Segment Anything Accurately in Camouflaged Videos

Video camouflaged object segmentation (VCOS), aiming at segmenting camouflaged objects that seamlessly blend into their environment, is a fundamental vision task with various real-world applications. With the release of SAM2, video segmentation has witnessed significant progress. However, SAM2's capability of segmenting camouflaged videos is suboptimal, especially when given simple prompts such as point and box. To address the problem, we propose Camouflaged SAM2 (CamSAM2), which enhances SAM2's ability to handle camouflaged scenes without modifying SAM2's parameters. Specifically, we introduce a decamouflaged token to provide the flexibility of feature adjustment for VCOS. To make full use of fine-grained and high-resolution features from the current frame and previous frames, we propose implicit object-aware fusion (IOF) and explicit object-aware fusion (EOF) modules, respectively. Object prototype generation (OPG) is introduced to abstract and memorize object prototypes with informative details using high-quality features from previous frames. Extensive experiments are conducted to validate the effectiveness of our approach. While CamSAM2 only adds negligible learnable parameters to SAM2, it substantially outperforms SAM2 on three VCOS datasets, especially achieving 12.2 mDice gains with click prompt on MoCA-Mask and 19.6 mDice gains with mask prompt on SUN-SEG-Hard, with Hiera-T as the backbone. The code will be available at https://github.com/zhoustan/CamSAM2.

arXiv:2505.23353공식 코드 · Papers with Code

Synthetic Generation and Latent Projection Denoising of Rim Lesions in Multiple Sclerosis

Quantitative susceptibility maps from magnetic resonance images can provide both prognostic and diagnostic information in multiple sclerosis, a neurodegenerative disease characterized by the formation of lesions in white matter brain tissue. In particular, susceptibility maps provide adequate contrast to distinguish between "rim" lesions, surrounded by deposited paramagnetic iron, and "non-rim" lesion types. These paramagnetic rim lesions (PRLs) are an emerging biomarker in multiple sclerosis. Much effort has been devoted to both detection and segmentation of such lesions to monitor longitudinal change. As paramagnetic rim lesions are rare, addressing this problem requires confronting the class imbalance between rim and non-rim lesions. We produce synthetic quantitative susceptibility maps of paramagnetic rim lesions and show that inclusion of such synthetic data improves classifier performance and provide a multi-channel extension to generate accompanying contrasts and probabilistic segmentation maps. We exploit the projection capability of our trained generative network to demonstrate a novel denoising approach that allows us to train on ambiguous rim cases and substantially increase the minority class. We show that both synthetic lesion synthesis and our proposed rim lesion label denoising method best approximate the unseen rim lesion distribution and improve detection in a clinically interpretable manner. We release our code and generated data at https://github.com/agr78/PRLx-GAN upon publication.

arXiv:2506.14123공식 코드 · Papers with Code

Sampling from Your Language Model One Byte at a Time

Tokenization is used almost universally by modern language models, enabling efficient text representation using multi-byte or multi-character tokens. However, prior work has shown that tokenization can introduce distortion into the model's generations. For example, users are often advised not to end their prompts with a space because it prevents the model from including the space as part of the next token. This Prompt Boundary Problem (PBP) also arises in languages such as Chinese and in code generation, where tokens often do not line up with syntactic boundaries. Additionally mismatching tokenizers often hinder model composition and interoperability. For example, it is not possible to directly ensemble models with different tokenizers due to their mismatching vocabularies. To address these issues, we present an inference-time method to convert any autoregressive LM with a BPE tokenizer into a character-level or byte-level LM, without changing its generative distribution at the text level. Our method efficient solves the PBP and is also able to unify the vocabularies of language models with different tokenizers, allowing one to ensemble LMs with different tokenizers at inference time as well as transfer the post-training from one model to another using proxy-tuning. We demonstrate in experiments that the ensemble and proxy-tuned models outperform their constituents on downstream evals.

arXiv:2507.01367

3D Gaussian Splatting Driven Multi-View Robust Physical Adversarial Camouflage Generation

Physical adversarial attack methods expose the vulnerabilities of deep neural networks and pose a significant threat to safety-critical scenarios such as autonomous driving. Camouflage-based physical attack is a more promising approach compared to the patch-based attack, offering stronger adversarial effectiveness in complex physical environments. However, most prior work relies on mesh priors of the target object and virtual environments constructed by simulators, which are time-consuming to obtain and inevitably differ from the real world. Moreover, due to the limitations of the backgrounds in training images, previous methods often fail to produce multi-view robust adversarial camouflage and tend to fall into sub-optimal solutions. Due to these reasons, prior work lacks adversarial effectiveness and robustness across diverse viewpoints and physical environments. We propose a physical attack framework based on 3D Gaussian Splatting (3DGS), named PGA, which provides rapid and precise reconstruction with few images, along with photo-realistic rendering capabilities. Our framework further enhances cross-view robustness and adversarial effectiveness by preventing mutual and self-occlusion among Gaussians and employing a min-max optimization approach that adjusts the imaging background of each viewpoint, helping the algorithm filter out non-robust adversarial features. Extensive experiments validate the effectiveness and superiority of PGA. Our code is available at:https://github.com/TRLou/PGA.

arXiv:2609.15545v3cs.LG

The Token Before the Value Is the Key

Induction circuits retrieve a value by matching a query to information carried from the token before that value. How this address forms, incorporates context, and supports sequential retrieval remains less clear. We use hybrid language models as a tractable testbed and identify the historical source key as a common causal interface between local preparation and global retrieval. Probes and restoration interventions connect this interface across controlled models, pretrained recurrent and sliding-window hybrids, and released language models. We organize the analysis by address context radius and sequential retrieval depth. First, shared local parameters couple predecessor Carrying to recurrent state processing; separating these demands or biasing initialization promotes address formation. Second, adapting input representations extends an established literal circuit to category matching, making semantically compatible records active competitors. Earlier context resolves this competition through the source key, with a sharp visibility boundary in a controlled single-read model and compositional relays across local layers. Third, fixed relations support memorization shortcuts, whereas sequential generalization requires learning an output-to-query bridge between retrievals. Controlled state-tracking and compositional in-context tasks demonstrate this requirement across stacked stages, looped recurrence, and intermediate-token generation. Together, these results connect circuit formation, contextual disambiguation, and sequential use through a shared retrieval interface. Code is available at https://github.com/ckpassenger/bind-match-copy.

arXiv:2606.11543v1cs.AI

SkillJuror: 에이전트 스킬 구성이 런타임 행동에 미치는 영향 측정

SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior

LLM 에이전트의 스킬(Skill) 조직 방식이 런타임 행동에 미치는 영향을 측정하는 SkillJuror 프레임워크를 제안하며, 점진적 공개(Progressive Disclosure) 방식이 평탄한 기준선 대비 리소스 활용을 증가시키고 성능을 향상시킴을 입증합니다.

arXiv:2606.21678v1cs.LG

디코드 가능하지만 충실하지 않음: 자연어 근거와 프로그램적 검증기의 결합

Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers

언어 모델은 예측에 대해 그럴듯한 근거(rationale)를 생성할 수 있지만, 이러한 설명이 모델의 내부 추론을 충실하게 반영하지 않을 수 있다. 우리는 추론 trace에 인라인 클레임을 삽입하고, 보조 일관성 헤드를 학습하여 근거 구간 은닉 상태로부터 프로그램적 검증기 출력을 예측하는 검증기 결합 추론(verifier-coupled reasoning) 프레임워크를 제안한다. 핵심 발견은 디코드 가능성과 충실성 사이의 격차이다: 일관성 학습은 근거 표현으로부터 검증기 정보를 안정적으로 디코드 가능하게 만들지만, 디코드 가능성이 충실한 생성을 보장하지는 않는다. LeanCheck(형식 정리 증명)에서는 근거 전용 및 증명 전용 풀링이 반사실적 충돌 하에서 완벽한 방향 분리를 달성한다. KataGo(바둑 엔진)에서는 해설 구간이 10분류 승률 버킷을 81% 정확도로 인코딩한다. 그러나 코드 환경에서는 모델이 98.6% 결합을 달성하면서도 생성된 설명은 비충실하다: 유창한 산문과 올바른 구조화된 클레임을 갖추었지만, 무관한 알고리즘을 설명한다. 통제된 사전학습 대 처음부터 비교는 이 격차가 용량에 의해 주도되지 않음을 보여준다. 합성 활성화 패칭은 인과적 영향(기준선 31% 대비 73–89%)을 확인하고, FEVER는 증거 전용 풀링이 원시 정확도를 희생하는 대가로 진정한 증거 민감도를 분리함을 밝힌다. 주장별 분석은 일관성 손실이 이진적 클레임보다 세분화된 클레임에 불균형적으로 이득을 준다는 것을 보여준다. 이러한 결과는 일관성 손실이 효과적인 진단 및 표현 형성 도구이지만, 충실한 추론을 위한 충분조건은 아님을 확립한다.

arXiv:2606.28092v1cs.CV

디노이저 응답의 스펙트럼 결합을 통한 확산 모델 귀속

Diffusion Model Attribution via Spectral Coupling of Denoiser Responses

생성 이미지를 그 원본 확산 모델에 귀속하는 것은 출처 검증과 지적재산권 보호에서 근본적인 과제이다. 이 문제는 특히 어려운데, 서로 다른 데이터셋으로 학습된 확산 모델이 유사한 score function에 수렴하여 결과적으로 유사한 출력 분포를 갖게 될 수 있기 때문이며, 그로 인해 생성된 이미지 자체는 귀속 근거로 신뢰할 수 없다. 기존 비침습적(non-invasive) 방법은 구조적으로 유사한 변형에서는 실패하거나, 모델이 동일한 autoencoder를 공유하면 사라지는 신호에 의존한다. 우리는 각 후보 모델의 디노이징(denoising) 행동을 지문화(fingerprinting)하여 원본 모델을 식별하는 비침습적 귀속 방법인 Spectral Denoising Signatures(SDS)를 제안한다. 핵심 통찰은 모델의 디노이징 score function이 독특한 스펙트럼 기하(spectral geometry)를 나타내며, 이는 디노이징 동안 공간 주파수 대역(spatial frequency bands) 전반에 에너지를 재분배하는 방식에 반영된다는 것이다. 주파수 제어 섭동(frequency-controlled perturbations)으로 이 행동을 탐색함으로써, SDS는 모델에 내재(intrinsic)된 안정적인 시그니처(signature)를 추출하며, 역변환(inversion), 최적화(optimization), 또는 생성 시점 등록(generation-time enrollment) 없이 표준 순전파(forward pass)만으로 수행된다. 실험 결과 SDS는 8개의 다양한 확산 모델에서 약 99.9%의 정확도를, 크로스 도메인 프롬프트 전환(cross-domain prompt shift)에서는 96.2%를 달성했으며, 학습 데이터, 아키텍처, 학습 절차의 변화에 걸친 비침습적 기준선(baseline)들을 능가하여, 스펙트럼 기하를 확산 모델 귀속을 위한 원칙적이고 실용적인 토대로 확립한다. 코드는 https://github.com/Pragati-Meshram/SGS 에서 제공된다.

arXiv:2602.23297v3cs.CV

PRIMA: LLM 기반 특징 집약을 통한 의학 진단을 위한 위험 통합 이미지-메타데이터 정렬 사전학습

PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation

의학 진단은 시각적 징후와 임상 메타데이터의 효과적인 종합을 필요로 합니다. 그러나 기존 방법들은 종종 메타데이터를 고립된 태그로 취급하여 임상 설명에 내재된 풍부한 의미론적 지식을 활용하지 못합니다. 본 논문에서는 도메인 특화 지식을 다중 모달(multi-modal) 표현 학습에 통합하는 프레임워크인 PRIMA(Pre-training with Risk-integrated Image-Metadata Alignment)를 제안합니다. 먼저 검색-증강 생성(RAG)을 통해 위험-질병 상관관계 코퍼스를 구축하여 Clinical ModernBERT를 개선하고, 텍스트 인코더에 진단 사전 정보를 임베딩합니다. 모달리티 격차(modality gap)를 해소하기 위해 DINOv3와 개선된 Clinical ModernBERT를 활용하는 이중 인코더 사전학습 전략을 도입하며, 이는 상호보완적인 4가지 손실 함수 모음으로 최적화됩니다. 이러한 손실 함수들은 다중 입도(multi-granular) 의미론적 정렬을 포착하고 소프트 라벨(soft label)을 통해 임상 상관관계의 모호성을 처리하도록 설계되었습니다. 마지막으로 Qwen3를 활용하여 정렬된 특징들을 융합함으로써 정밀한 질병 분류를 수행합니다. 광범위한 실험을 통해 PRIMA가 픽셀 수준의 특징과 추상적인 임상 전문 지식을 효과적으로 조화시킴을 입증합니다. PAD-UFES-20 및 AQUA 벤치마크 전반에서 PRIMA는 각각 72.04% 및 85.22%의 평균 F1 점수를 달성하여 강력한 이미지 전용, 메타데이터 융합, 의료 비전-언어 베이스라인보다 뛰어난 성능을 보였습니다. 특히 본 프레임워크는 대규모 데이터 수집이나 과도한 계산 자원 없이도 강력한 성능을 달성합니다. 코드는 https://github.com/yqwang01/PRIMA 에서 확인할 수 있습니다.

arXiv:2608.03447v2cs.LG

근사 추측 디코딩

Approximate Speculative Decoding

추측 디코딩(Speculative decoding)은 타깃 모델을 사용해 초안 블록을 병렬로 검증함으로써 자기회귀적 생성을 가속화합니다. 표준 탐욕적 검증(greedy verification)에서는 디코딩이 타깃 argmax와 달라지는 첫 번째 초안 토큰에서 중단되며, 타깃 점수가 매겨진 나머지 접미사는 폐기됩니다. 이러한 불일치를 수용하면 디코딩 궤적이 변경되지만, 실현된 접두 문맥 하에서 해당 토큰들이 타깃 탐욕적 상태를 유지할 때 연속적인 접미사를 재사용할 수 있게 됩니다. 본 논문에서는 이진 첫 번째 불일치 절단(truncation) 방식을 예산 기반 최장 접두 문맥 선택으로 대체하는 학습이 필요 없는 검증기인 근사 추측 디코딩(Approximate Speculative Decoding, ASD)을 소개합니다. ASD는 로컬 타깃 로짓 후회(regret) 게이트, 블록당 예외 상한선, 영구적인 요청 수준의 후회 예산에 따라 선택된 불일치를 수용한 뒤, 추가적인 근사 결정이나 타깃 모델의 순전파 연산 없이 연속적인 타깃 탐욕 접미사를 재사용합니다. ASD는 새로운 초안 모델이나 미세조정을 필요로 하지 않으며, 예산이 0일 때는 표준 탐욕 검증으로 정확히 환원됩니다. 실험 결과, ASD는 동일 조건의 엄격한 검증 대비 고정 워크로드 처리량을 3.05%~15.26% 향상시키며, 7개의 Qwen3-14B + DSpark-14B 태스크 전반에서 평균 7.78%의 이득을 달성합니다. 또한 DeepSeek-V4-Flash(284B)와 DSpark를 결합한 FP4-to-FP8 호환성 설정에서도 GSM8K 및 MATH-500 태스크에서 검증기 측 수락률을 대략 10%~16% 높입니다. 소스 코드는 https://github.com/Kissmetothemoon/ASD 에서 공개되어 있습니다.

arXiv:2608.18931v1cs.CL

야생에서의 테스트 타임 스케일링: 왜 탐험이 아닌 활용이 병목인가

Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck

테스트 타임 스케일링(TTS)은 여러 후보 생성, 부분 시퀀스 탐색, 초안의 반복적 개선 등 추가적인 추론(inference) 연산을 투입하여 언어 모델(language model)의 출력을 향상합니다. 이러한 기법들은 수학 및 코드 영역에서 큰 성능 향상을 가져왔지만, 검증이 직관적인 태스크를 중심으로 개발되고 스트레스 테스트를 거쳤습니다. 본 연구에서는 의학, 법률, 금융, 일반 대화, 창의적 글쓰기에 걸친 5개의 개방형 생성 벤치마크 전반에서 5가지 TTS 계열 기법을 연산 정규화(compute-normalised)하여 최초로 비교 분석합니다. 이는 각 기법의 토큰(token) 예산 효율성을 탐험(exploration)과 활용(exploitation)으로 분해하는 통합 프레임워크를 기반으로 합니다. 그 결과는 이러한 분해의 어느 측면을 살펴보느냐에 따라 달라집니다. 탐험을 스케일링하는 것은 효과적입니다. 풀(pool) 내 최적 후보의 품질은 모든 환경에서 연산량에 따라 꾸준히 향상됩니다. 문제가 발생하는 지점은 풍부한 후보 풀을 최종 출력으로 변환하는 단계인 활용입니다. 최첨단 생성 모델을 사용할 때 보상 모델(reward model)과 실제 품질 간의 상관관계는 $ρ_v \approx 0.12$에 불과하여, 예산과 무관하게 선택이 거의 무작위에 가깝게 이루어집니다. 트리 탐색은 다양성 붕괴(diversity collapse)를 통해 이러한 실패를 증폭합니다. 반복 개선은 5개 벤치마크 중 단 하나에서만 도움이 되었으며, 다른 곳에서 나타난 외견상의 이득은 교란된 결과였습니다. 여러 후보 간의 종합(Fusion)만이 단일 샘플 베이스라인 대비 일관되게 향상되었으나, 여전히 활용 가능한 품질의 약 40%만을 회복하는 데 그쳤습니다. 병목은 후보 풀이 아니라, 그 안에서 선택하는 과정에 있습니다.