PyTorchKR Papers

“Code Generation”

총 10,163편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2412.02780공식 코드 · Papers with Code

WxC-Bench: A Novel Dataset for Weather and Climate Downstream Tasks

High-quality machine learning (ML)-ready datasets play a foundational role in developing new artificial intelligence (AI) models or fine-tuning existing models for scientific applications such as weather and climate analysis. Unfortunately, despite the growing development of new deep learning models for weather and climate, there is a scarcity of curated, pre-processed machine learning (ML)-ready datasets. Curating such high-quality datasets for developing new models is challenging particularly because the modality of the input data varies significantly for different downstream tasks addressing different atmospheric scales (spatial and temporal). Here we introduce WxC-Bench (Weather and Climate Bench), a multi-modal dataset designed to support the development of generalizable AI models for downstream use-cases in weather and climate research. WxC-Bench is designed as a dataset of datasets for developing ML-models for a complex weather and climate system, addressing selected downstream tasks as machine learning phenomenon. WxC-Bench encompasses several atmospheric processes from meso-$\beta$ (20 - 200 km) scale to synoptic scales (2500 km), such as aviation turbulence, hurricane intensity and track monitoring, weather analog search, gravity wave parameterization, and natural language report generation. We provide a comprehensive description of the dataset and also present a technical validation for baseline analysis. The dataset and code to prepare the ML-ready data have been made publicly available on Hugging Face -- https://huggingface.co/datasets/nasa-impact/WxC-Bench

arXiv:2412.18404공식 코드 · Papers with Code

Extract Free Dense Misalignment from CLIP

Recent vision-language foundation models still frequently produce outputs misaligned with their inputs, evidenced by object hallucination in captioning and prompt misalignment in the text-to-image generation model. Recent studies have explored methods for identifying misaligned elements, aiming not only to enhance interpretability but also to improve model performance. However, current approaches primarily rely on large foundation models in a zero-shot manner or fine-tuned models with human annotations, which limits scalability due to significant computational costs. This work proposes a novel approach, dubbed CLIP4DM, for detecting dense misalignments from pre-trained CLIP, specifically focusing on pinpointing misaligned words between image and text. We carefully revamp the gradient-based attribution computation method, enabling negative gradient of individual text tokens to indicate misalignment. We also propose F-CLIPScore, which aggregates misaligned attributions with a global alignment score. We evaluate our method on various dense misalignment detection benchmarks, covering various image and text domains and misalignment types. Our method demonstrates state-of-the-art performance among zero-shot models and competitive performance with fine-tuned models while maintaining superior efficiency. Our qualitative examples show that our method has a unique strength to detect entity-level objects, intangible objects, and attributes that can not be easily detected for existing works. We conduct ablation studies and analyses to highlight the strengths and limitations of our approach. Our code is publicly available at https://github.com/naver-ai/CLIP4DM.

arXiv:2503.18159공식 코드 · Papers with Code

DiffusionTalker: Efficient and Compact Speech-Driven 3D Talking Head via Personalizer-Guided Distillation

Real-time speech-driven 3D facial animation has been attractive in academia and industry. Traditional methods mainly focus on learning a deterministic mapping from speech to animation. Recent approaches start to consider the nondeterministic fact of speech-driven 3D face animation and employ the diffusion model for the task. Existing diffusion-based methods can improve the diversity of facial animation. However, personalized speaking styles conveying accurate lip language is still lacking, besides, efficiency and compactness still need to be improved. In this work, we propose DiffusionTalker to address the above limitations via personalizer-guided distillation. In terms of personalization, we introduce a contrastive personalizer that learns identity and emotion embeddings to capture speaking styles from audio. We further propose a personalizer enhancer during distillation to enhance the influence of embeddings on facial animation. For efficiency, we use iterative distillation to reduce the steps required for animation generation and achieve more than 8x speedup in inference. To achieve compactness, we distill the large teacher model into a smaller student model, reducing our model's storage by 86.4\% while minimizing performance loss. After distillation, users can derive their identity and emotion embeddings from audio to quickly create personalized animations that reflect specific speaking styles. Extensive experiments are conducted to demonstrate that our method outperforms state-of-the-art methods. The code will be released at: https://github.com/ChenVoid/DiffusionTalker.

arXiv:2503.17109공식 코드 · Papers with Code

Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval

Zero-Shot Composed Image Retrieval (ZS-CIR) involves diverse tasks with a broad range of visual content manipulation intent across domain, scene, object, and attribute. The key challenge for ZS-CIR tasks is to modify a reference image according to manipulation text to accurately retrieve a target image, especially when the reference image is missing essential target content. In this paper, we propose a novel prediction-based mapping network, named PrediCIR, to adaptively predict the missing target visual content in reference images in the latent space before mapping for accurate ZS-CIR. Specifically, a world view generation module first constructs a source view by omitting certain visual content of a target view, coupled with an action that includes the manipulation intent derived from existing image-caption pairs. Then, a target content prediction module trains a world model as a predictor to adaptively predict the missing visual information guided by user intention in manipulating text at the latent space. The two modules map an image with the predicted relevant information to a pseudo-word token without extra supervision. Our model shows strong generalization ability on six ZS-CIR tasks. It obtains consistent and significant performance boosts ranging from 1.73% to 4.45% over the best methods and achieves new state-of-the-art results on ZS-CIR. Our code is available at https://github.com/Pter61/predicir.

arXiv:2504.04178공식 코드 · Papers with Code

MSL: Not All Tokens Are What You Need for Tuning LLM as a Recommender

Large language models (LLMs), known for their comprehension capabilities and extensive knowledge, have been increasingly applied to recommendation systems (RS). Given the fundamental gap between the mechanism of LLMs and the requirement of RS, researchers have focused on fine-tuning LLMs with recommendation-specific data to enhance their performance. Language Modeling Loss (LML), originally designed for language generation tasks, is commonly adopted. However, we identify two critical limitations of LML: 1) it exhibits significant divergence from the recommendation objective; 2) it erroneously treats all fictitious item descriptions as negative samples, introducing misleading training signals. To address these limitations, we propose a novel Masked Softmax Loss (MSL) tailored for fine-tuning LLMs on recommendation. MSL improves LML by identifying and masking invalid tokens that could lead to fictitious item descriptions during loss computation. This strategy can effectively avoid the interference from erroneous negative signals and ensure well alignment with the recommendation objective supported by theoretical guarantees. During implementation, we identify a potential challenge related to gradient vanishing of MSL. To overcome this, we further introduce the temperature coefficient and propose an Adaptive Temperature Strategy (ATS) that adaptively adjusts the temperature without requiring extensive hyperparameter tuning. Extensive experiments conducted on four public datasets further validate the effectiveness of MSL, achieving an average improvement of 42.24% in NDCG@10. The code is available at https://github.com/WANGBohaO-jpg/MSL.

arXiv:2505.04196

A Large Language Model for Feasible and Diverse Population Synthesis

Generating a synthetic population that is both feasible and diverse is crucial for ensuring the validity of downstream activity schedule simulation in activity-based models (ABMs). While deep generative models (DGMs), such as variational autoencoders and generative adversarial networks, have been applied to this task, they often struggle to balance the inclusion of rare but plausible combinations (i.e., sampling zeros) with the exclusion of implausible ones (i.e., structural zeros). To improve feasibility while maintaining diversity, we propose a fine-tuning method for large language models (LLMs) that explicitly controls the autoregressive generation process through topological orderings derived from a Bayesian Network (BN). Experimental results show that our hybrid LLM-BN approach outperforms both traditional DGMs and proprietary LLMs (e.g., ChatGPT-4o) with few-shot learning. Specifically, our approach achieves approximately 95% feasibility, significantly higher than the ~80% observed in DGMs, while maintaining comparable diversity, making it well-suited for practical applications. Importantly, the method is based on a lightweight open-source LLM, enabling fine-tuning and inference on standard personal computing environments. This makes the approach cost-effective and scalable for large-scale applications, such as synthesizing populations in megacities, without relying on expensive infrastructure. By initiating the ABM pipeline with high-quality synthetic populations, our method improves overall simulation reliability and reduces downstream error propagation. The source code for these methods is available for research and practical application.

arXiv:2506.23309공식 코드 · Papers with Code

SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splatting

In contemporary surgical research and practice, accurately comprehending 3D surgical scenes with text-promptable capabilities is particularly crucial for surgical planning and real-time intra-operative guidance, where precisely identifying and interacting with surgical tools and anatomical structures is paramount. However, existing works focus on surgical vision-language model (VLM), 3D reconstruction, and segmentation separately, lacking support for real-time text-promptable 3D queries. In this paper, we present SurgTPGS, a novel text-promptable Gaussian Splatting method to fill this gap. We introduce a 3D semantics feature learning strategy incorporating the Segment Anything model and state-of-the-art vision-language models. We extract the segmented language features for 3D surgical scene reconstruction, enabling a more in-depth understanding of the complex surgical environment. We also propose semantic-aware deformation tracking to capture the seamless deformation of semantic features, providing a more precise reconstruction for both texture and semantic features. Furthermore, we present semantic region-aware optimization, which utilizes regional-based semantic information to supervise the training, particularly promoting the reconstruction quality and semantic smoothness. We conduct comprehensive experiments on two real-world surgical datasets to demonstrate the superiority of SurgTPGS over state-of-the-art methods, highlighting its potential to revolutionize surgical practices. SurgTPGS paves the way for developing next-generation intelligent surgical systems by enhancing surgical precision and safety. Our code is available at: https://github.com/lastbasket/SurgTPGS.

arXiv:2607.14962v3cs.LG

텍스트-이미지 생성에서 대표 다양성을 위한 다축 Max@K 강화학습

Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation

Text-to-image (T2I) models produce realistic, prompt-aligned images, but samples for the same prompt often cover only a few visually distinct modes. For person-centric prompts, this lack of diversity can reflect or amplify demographic skew. We formalize representative diversity as target-mode coverage: how well a set of samples covers predefined modes, such as colors or perceived-race categories. We then propose MultiMaxPO, a reinforcement learning post-training method that improves this coverage in diffusion T2I models by optimizing a multi-axis max@K objective. Given one score per mode for each sample, multi-axis max@K takes the maximum over samples for each mode and then sums these maxima. A sample therefore receives credit for a mode only when it raises that mode's maximum, so different samples are rewarded for different modes. We verify this credit assignment on a synthetic Gaussian mixture and on Stable Diffusion 3.5 Medium with pixel-based color rewards, and then apply it to perceived-appearance fairness. On held-out prompts and under three automatic evaluators, MultiMaxPO improves the Fairness Score over the base model by 0.22-0.36 on Stable Diffusion 3.5 Medium and by 0.07-0.19 on FLUX.1-dev, outperforms matched RL baselines, and maintains image quality and text alignment. Code is available at https://github.com/KuOnoda/multi-axis-maxk.

arXiv:2607.03236v2cs.CL

TACG: 확산 언어 모델 디코딩을 위한 궤적 인식 커밋 게이팅

TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding

Diffusion language models predict many masked positions at each denoising step, but their efficiency and output quality depend on which predictions enter the evolving context. Confidence is informative but incomplete for this decision: at matched confidence, proposals with different identity and support histories exhibit different one-step revision rates. We propose Trajectory-Aware Commit Gating (TACG), a training-free decoder that separates the current token proposal from the decision to commit it. TACG scores temporal support for the current proposal by contrasting its logits with an exponential moving average of earlier logits, requires the proposal identity to persist through a History Gate, and promotes a capped number of additional positions per step. The gate decides which positions to commit and when; every written token remains the contemporaneous base proposal. On complete four-task LLaDA evaluations under the stated single-seed protocol, measured task-specific operating points reach 51.02\% mean accuracy with 67.82 mean model evaluations, compared with 48.00\% and 76.59 for Native Confidence. Under a matched per-event commit-count replay, history-guided selection exceeds confidence ranking by 8.3 accuracy points at identical per-sample NFE. History-reference interventions alter commitment timing while preserving the base-proposal write rule. Experiments on Dream and LLaDA2-Mini characterize quality--cost behavior across backbones and generation lengths. The code is publicly available at https://github.com/Clarence-CV/TACG-DLLM.

arXiv:2605.17609v3cs.LG

Adaptive Generate-Rank-Verify: Inference-Time Search with Costly Verification

Many inference-time language-model pipelines combine a cheap reward signal with an expensive verifier, such as exact answer checking in mathematical reasoning or hidden-test execution in code generation. We formalize this setting using a learning-theoretic lens as generative active search: a cost-sensitive first-positive search problem in which a policy adaptively samples candidates from an unknown distribution, observes cheap scores, and pays for verifier labels until it finds a positive example. For a fixed prompt, the generator and reward model induce two unknown objects: a distribution over reward scores and a score-conditioned success function. When these quantities are known, we characterize the distribution-aware optimal policy using a dynamic programming approach. In the realistic and practical setting where both the score distribution and success function are unknown, we propose ADAP, a shellwise adaptive generate-rank-verify algorithm that progressively increases the number of sampled responses and top-ranked verifications. Under the monotonicity assumption that higher reward scores are no less likely to pass verification, we show that ADAP achieves expected cost within a constant factor of the distribution-aware optimum. We complement this result with learning-theoretic lower bounds, based on a centered star number, showing that structural assumptions on the score--label relationship are necessary. Experiments on mathematical reasoning and competitive programming validate the predicted advantage over both fixed non-adaptive policies and difficulty-adaptive baselines.

arXiv:2606.24087v1cs.LG

NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction

두피 EEG(전기뇌파)에서 연속 음성을 복원할 때 약하고 가변적인 신경 신호와 음성의 시간·하모닉 구조 불일치를 다루기 위해, EEG 조건 하에 노이즈가 섞인 음향 상태를 깨끗한 음성으로 운반하는 결정적 확률 흐름 속도장을 학습하는 조건부 flow matching 프레임워크 NeuroSonic을 제안한다. CineBrain과 EAV 벤치마크에서 교차 피험자 평가 시 GAN·확산·mean-flow 대비 분포 현실성, 스펙트럼 충실도, 지각 품질을 개선했으며 아티팩트가 많은 구간에서 전체 지각 품질이 최대 26.3% 향상되었다.

arXiv:2606.24937v1cs.AI

The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

The Hitchhiker's Guide to Agentic AI is a comprehensive practitioner's reference for building autonomous AI systems. The book covers the full stack from first principles to production deployment, organized around a central thesis: building great agentic systems requires understanding every layer of the pipeline, not just one. The book opens with the LLM substrate -- transformer architecture, GPU systems, training and fine-tuning (SFT,LoRA, MoE), model compression, and inference optimization -- treated as essential foundations rather than the primary focus. It then develops the alignment and reasoning layer: reinforcement learning from human feedback (RLHF), PPO, DPO and its variants, GRPO, reward modeling, and RL for large reasoning models including chain-of-thought and test-time scaling. The second half is devoted to agentic AI proper. Topics include agentic training and trajectory-based RL, retrieval-augmented generation (RAG and Agentic RAG), memory systems (in-context, external, episodic, and semantic), agent harness design and context management, and a taxonomy of agent design patterns. Inter-agent coordination is covered in depth: the Model Context Protocol (MCP), agent skills and tool use, the Agent-to-Agent (A2A) communication protocol, and multi-agent architectures spanning centralized, decentralized, and hierarchical topologies. The book concludes with agent development frameworks, agentic UI design, evaluation methodology for agentic tasks, and production deployment. Each chapter pairs rigorous theoretical foundations with implementation guidance, code examples, and references to the primary literature.

arXiv:2509.25699v4cs.CV

AIM-CoT: 비전-언어 추론을 위한 능동적 정보 기반 다중 모달 사고의 연쇄

AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

교차 모달 사고의 연쇄(Interleaved-Modal Chain-of-Thought, I-MCoT)는 시각적 질의응답(VQA)과 같은 비전-언어 추론을 발전시킨다. 이 패러다임은 입력 이미지에서 특별히 선택된 시각적 증거를 비전-언어 모델(VLM)의 컨텍스트에 통합하여, 모델이 이러한 세부 사항에 추론 논리를 그라운딩할 수 있게 한다. 따라서 I-MCoT 프레임워크의 효과는 무엇을 볼지(증거 선택)와 언제 볼지(삽입 트리거)를 식별하는 데 달려 있다. 그러나 기존 방법들은 두 측면 모두에서 부족하다. 첫째, 선택 측면에서 기존 방법들은 어텐션 신호에 의존하는데, 이는 특히 간결한 텍스트 쿼리와 정보가 풍부한 이미지 사이의 심각한 세분성 불균형 하에서 신뢰할 수 없다. 둘째, 트리거 측면에서 정적 트리거를 채택하여 VLM의 시각적 증거에 대한 동적 수요를 포착하지 못한다. 이를 위해 우리는 증거 선택과 삽입 트리거를 모두 개선하기 위해 새로운 I-MCoT 프레임워크인 능동적 정보 기반 다중 모달 사고의 연쇄(AIM-CoT)를 제안한다: (1) 텍스트 컨텍스트 강화를 통해 세분성 불균형을 완화하는 컨텍스트 강화 어텐션 맵 생성(CAG); (2) 정보 수색(information foraging) 과정을 통해 가장 정보가 풍부한 증거를 선제적으로 선택하는 능동적 시각 탐색(AVP); (3) VLM의 어텐션이 텍스트에서 시각적 컨텍스트로 이동할 때 삽입을 정확히 활성화하는 동적 어텐션 전환 트리거(DAT). 세 개의 벤치마크와 네 개의 백본에 걸친 실험은 AIM-CoT의 일관된 우수성을 보여준다. 코드는 https://anonymous.4open.science/r/AIMCoT 에서 제공된다.

arXiv:2608.15964v1cs.CL

표적형 합성 다국어 데이터로 LLM 성능 향상

LLMs Get Smarter from Targeted Synthetic Multilingual Data

언어별 역량(language-specific competency, LSC)은 프롬프트 언어에 따라 언어 모델의 성능이 달라지는 현상이다. 즉, 동일한 의미의 질의에 대해 서로 다른 언어로 프롬프트하면 언어 모델이 서로 다른(그리고 잠재적으로 부정확한) 응답을 낸다. 기존 연구는 이를 언어 간 의미 표현의 내부적 불일치로 돌린다. 현재 문헌에서 LSC를 다루는 주된 접근은 두 가지이다: (1) 모든 질의를 영어로 라우팅하여 성능을 높이되, 언어 표현력을 영어에 한정하는 방법; (2) 언어 균형 데이터로 학습하여 언어 간 성능을 맞추되, 전체 성능을 낮추는 방법이다. 본 연구는 데이터 중심 관점에서 HOTFIXR(Hardness Optimized Training data For Improving X-Lingual Reasoning)을 제안한다. 이는 모델로 학생 모델의 다국어 약점을 탐지·학습하고 이를 완화할 데이터를 생성하는 데이터 생성 프레임워크이다. HOTFIXR은 다국어 성능을 개선할 수 있는 다국어 합성 학습 데이터를 생성할 수 있다. 우리는 분포 내 과제 3개, 분포 외(OOD) 과제 3개, OOD 언어 4개에서 평가한다. 평균적으로 HOTFIXR은 (1) 분포 내 성능을 6.2% 향상하고, (2) OOD 과제에서 (미세조정으로 유발된) 치명적 망각을 3.7% 줄이며, (3) OOD 언어에서 7.1% 개선한다. 전반적으로 많은 실제 응용이 다국어 LLM을 요구하므로, 본 연구는 LLM의 다국어 숙련도를 높이려는 노력에 기여한다. 논문이 수락되면 코드를 공개할 예정이다.

arXiv:2609.03629v1cs.CV

EraseSAE: 희소 오토인코더를 통한 텍스트-비디오 확산 모델에서의 정밀한 개념 소거

EraseSAE: Surgical Concept Erasure in Text-to-Video Diffusion Models via Sparse Autoencoders

텍스트-비디오(T2V) 확산 모델의 최근 발전은 놀라운 생성 능력을 입증했지만, 정밀하게 정제되지 않은 학습(training) 데이터에 대한 의존도로 인해 시급한 안전성(safety) 및 저작권 문제가 제기되고 있습니다. 개념 소거(concept erasure)는 사전학습(pre-training)된 모델에서 원치 않는 의미론적 요소를 제거하면서 나머지 개념은 보존하는 원칙적인 해결책을 제공합니다. 그러나 기존 접근 방식은 일반적으로 개념 표현의 세밀하고 분산된 특성과 일치하지 않는 조악한 세분성(granularity)에서 작동하여, 불완전한 제거로 이어지거나 생성 품질을 저하시킵니다. 본 논문에서는 정밀한(surgical) 소거를 수행하려면 각 단위가 단일한 해석 가능 개념을 인코딩하는 단일 의미적(monosemantic) 특징(feature) 수준에서의 개입이 근본적으로 필요하다고 주장합니다. 이를 위해, 원칙적인 '분해-속성 분석-소거(decompose-attribute-erase)' 파이프라인을 통해 DiT 기반 T2V 확산 모델에서 정밀한 개념 소거를 달성하고자 희소 오토인코더(sparse autoencoder)를 활용하는 새로운 프레임워크인 EraseSAE를 제안합니다. 먼저, 시공간적 일관성을 보존하면서 조밀한 시공간 활성화를 얽힘이 풀린 해석 가능한 희소 특징으로 분해하는 분할 합성곱 희소 오토인코더(Partitioned Convolutional Sparse Autoencoder)를 도입합니다. 그런 다음 대조적 속성 분석 메커니즘을 통해 쌍을 이루는 프롬프트(prompt)의 활성화를 대조하여 개념 특화된 특징 커널을 분리합니다. 추론(inference) 시에는 식별된 커널에서 파생된 타임스텝 단위의 시공간 마스크를 적용하여 목표 개념이 활성화된 영역으로만 소거를 국한함으로써 관련 없는 콘텐츠는 온전히 유지합니다. 다양한 확산 모델과 개념 소거 태스크 전반에 걸친 광범위한 실험을 통해 EraseSAE가 최소한의 품질 저하만으로 정밀하고 강력한 개념 제거를 달성하며 최첨단 방법들을 크게 능가함을 입증합니다. 코드는 https://github.com/HiDream-ai/EraseSAE 에서 확인할 수 있습니다.

arXiv:2610.08738v1cs.CL

계층적 표현의 노이즈 제거: 언어 모델링을 위한 결합 연속 확산

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

확산 언어 모델(DLM)은 순서에 구애받지 않는 병렬 텍스트 생성의 가능성을 지니고 있습니다. 최근 정교하게 설계된 토큰 표현과 확산/플로우 공간에 힘입어 연속 확산 및 플로우 매칭 모델이 상당한 발전을 이루었습니다. 본 연구에서는 최소한의 연산 및 매개변수 오버헤드로 연속 DLM을 더욱 향상시키는 단순한 프레임워크인 계층적 연속 확산 언어 모델(H-CDLM)을 소개합니다. 결합 확산에 관한 이산 DLM 및 연속 이미지 확산 연구를 바탕으로, 여러 모달리티를 병렬로 확산시킵니다. 이러한 모달리티들은 서로 다른 의미적 세분성으로 토큰을 나타내며, 본 구현에서는 토큰 자체와 사전학습된 토큰 임베딩을 클러스터링하여 얻은 더 거친 클러스터를 사용합니다. 또한 모달리티 간의 상호작용을 강화하기 위해 모달리티별 샘플러와 스케줄을 허용하는 일반적인 설정을 제안합니다. 이를 CoBit에 적용하여 얻은 H-CoBit은 여러 벤치마크에서 큰 폭의 실증적 향상을 보였습니다. 데이터셋 엔트로피에서 H-CoBit은 MAUVE를 향상시키고 LM1B에서 49.4, OWT에서 50.4의 생성 퍼플렉시티(GenPPL)를 기록하여 기준선 대비 각각 24.2점 및 20.7점 향상되었으며, 비슷한 크기의 이산 DLM마저 능가했습니다. GSM8K에서는 27.4%의 정확도를 달성하여 기존의 연속 확산 및 플로우 기반 모델들을 능가했습니다. 나아가 H-CDLM을 플로우 매칭 모델인 FLM에 적용하여 H-FLM을 통해 일관된 성능 향상을 얻었으며, 이 프레임워크가 연속 생성 패러다임 전반에 걸쳐 일반화됨을 입증했습니다. 코드는 https://github.com/matol-16/HCDLM.git 에 공개될 예정입니다.

arXiv:2610.11067v1cs.CV

일반화 가능한 파운데이션 모델 적응을 위한 확산 메타 프롬프팅 및 스티어링

Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

프롬프트 학습(Prompt learning)은 파운데이션 모델(foundation model)을 적응시키는 널리 쓰이는 방법이지만, 학습된 프롬프트는 대개 특정 태스크에 국한되어 새로운 클래스, 도메인, 또는 태스크의 조합으로 일반화되지 못합니다. 본 논문에서는 확산 모델(diffusion model)을 사용하여 학습된 프롬프트의 분포를 모델링하는 프레임워크인 확산 메타 프롬프트(Diffusion Meta-Prompt, DMP) 모델을 소개합니다. 기존에 학습된 프롬프트 저장소가 주어지면, DMP는 원본 태스크 예시나 태스크 손실(task loss)에 접근하지 않고도 학습 및 샘플링되며, 자연어 태스크 설명을 조건으로 삼아 새로운 프롬프트를 합성합니다. 샘플링 안정성을 향상하기 위해, 당사는 DMP를 재학습하거나 테스트 클래스에 접근하지 않고도 확산 샘플링 중에 저장소 내 최적의 학습 선택 프롬프트를 잠재 앵커(latent anchor)로 사용하는 DMP용 테스트 타임 스티어링(test-time steering) 전략을 제안합니다. DMP는 분류(classification), 검색, 텍스트-이미지 생성 태스크 전반에서 일반화 성능을 향상시키며, 명시적인 학습 없이도 개념 조합과 네거티브 프롬프팅(negative prompting)을 지원합니다. 또한 프롬프트 검색 방식에 비해 스토리지 및 추론(inference) 비용을 90% 이상 절감합니다. 복합 분류에서 DMP는 55쌍의 데이터셋에 걸쳐 기존 메타러닝(meta-learning) 방식 대비 평균 최대 2.0%의 성능 향상을 달성하며, Eurosat 및 Flowers와 같은 특정 데이터셋 쌍에서는 최대 8.5%의 향상을 보입니다. DMP는 계층적 분류 태스크에서도 약 2~9%의 향상을 보이며 태스크 간 일반화 성능 역시 강화합니다. 나아가 당사는 DMP에서 샘플링된 프롬프트의 기대 태스크 손실을 제한하는 이론적 보장을 제공합니다. 코드는 https://github.com/DeepakSridhar/dmp 에서 제공됩니다.

arXiv:2611.22499v1cs.CL

음절 표현에서 제스처 생성을 거쳐 조음으로: 스파이킹 신경 모델

From syllable representation via gesture generation to articulation: A spiking neural model

말을 생성하려면 음운 표현을 서로 조율된 성도 움직임으로 바꿔야 한다. 신경공학 프레임워크(NEF)와 의미 포인터 아키텍처(SPA)에 기반한 스파이킹 신경 감각운동 모델을 제시한다. 각 음절이 나타날 때 신경 진동자 하나가 대응한다. 변하는 위상이 음절의 조음 동작을 조율하고 다음 음절을 시작한다. 위상에 따라 달라지는 명령은 모음, 구강 협착, 연구개·인두, 성문 동작을 담당하는 서로 다른 신경 집단을 활성화한다. 신경 램프 적분기는 동작 활성의 상승과 하강을 조절한다. 이렇게 얻은 궤적은 조음 목표와 함께 조음 운동의 범위와 시간 경과를 정한다. 서로 반대 작용을 하는 신경 집단이 기하학적 조음 모델에 제어 신호를 보낸다. 참조 녹음 30건은 어휘 단어 여덟 개, 그중 한 단어의 다른 발음, 음절 시퀀스 하나, 개별 음절로 구성된 입력 말뭉치를 포괄한다. 신경 구현이 달라져도 활성화가 일관되는지 평가하기 위해, 단어 두 개와 음절 시퀀스 하나를 다루는 짝지은 추가 시뮬레이션 하위 말뭉치 두 개를 사용했다. 각 말뭉치에는 실행 60회가 포함됐다. 램프의 변화율 계수를 고정하고 동작 명령의 진폭을 조정하자 정해진 기준을 만족한 실현이 60회 중 57회에서 60회로 늘었다. 이 모델은 주어진 위상 규칙에서 신경 진동자와 램프의 동역학이 조음 타이밍을 어떻게 생성하는지 보여 준다. 순서가 있는 음절 인터페이스로 앞서 개발한 심성 어휘 모델을 보완한다. 모든 구성 요소의 연속 온라인 실행과 음향 확장 모듈 연결은 후속 통합 과제다. 공개 코드를 사용하고 동기화된 시각화를 통해 신경 활성화 궤적과 그에 따른 성도 움직임을 살펴볼 수 있다.

arXiv:2003.10739공식 코드 · Papers with Code

Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives

While the depth of modern Convolutional Neural Networks (CNNs) surpasses that of the pioneering networks with a significant margin, the traditional way of appending supervision only over the final classifier and progressively propagating gradient flow upstream remains the training mainstay. Seminal Deeply-Supervised Networks (DSN) were proposed to alleviate the difficulty of optimization arising from gradient flow through a long chain. However, it is still vulnerable to issues including interference to the hierarchical representation generation process and inconsistent optimization objectives, as illustrated theoretically and empirically in this paper. Complementary to previous training strategies, we propose Dynamic Hierarchical Mimicking, a generic feature learning mechanism, to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Each branch can emerge from certain locations of the main branch dynamically, which not only retains representation rooted in the backbone network but also generates more diverse representations along its own pathway. We go one step further to promote multi-level interactions among different branches through an optimization formula with probabilistic prediction matching losses, thus guaranteeing a more robust optimization process and better representation ability. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method over its corresponding counterparts using diverse state-of-the-art CNN architectures. Code and models are publicly available at https://github.com/d-li14/DHM

arXiv:2103.12204공식 코드 · Papers with Code

Human-like Controllable Image Captioning with Verb-specific Semantic Roles

Controllable Image Captioning (CIC) -- generating image descriptions following designated control signals -- has received unprecedented attention over the last few years. To emulate the human ability in controlling caption generation, current CIC studies focus exclusively on control signals concerning objective properties, such as contents of interest or descriptive patterns. However, we argue that almost all existing objective control signals have overlooked two indispensable characteristics of an ideal control signal: 1) Event-compatible: all visual contents referred to in a single sentence should be compatible with the described activity. 2) Sample-suitable: the control signals should be suitable for a specific image sample. To this end, we propose a new control signal for CIC: Verb-specific Semantic Roles (VSR). VSR consists of a verb and some semantic roles, which represents a targeted activity and the roles of entities involved in this activity. Given a designated VSR, we first train a grounded semantic role labeling (GSRL) model to identify and ground all entities for each role. Then, we propose a semantic structure planner (SSP) to learn human-like descriptive semantic structures. Lastly, we use a role-shift captioning model to generate the captions. Extensive experiments and ablations demonstrate that our framework can achieve better controllability than several strong baselines on two challenging CIC benchmarks. Besides, we can generate multi-level diverse captions easily. The code is available at: https://github.com/mad-red/VSR-guided-CIC.