PyTorchKR Papers

“Code Generation”

총 10,186편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2611.21299v1cs.AI

Turing Agent: 수학 연구를 위한 에이전트 기반 형식 검증

Turing Agent: Agentic Formal Verification for Mathematical Research

대규모 언어 모델(LLM)은 벤치마크 문제 해결을 넘어 최전선의 연구 문제에 도전하며 수학적 발견을 빠르게 발전시키고 있다. 이 발전으로 증명이 대량으로 만들어지는 시대가 열리면서 근본적인 문제가 생겼다. 늘어나는 AI 생성 증명의 정확성을 어떻게 확인하고, 이를 기계로 검증한 신뢰할 수 있는 수학 지식으로 바꿀 수 있는가? 본 연구는 연구 수준 수학의 종단 간 형식 검증을 조율하도록 설계한 에이전트 실행 환경 Turing Agent를 소개한다. 두 수준의 아키텍처는 정리 형식화 에이전트와 증명 형식화 에이전트를 통합한다. 구조화된 증명 분해, 병렬 검증, 축적된 증명 경험, 선택적 의미 검토를 활용한다. 이론 컴퓨터과학(TCS) 연구자와의 협력을 바탕으로, 복잡한 양화와 의존 관계, 엄격한 의미적 요구를 갖는 특히 어려운 TCS 영역을 중심으로 평가했다. GLM-5.3과 DeepSeek-V4-Pro를 사용해 직접 생성 및 Codex CLI와 Claude Code 등 범용 코딩 실행 환경과 비교했다. Turing Agent는 Formal TCS Bench에서 원문에 충실한 형식화 성공률 약 57~62%를, RLMEval의 TCS 중심 하위 집합에서는 전체 pass@1 83%를 달성했다. 특히 DeepSeek 기반 Turing Agent는 OpenAI의 2026년 10월 문제 모음에서 이전에 해결되지 않은 세 가지 TCS 과제의 형식 검증을 완료했다. 특히 어려웠던 문제 하나에는 형식적으로 검증한 보조 명제 약 100개가 필요했다. 또한 형식화 과정을 직접 탐색하고 개선하는 중에 더 간단한 대안 증명을 찾거나, 원래 증명에서 불필요한 의존을 드러내거나, 재사용 가능한 수학 지식을 검증의 부산물로 생성할 수 있다.

arXiv:2607.06136v1cs.CV

초고해상도 이미지 편집을 위한 튜닝-프리 잠재 확산 모델

Tuning-Free Latent Diffusion Models for Ultrahigh-Resolution Image Editing

기존 잠재 확산 모델(LDM)이 메모리 제약으로 1K 미만 해상도에 제한되는 문제를 해결하기 위해, UltraDiffEdit는 다중 스케일 점진적 편집, 멀티 패치 인코딩, 글로벌-로컬 일관성 디노이징, 패치 기반 하이브리드 샘플링을 결합한 튜닝-프리(tuning-free) 이미지 편집 프레임워크를 제안한다. 추가 학습 없이 기존 사전학습된 LDM(SDXL, ControlNet 등)을 활용하여 단일 RTX 3090 GPU만으로 최대 8K 해상도의 실제 이미지 편집을 달성하며 경계 아티팩트와 의미론적 일관성 문제를 효과적으로 해결한다.

arXiv:2607.08079v1cs.AI

PARA-PV: 고정된 기반 모델과 분포 이동 보정 기반의 물리 인식 검색-증강 태양광 발전 예측

PARA-PV: Physics-Aware Retrieval-Augmented PV Prediction Based on Frozen Foundation Model and Distribution Shift Correction

PARA-PV는 태양광(PV) 발전 예측의 정확도를 높이기 위해 물리적 지식을 예측 전 과정에 내재화한 검색-증강(retrieval-augmented) 프레임워크로, 물리 인식 검색 학습기, 고정된 Chronos 기반 모델 기반의 시간적 사전(prior) 보정, 분포 이동 보정 모듈, 그리고 피크·램핑·야간 등 레짐별 적응적 손실 함수를 결합한다. 두 개의 지리적으로 상이한 PV 발전소 실험에서 7개의 베이스라인 대비 일관되게 우수한 포인트 및 확률적 예측 성능을 1M 미만의 학습 가능 파라미터로 달성하였다.

arXiv:2607.27113v1cs.CV

Veritas++: 지각 능력이 강화된 AI 생성 이미지 탐지를 위한 가치 인식 온-정책 지식 증류

Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection

이미지 생성 모델의 역량이 발전함에 따라 오픈 미디어에서 합성 이미지를 일상적으로 접하게 되었으며, 이에 따라 강건하고 일반화 가능한 AI 생성 이미지(AI-Generated Image, AIGI) 탐지의 중요성이 점차 커지고 있습니다. 멀티모달 대규모 언어 모델(MLLM)은 블랙박스 형태의 이진 점수화 방식에 대한 투명한 대안을 제시하지만, 현재 MLLM 기반 탐지 모델들은 여전히 미세한 이상 현상을 포착하는 데 있어 두드러진 지각 병목 현상을 보입니다. 이러한 모델들은 시각적 증거가 어떻게 구성되고 종합되는지에 주로 초점을 맞추어, 본질적인 지각 능력은 덜 최적화된 상태로 남겨둡니다. 이러한 격차를 완화하기 위해 본 논문에서는 신뢰할 수 있는 지각을 진위 추론의 기반으로 확립하는 지각 강화 추론 프레임워크인 Veritas++를 제안합니다. 모델의 설명 능력을 직접 최적화하기보다는, 미세한 시각적 세부 사항 포착, 의미론적 이상 현상 포착, 픽셀 수준의 차이 포착이라는 세 가지 기본 지각 능력에 AIGI 탐지의 기반(grounding)을 둡니다. 이러한 통찰을 바탕으로, 개방형 설명 지도(supervision)를 검증 가능한 보상으로 대체하여 이러한 역량을 명시적으로 강화하는 지각 지향 학습(PoRL)을 도입합니다. 나아가 강화된 지각을 추론과 통합하기 위해, 균일한 지도보다 가치가 높은 지식 증류 신호를 우선시하여 특권적인 자기 교사(privileged self-teacher)를 통해 지각 인식 추론을 내재화하는 적응형 증류 메커니즘인 가치 인식 온-정책 지식 증류(VaOPD)를 도입합니다. 표준, 실제 환경(in-the-wild), 최신 벤치마크 전반에 걸친 광범위한 실험을 통해 Veritas++가 유망한 일반화 성능을 달성함을 입증합니다. 지각 학습은 지각 격차를 효과적으로 해소하고 탐지 성능의 원활한 향상을 이끌어내며, VaOPD는 기존 성능을 저하시키지 않으면서 효율적인 역량 발전을 가능하게 합니다. 코드와 체크포인트는 https://github.com/EricTan7/VeritasPP 에서 확인할 수 있습니다.

arXiv:2608.24439v1cs.CV

DoublesEval: 프로 배드민턴 복식 경기를 통한 비전-언어 모델의 다중 에이전트 전술적 추론 진단

DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

비전-언어 모델(VLM)은 눈에 보이는 장면의 내용을 설명하는 데는 뛰어나지만, 행동의 의미가 조율된 역할과 시공간적 의존성에 좌우되는 동적인 다중 에이전트(multi-agent) 상호작용을 추론하는 데는 어려움을 겪습니다. 본 연구에서는 이러한 역량을 다중 에이전트 전술적 추론(multi-agent tactical reasoning)으로 정형화하고, 구조적으로 다루기 쉬운 테스트베드로 프로 배드민턴 복식을 활용하는 진단 평가 프레임워크인 DoublesEval을 소개합니다. DoublesEval은 랠리를 전술적으로 유의미한 순간들로 분해하는 주요 순간 기반 프로토콜을 사용하며, 원자적 인식, 세그먼트 내 복합적 이해, 세그먼트 간 인과적 추론, 고수준 전술적 추상화의 네 가지 해석 가능한 차원에 걸쳐 모델을 탐색합니다. 이러한 설계는 단순히 답변의 정답 유무만을 측정하기보다 추론이 실패하는 지점을 격리하여 파악합니다. 관찰된 실패 모드를 해결하기 위해, 추론(inference) 시 매개변수(parameter) 업데이트나 정답 라벨 없이도 모델 자체의 저수준 전술 예측을 활용해 후보 답변을 재순위화하는 경량의 제약 조건 유도형 테스트 타임 일관성 검사기인 TacticCheck를 제안합니다. 정선된 60개의 랠리(약 9.6K개의 구조화된 인스턴스 도출)에서 제로샷 프로토콜을 통해 네 가지 대표적인 오픈소스 VLM을 평가한 결과, 모델들이 모든 진단 수준에서 여전히 취약하며 특히 공간 상태, 상호작용 바인딩, 최종 증거에서 뚜렷한 병목 현상을 보임을 확인했습니다. TacticCheck는 평가된 모든 모델에서 일관된 성능 향상을 제공하지만, 견고한 전술적 추론에는 여전히 상당한 격차가 남아 있습니다. 이러한 결과는 차세대 VLM을 위한 구조화되고 상호작용을 인식하는 평가 패러다임의 필요성을 강조합니다. 소스 코드는 GitHub 저장소에서 확인할 수 있습니다.

arXiv:2609.11722v1cs.CV

아바타-애즈-이미지(Avatar-As-Image) 재고: 고충실도 정합만으로 충분하다

Revisiting Avatar-As-Image: High-Fidelity Registration is All You Need

기저 인체 모델(body model) 위에 표준화된 2D UV 텍스처 및 변위 맵(displacement map) 형태로 3D 착의 인체를 표현하는 방식은 오랫동안 연구되어 왔습니다. 이러한 간결한 표현 방식은 사전학습된 이미지 네트워크가 3D 아바타를 처리, 생성 및 편집할 수 있도록 지원하여 매력적이지만, 스캔 데이터가 정밀하게 정렬되고 고충실도 정합(high-fidelity registration)을 통해 대응 관계가 형성되어야만 유용합니다. 이러한 전제 조건은 지금까지 충족된 적이 없으며, 우리는 이것이 착의 인체 대상의 기존 UV 기반 방법들의 품질이 제한적이었던 이유라고 주장합니다. 그 중요성에도 불구하고, 임의의 착의 스캔 데이터로부터 UV 텍스처를 갖춘 고충실도 SMPL(-X)+D 정합을 생성하는 공개 방법은 존재하지 않습니다. 본 논문에서는 이러한 간극을 메우기 위해 다단계 최적화(optimization) 파이프라인인 AvaImg를 제시합니다. 이는 부호화 감김 수(signed winding numbers)를 통해 의복 내부 신체(body-inside-clothing) 제약 조건을 적용하며, 3단계 효율성 캐스케이드를 통해 실현 가능하게 만들고(~10배 실행 시간 단축, ~95% 저장 공간 절약), 조밀(coarse-to-fine) 변위 최적화를 사용하여 미세한 표면 디테일을 복원합니다. AvaImg는 6개 데이터셋 전반에서 바디 피팅, 형상 추정 및 표면 정합 면에서 모든 베이스라인을 능가하여 스캔과 거의 구분할 수 없는 텍스처 정합 결과(PSNR=34.48dB)를 제공합니다. 이미지 파운데이션 모델과 즉각 호환되는 AvaImg의 아바타-애즈-이미지 표현을 검증(validation)하기 위해, 동결된 FLUX VAE를 통해 UV 맵을 오토인코딩합니다. 이는 스캔 대비 단 0.76mm의 챔퍼 오차(Chamfer error) 증가만을 달성하며, 결과 맵이 자연 이미지 분포 내에 위치함을 보여주어 3D 아바타 생성을 위한 2D 생성 사전확률(generative priors)의 활용을 뒷받침합니다. 코드, 데이터 및 Singularity 컨테이너는 https://yuxuan-xue.com/avaimg 에서 제공될 예정입니다.

arXiv:2610.16141v1cs.AI

팬텀 병합: 대규모 언어 모델 에이전트가 하나를 선택하고 다른 대상에 대해 설명할 때

Phantom Merge: When Your Large Language Model Agents Pick One but Tell You About Another

대규모 언어 모델(LLM) 에이전트가 여러 엔티티를 다루는 작업이 점차 늘어남에 따라, 신뢰할 수 있는 응답을 생성하기 위해서는 관련 근거를 검색하는 것뿐만 아니라 올바른 엔티티와의 연관성을 보존하는 것이 필요합니다. 본 논문에서는 작업이 올바르게 완료된 경우에도 에이전트가 자신이 확정한 엔티티에 뒷받침되지 않는 주장을 귀속시키는 엔티티-근거 결합 오류인 팬텀 병합(Phantom Merge, PM)을 규명합니다. 저자들은 PM을 주장 단위에서 정형화하고 4개의 에이전트 도메인과 4개의 백본 패밀리 전반에서 그 유병률과 오류 하위 유형을 특성화합니다. 이 문제를 해결하기 위해, 주장 단위 위험 추정 및 교정을 위한 2단계 프레임워크를 제안합니다. 첫째, 앵커 기반 위험(Anchor-Grounded Risk, AGR)은 에이전트의 방출 전 은닉 상태의 표현 판독값과 야코비안(Jacobian) 기반 기능 정보를 결합하여 상호 보완적인 슬롯 단위 및 앵커 값 위험 점수를 제공합니다. 둘째, 고정 앵커 교정(Fixed-Anchor Correction, FAC)은 플래그가 지정된 주장을 감사하고 관찰된 앵커 근거를 사용하여 선택적으로 수정하거나, 뒷받침되는 대체물이 없는 경우 제거합니다. 홀드아웃된 Shopping 탐지 코호트에서 AGR-slot은 96.2%의 AUROC와 97.3%의 평균 정밀도를 달성하여 평가된 탐지 베이스라인을 능가했습니다. 엔드투엔드 Shopping 완화 평가에서 FAC는 궤적 단위 PM을 51.4%에서 6.2%로 감소시켰으며, 원본 대비 출력 주장 비율은 80.7%를 기록했습니다. 이러한 결과는 엔티티별 그라운딩(grounding) 실패의 보편성과 내부 위험 추정과 근거 제약 주장 교정을 결합하는 방식의 효과성을 입증합니다. 본 연구의 코드는 https://github.com/LabRAI/PhantomMerge 에서 공개되어 있습니다.

arXiv:1705.03386공식 코드 · Papers with Code

Cell Tracking via Proposal Generation and Selection

Microscopy imaging plays a vital role in understanding many biological processes in development and disease. The recent advances in automation of microscopes and development of methods and markers for live cell imaging has led to rapid growth in the amount of image data being captured. To efficiently and reliably extract useful insights from these captured sequences, automated cell tracking is essential. This is a challenging problem due to large variation in the appearance and shapes of cells depending on many factors including imaging methodology, biological characteristics of cells, cell matrix composition, labeling methodology, etc. Often cell tracking methods require a sequence-specific segmentation method and manual tuning of many tracking parameters, which limits their applicability to sequences other than those they are designed for. In this paper, we propose 1) a deep learning based cell proposal method, which proposes candidates for cells along with their scores, and 2) a cell tracking method, which links proposals in adjacent frames in a graphical model using edges representing different cellular events and poses joint cell detection and tracking as the selection of a subset of cell and edge proposals. Our method is completely automated and given enough training data can be applied to a wide variety of microscopy sequences. We evaluate our method on multiple fluorescence and phase contrast microscopy sequences containing cells of various shapes and appearances from ISBI cell tracking challenge, and show that our method outperforms existing cell tracking methods. Code is available at: https://github.com/SaadUllahAkram/CellTracker

arXiv:2201.11932공식 코드 · Papers with Code

Deep Generative Model for Periodic Graphs

Periodic graphs are graphs consisting of repetitive local structures, such as crystal nets and polygon mesh. Their generative modeling has great potential in real-world applications such as material design and graphics synthesis. Classical models either rely on domain-specific predefined generation principles (e.g., in crystal net design), or follow geometry-based prescribed rules. Recently, deep generative models has shown great promise in automatically generating general graphs. However, their advancement into periodic graphs have not been well explored due to several key challenges in 1) maintaining graph periodicity; 2) disentangling local and global patterns; and 3) efficiency in learning repetitive patterns. To address them, this paper proposes Periodical-Graph Disentangled Variational Auto-encoder (PGD-VAE), a new deep generative models for periodic graphs that can automatically learn, disentangle, and generate local and global graph patterns. Specifically, we develop a new periodic graph encoder consisting of global-pattern encoder and local-pattern encoder that ensures to disentangle the representation into global and local semantics. We then propose a new periodic graph decoder consisting of local structure decoder, neighborhood decoder, and global structure decoder, as well as the assembler of their outputs that guarantees periodicity. Moreover, we design a new model learning objective that helps ensure the invariance of local-semantic representations for the graphs with the same local structure. Comprehensive experimental evaluations have been conducted to demonstrate the effectiveness of the proposed method. The code of proposed PGD-VAE is availabe at https://github.com/shi-yu-wang/PGD-VAE.

arXiv:2307.08941공식 코드 · Papers with Code

MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models

Fine-tuning a pre-trained language model (PLM) emerges as the predominant strategy in many natural language processing applications. However, this process is known to be expensive, especially on edge devices with low computing power. While general approaches (e.g. quantization and distillation) have been widely studied to reduce the compute/memory of PLM fine-tuning, one-shot compression techniques specifically designed for fine-tuning remain largely unexplored. In this paper, we investigate the neural tangent kernel (NTK)--which reveals the gradient descent dynamics of neural networks--of the multilayer perceptrons (MLP) modules in a PLM and propose to coin a lightweight PLM through NTK-approximating MLP fusion. By incorporating NTK into the compression process, MLP Fusion not only preserves the original model's output but also maintains its training dynamics. To achieve this, we reconsider the MLP as a bundle of sub-MLPs and cluster them into a given number of centroids, which can then be restored as a compressed MLP and surprisingly well approximate the NTK of the original PLM. Our approach is applicable to both standard MLP modules and Mixture-of-Experts (MoE) modules in PLMs, demonstrating its scalability and versatility. Additionally, we provide theoretical derivations to demonstrate how the proposed compression preserves the NTK. Extensive experiments of PLM fine-tuning on both natural language understanding and generation tasks are provided to verify the effectiveness of MLP fusion. Our code is available at https://github.com/weitianxin/MLP_Fusion.

arXiv:2311.16081공식 코드 · Papers with Code

ViT-Lens: Towards Omni-modal Representations

Aiming to advance AI agents, large foundation models significantly improve reasoning and instruction execution, yet the current focus on vision and language neglects the potential of perceiving diverse modalities in open-world environments. However, the success of data-driven vision and language models is costly or even infeasible to be reproduced for rare modalities. In this paper, we present ViT-Lens-2 that facilitates efficient omni-modal representation learning by perceiving novel modalities with a pretrained ViT and aligning them to a pre-defined space. Specifically, the modality-specific lens is tuned to project any-modal signals to an intermediate embedding space, which are then processed by a strong ViT with pre-trained visual knowledge. The encoded representations are optimized toward aligning with the modal-independent space, pre-defined by off-the-shelf foundation models. ViT-Lens-2 provides a unified solution for representation learning of increasing modalities with two appealing advantages: (i) Unlocking the great potential of pretrained ViTs to novel modalities effectively with efficient data regime; (ii) Enabling emergent downstream capabilities through modality alignment and shared ViT parameters. We tailor ViT-Lens-2 to learn representations for 3D point cloud, depth, audio, tactile and EEG, and set new state-of-the-art results across various understanding tasks, such as zero-shot classification. By seamlessly integrating ViT-Lens-2 into Multimodal Foundation Models, we enable Any-modality to Text and Image Generation in a zero-shot manner. Code and models are available at https://github.com/TencentARC/ViT-Lens.

arXiv:2402.08756

Learning How To Ask: Cycle-Consistency Refines Prompts in Multimodal Foundation Models

When LLMs perform zero-shot inference, they typically use a prompt with a task specification, and generate a completion. However, there is no work to explore the possibility of the reverse - going from completion to task specification. In this paper, we employ both directions to perform cycle-supervised learning entirely in-context. Our goal is to create a forward map f : X -> Y (e.g. image -> generated caption), coupled with a backward map g : Y -> X (e.g. caption -> generated image) to construct a cycle-consistency "loss" (formulated as an update to the prompt) to enforce g(f(X)) ~= X. The technique, called CyclePrompt, uses cycle-consistency as a free supervisory signal to iteratively craft the prompt. Importantly, CyclePrompt reinforces model performance without expensive fine-tuning, without training data, and without the complexity of external environments (e.g. compilers, APIs). We demonstrate CyclePrompt in two domains: code generation and image captioning. Our results on the HumanEval coding benchmark put us in first place on the leaderboard among models that do not rely on extra training data or usage of external environments, and third overall. Compared to the GPT4 baseline, we improve accuracy from 80.5% to 87.2%. In the vision-language space, we generate detailed image captions which outperform baseline zero-shot GPT4V captions, when tested against natural (VQAv2) and diagrammatic (FigureQA) visual question-answering benchmarks. To the best of our knowledge, this is the first use of self-supervised learning for prompting.

arXiv:2403.04317공식 코드 · Papers with Code

Online Adaptation of Language Models with a Memory of Amortized Contexts

Due to the rapid generation and dissemination of information, large language models (LLMs) quickly run out of date despite enormous development costs. To address the crucial need to keep models updated, online learning has emerged as a critical tool when utilizing LLMs for real-world applications. However, given the ever-expanding corpus of unseen documents and the large parameter space of modern LLMs, efficient adaptation is essential. To address these challenges, we propose Memory of Amortized Contexts (MAC), an efficient and effective online adaptation framework for LLMs with strong knowledge retention. We propose a feature extraction and memory-augmentation approach to compress and extract information from new documents into compact modulations stored in a memory bank. When answering questions, our model attends to and extracts relevant knowledge from this memory bank. To learn informative modulations in an efficient manner, we utilize amortization-based meta-learning, which substitutes an otherwise required optimization process with a single forward pass of the encoder. Subsequently, we learn to choose from and aggregate selected documents into a single modulation by conditioning on the question, allowing us to adapt a frozen language model during test time without requiring further gradient updates. Our experiment demonstrates the superiority of MAC in multiple aspects, including online adaptation performance, time, and memory efficiency. In addition, we show how MAC can be combined with and improve the performance of popular alternatives such as retrieval augmented generations (RAGs). Code is available at: https://github.com/jihoontack/MAC.

arXiv:2404.01330공식 코드 · Papers with Code

P-Hologen: An End-to-End Generative Framework for Phase-Only Holograms

Holography stands at the forefront of visual technology, offering immersive, three-dimensional visualizations through the manipulation of light wave amplitude and phase. Although generative models have been extensively explored in the image domain, their application to holograms remains relatively underexplored due to the inherent complexity of phase learning. Exploiting generative models for holograms offers exciting opportunities for advancing innovation and creativity, such as semantic-aware hologram generation and editing. Currently, the most viable approach for utilizing generative models in the hologram domain involves integrating an image-based generative model with an image-to-hologram conversion model, which comes at the cost of increased computational complexity and inefficiency. To tackle this problem, we introduce P-Hologen, the first end-to-end generative framework designed for phase-only holograms (POHs). P-Hologen employs vector quantized variational autoencoders to capture the complex distributions of POHs. It also integrates the angular spectrum method into the training process, constructing latent spaces for complex phase data using strategies from the image processing domain. Extensive experiments demonstrate that P-Hologen achieves superior quality and computational efficiency compared to the existing methods. Furthermore, our model generates high-quality unseen, diverse holographic content from its learned latent space without requiring pre-existing images. Our work paves the way for new applications and methodologies in holographic content creation, opening a new era in the exploration of generative holographic content. The code for our paper is publicly available on https://github.com/james0223/P-Hologen.

arXiv:2408.01697공식 코드 · Papers with Code

Invariant Graph Learning Meets Information Bottleneck for Out-of-Distribution Generalization

Graph out-of-distribution (OOD) generalization remains a major challenge in graph learning since graph neural networks (GNNs) often suffer from severe performance degradation under distribution shifts. Invariant learning, aiming to extract invariant features across varied distributions, has recently emerged as a promising approach for OOD generation. Despite the great success of invariant learning in OOD problems for Euclidean data (i.e., images), the exploration within graph data remains constrained by the complex nature of graphs. Existing studies, such as data augmentation or causal intervention, either suffer from disruptions to invariance during the graph manipulation process or face reliability issues due to a lack of supervised signals for causal parts. In this work, we propose a novel framework, called Invariant Graph Learning based on Information bottleneck theory (InfoIGL), to extract the invariant features of graphs and enhance models' generalization ability to unseen distributions. Specifically, InfoIGL introduces a redundancy filter to compress task-irrelevant information related to environmental factors. Cooperating with our designed multi-level contrastive learning, we maximize the mutual information among graphs of the same class in the downstream classification tasks, preserving invariant features for prediction to a great extent. An appealing feature of InfoIGL is its strong generalization ability without depending on supervised signal of invariance. Experiments on both synthetic and real-world datasets demonstrate that our method achieves state-of-the-art performance under OOD generalization for graph classification tasks. The source code is available at https://github.com/maowenyu-11/InfoIGL.

arXiv:2409.16631공식 코드 · Papers with Code

Enhancing Nighttime UAV Tracking with Light Distribution Suppression

Visual object tracking has boosted extensive intelligent applications for unmanned aerial vehicles (UAVs). However, the state-of-the-art (SOTA) enhancers for nighttime UAV tracking always neglect the uneven light distribution in low-light images, inevitably leading to excessive enhancement in scenarios with complex illumination. To address these issues, this work proposes a novel enhancer, i.e., LDEnhancer, enhancing nighttime UAV tracking with light distribution suppression. Specifically, a novel image content refinement module is developed to decompose the light distribution information and image content information in the feature space, allowing for the targeted enhancement of the image content information. Then this work designs a new light distribution generation module to capture light distribution effectively. The features with light distribution information and image content information are fed into the different parameter estimation modules, respectively, for the parameter map prediction. Finally, leveraging two parameter maps, an innovative interweave iteration adjustment is proposed for the collaborative pixel-wise adjustment of low-light images. Additionally, a challenging nighttime UAV tracking dataset with uneven light distribution, namely NAT2024-2, is constructed to provide a comprehensive evaluation, which contains 40 challenging sequences with over 74K frames in total. Experimental results on the authoritative UAV benchmarks and the proposed NAT2024-2 demonstrate that LDEnhancer outperforms other SOTA low-light enhancers for nighttime UAV tracking. Furthermore, real-world tests on a typical UAV platform with an NVIDIA Orin NX confirm the practicality and efficiency of LDEnhancer. The code is available at https://github.com/vision4robotics/LDEnhancer.

arXiv:2409.19143

Diverse Code Query Learning for Speech-Driven Facial Animation

Speech-driven facial animation aims to synthesize lip-synchronized 3D talking faces following the given speech signal. Prior methods to this task mostly focus on pursuing realism with deterministic systems, yet characterizing the potentially stochastic nature of facial motions has been to date rarely studied. While generative modeling approaches can easily handle the one-to-many mapping by repeatedly drawing samples, ensuring a diverse mode coverage of plausible facial motions on small-scale datasets remains challenging and less explored. In this paper, we propose predicting multiple samples conditioned on the same audio signal and then explicitly encouraging sample diversity to address diverse facial animation synthesis. Our core insight is to guide our model to explore the expressive facial latent space with a diversity-promoting loss such that the desired latent codes for diversification can be ideally identified. To this end, building upon the rich facial prior learned with vector-quantized variational auto-encoding mechanism, our model temporally queries multiple stochastic codes which can be flexibly decoded into a diverse yet plausible set of speech-faithful facial motions. To further allow for control over different facial parts during generation, the proposed model is designed to predict different facial portions of interest in a sequential manner, and compose them to eventually form full-face motions. Our paradigm realizes both diverse and controllable facial animation synthesis in a unified formulation. We experimentally demonstrate that our method yields state-of-the-art performance both quantitatively and qualitatively, especially regarding sample diversity.

arXiv:2412.12101공식 코드 · Papers with Code

InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders

Protein language models (PLMs) have demonstrated remarkable success in protein modeling and design, yet their internal mechanisms for predicting structure and function remain poorly understood. Here we present a systematic approach to extract and analyze interpretable features from PLMs using sparse autoencoders (SAEs). By training SAEs on embeddings from the PLM ESM-2, we identify up to 2,548 human-interpretable latent features per layer that strongly correlate with up to 143 known biological concepts such as binding sites, structural motifs, and functional domains. In contrast, examining individual neurons in ESM-2 reveals up to 46 neurons per layer with clear conceptual alignment across 15 known concepts, suggesting that PLMs represent most concepts in superposition. Beyond capturing known annotations, we show that ESM-2 learns coherent concepts that do not map onto existing annotations and propose a pipeline using language models to automatically interpret novel latent features learned by the SAEs. As practical applications, we demonstrate how these latent features can fill in missing annotations in protein databases and enable targeted steering of protein sequence generation. Our results demonstrate that PLMs encode rich, interpretable representations of protein biology and we propose a systematic framework to extract and analyze these latent features. In the process, we recover both known biology and potentially new protein motifs. As community resources, we introduce InterPLM (interPLM.ai), an interactive visualization platform for exploring and analyzing learned PLM features, and release code for training and analysis at github.com/ElanaPearl/interPLM.

arXiv:2502.02885공식 코드 · Papers with Code

Expertized Caption Auto-Enhancement for Video-Text Retrieval

Video-text retrieval has been stuck in the information mismatch caused by personalized and inadequate textual descriptions of videos. The substantial information gap between the two modalities hinders an effective cross-modal representation alignment, resulting in ambiguous retrieval results. Although text rewriting methods have been proposed to broaden text expressions, the modality gap remains significant, as the text representation space is hardly expanded with insufficient semantic enrichment.Instead, this paper turns to enhancing visual presentation, bridging video expression closer to textual representation via caption generation and thereby facilitating video-text matching.While multimodal large language models (mLLM) have shown a powerful capability to convert video content into text, carefully crafted prompts are essential to ensure the reasonableness and completeness of the generated captions. Therefore, this paper proposes an automatic caption enhancement method that improves expression quality and mitigates empiricism in augmented captions through self-learning.Additionally, an expertized caption selection mechanism is designed and introduced to customize augmented captions for each video, further exploring the utilization potential of caption augmentation.Our method is entirely data-driven, which not only dispenses with heavy data collection and computation workload but also improves self-adaptability by circumventing lexicon dependence and introducing personalized matching. The superiority of our method is validated by state-of-the-art results on various benchmarks, specifically achieving Top-1 recall accuracy of 68.5% on MSR-VTT, 68.1% on MSVD, and 62.0% on DiDeMo. Our code is publicly available at https://github.com/CaryXiang/ECA4VTR.

arXiv:2504.07126

Proposed 2MW Wind Turbine for Use in the Governorate of Dhofar at the Sultanate of Oman

In this work, we propose a preliminary design of a horizontal-axis wind turbine (HAWT) as a candidate for the Dhofar Wind Farm project, in the southern Omani Governorate "Dhofar", at the southwest part of the Sultanate of Oman. This wind farm (under construction) is considered to be the first commercial, utility-scale (50MW) wind farm in the GCC (Gulf Cooperation Council) area. The proposed wind turbine has an expected electricity generation of 2MW. We studied the wind atlas of Oman and from which we determined the maximum possible mean wind speed in the entire Sultanate and built our design based on that reference value, which is 6m/s (21.6km/h). After this, we applied a set of modeling equations that estimate the power output from the wind turbine rotor and matched the target electric power to the design variables using a MATLAB computer code. We reached a suitable design and we present here the distribution of the blade angle (twist angle), and the power per unit span along the rotor blade. The rotor design has 3 blades with a diameter of 70m and a rotational speed of 24rpm. This rotor gives 2.37MW of output power, which exceeds the target 2MW output, allowing for about 15% of power losses in the gearbox and generator. We utilized some commercial designs of wind turbines from different international manufacturers as references for typical limits or recommended values of some design parameters.