PyTorchKR Papers

“Code Generation”

총 10,163편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2606.30246v1cs.AI

Clarus: 웹 규모 과학적 협업을 위한 자율 연구 에이전트 조율

Clarus: Coordinating Autonomous Research Agents toward Web-Scale Scientific Collaboration

기존의 자율 연구 에이전트는 연구 과정의 일부를 지원할 수 있지만, 대부분의 시스템은 여전히 연구를 고립된 보조 작업이나 폐쇄형 워크플로우로 취급한다. 따라서 자율 과학에는 프로젝트, 에이전트, 디지털 및 물리적 자원을 조율하는 협업 인프라가 필요하다. 우리는 이를 코드 중심 실행 루프에서 연구 지향 협업 프로세스로의 전환으로 규정하며, 여기서는 질문, 증거, 참여자, 자원이 불확실성 하에 조율되어야 한다. 이 관점에서 에이전트는 AI 시스템, 인간 연구자, 팀, 연구소, 또는 조직 기반 참여자가 될 수 있다. 이를 위해 우리는 자율 연구 에이전트를 웹 규모의 과학적 협업으로 조율하기 위한 협업 인프라인 Clarus를 제시한다. Clarus는 연구를 개방적이고, 감사 가능하며, 귀속 가능하고, 자원 인식 가능한 다단계 협업 프로세스로 재정식화한다. 최소한의 프로젝트-에이전트-자원 객체 모델을 정의하고, 연구 응용(Research Application), 디지털 협업(Digital Collaboration), 물리적 기반(Physical Substrate), 물리적 세계(Physical World)의 네 개 레이어를 통해 과학적 협업을 구성한다. 핵심 모듈은 플러그형 메커니즘으로 구현되어 있어, Clarus가 작업 위험도, 협업 구조, 자원 제약에 적응할 수 있도록 한다. 통제된 논문 생성 사례 연구를 통해, Clarus가 연구 목표를 단계, 작업, 참여자에 걸쳐 추적 가능하고 검토 가능하며 귀속 가능하고 축적 가능한 협업 네트워크로 구성할 수 있음을 보인다. 객체 모델, 협업 프로토콜, 신뢰 메커니즘, 프로토타입 검증을 종합하여 개방형 연구 네트워크를 위한 초기 기반을 제공한다. Clarus는 현재 clarus.holosai.io에서 이용 가능하다.

arXiv:2607.13753v2cs.CL

사후 학습이 신뢰도를 변화시킨다: SFT, RL, OPD가 CoT 전·중·후 보정을 형성하는 방식에 대한 3단계 분석

Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration

대규모 언어 모델(LLM)은 지도 미세조정, 강화학습, 온-정책 증류를 통해 추론 능력에서 큰 향상을 이루었지만, 이러한 사후 학습 방법들은 보통 최종 답변 정확도로만 평가된다. 우리는 추론 과정에서 이들이 신뢰도(confidence)를 어떻게 재구성하는지 연구한다. 우리는 사고의 연쇄(Chain-of-Thought) 생성 전, 중, 후의 신뢰도를 평가하는 3단계 보정 프레임워크를 도입하며, 이는 각각 난이도 추정, 조기 종료, 답변 집계에 대응한다. 수학적 추론 벤치마크에서의 통제된 비교를 통해, OPD가 사전 추론 신뢰도에 가장 유용하고, SFT가 조기 종료를 위한 최상의 온라인 신호를 제공하며, RL이 집계를 위한 가장 신뢰할 수 있는 trace 수준 신호를 생성함을 발견한다. 또한 신뢰도의 신뢰성은 위치에 따라 달라짐을 보인다: RL 신뢰도는 경로 확정(path-commitment) 단계 이후에 정보를 제공하기 시작하는 반면, OPD 신뢰도는 초기에는 유용하지만 이후 역보정(inversely calibrated)될 수 있다. 이 관찰에 기반하여, 신뢰할 수 있는 상대 위치 구간에서만 신뢰도를 사용하는 위치 인식(position-aware) 신뢰도 전략 PosConf를 제안한다. PosConf는 다수결 투표(majority voting) 대비 RL 답변 집계를 6.1점 향상시키며, 촘촘한 토큰 예산 하에서 OPD 조기 종료를 일관되게 개선하여, 후반 역보정 구간을 피함으로써 최대 4.3점의 이득을 보인다. 이는 추론 모델에서 신뢰도를 단계별로 그리고 위치 인식적으로 사용해야 함을 보여준다. 코드는 https://github.com/EIT-NLP/Post-Training-Calibration 에서 제공된다.

arXiv:2608.12980v1cs.CV

DiCoR: 효율적인 참조 기반 원격 탐사 이미지 분할을 위한 분리형 지시 대상 판별 및 윤곽 재보정

DiCoR: Decoupled Referent Disambiguation and Contour Recalibration for Efficient Referring Remote Sensing Image Segmentation

참조 기반 원격 탐사 이미지 분할(RRSIS)은 원격 탐사 영상에서 자연어 표현으로 지정된 대상을 분할하는 것을 목표로 한다. 기존 방법은 주로 결합 융합 분할(JFS) 또는 분리형 프롬프트 분할(DPS)을 따른다. JFS는 효율적이지만, 지시 대상 위치 추정과 마스크 분할이 통합된 하나의 목적 함수 아래에서 함께 최적화되기 때문에 정확도가 제한되는 경우가 많다. 반면 DPS는 공간 프롬프트와 파운데이션 분할 모델을 사용해 위치 추정과 마스크 생성을 분리하지만, 그 대가로 메모리 소비와 추론 지연 시간이 더 커진다. 이러한 간극을 메우기 위해, 우리는 효율적인 JFS 파이프라인을 기반으로 구축된 분리형 지시 대상 판별 및 윤곽 재보정 프레임워크인 DiCoR를 제안한다. DiCoR는 두 가지 핵심 과제, 즉 모호한 후보들 사이에서 올바른 지시 대상을 구별하는 것과 위치 추정 이후 거친 마스크를 정교화하는 것을 다룬다. 판별 인지형 위치 추정 유도 전략은 적응형 언어적 단서를 사용해 두드러진 후보 영역의 순위를 매기고, 그 결과로 얻은 위치 추정 사전 정보(prior)를 융합된 특징에 주입한다. 또한 경량 윤곽 재보정 모듈은 국소화된 윤곽 지도(supervision) 하에서 거친 로짓(logit)에 대한 잔차 보정값을 예측하여, 제한적인 연산 오버헤드로 마스크 품질을 향상시킨다. RefSegRS, RRSIS-D, RISBench에서의 실험 결과, DiCoR는 세 벤치마크 모두에서 가장 높은 분할 정확도를 달성했다. RefSegRS에서는 경쟁력 있는 JFS 방법 대비 mIoU와 gIoU를 각각 5.28%, 2.87% 향상시키면서, 대표적인 DPS 방법보다 4.7% 더 빠르게 동작하여 정확도와 효율성 사이의 바람직한 균형을 보여준다. 코드는 https://github.com/zyGao1126/DiCoR 에서 확인할 수 있다.

arXiv:2605.15913v5cs.CL

자동 분할 및 블록 증류를 통한 블록 어텐션 일반화

Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation

서로를 어텐션(attention)할 수 없는 개별 블록으로 입력을 처리하는 블록 어텐션은 검색-증강 생성(RAG)과 같은 롱컨텍스트(long-context) 시나리오에서 KV 캐시 재사용을 개선할 수 있는 상당한 잠재력을 제공합니다. 그러나 더 광범위한 적용은 두 가지 주요 과제로 인해 저해됩니다. 입력 텍스트를 의미 있고 독립적인 블록으로 분할하는 데 따르는 어려움과, 성능 저하 위험이 있는 기존 블록 미세조정(fine-tuning) 방식의 비효율성입니다. 이를 해결하기 위해, 본 연구에서는 먼저 도서, 코드, 웹 텍스트, 대화를 포함하여 2k에서 32k에 이르는 텍스트 길이를 가진 16개 범주에 걸쳐 3만 개 이상의 인스턴스를 포함하는 대규모의 다양한 의미론적 분할 데이터셋(dataset)인 SemanticSeg를 구축합니다. 이 데이터셋을 사용하여, 제어 가능한 세분성으로 텍스트를 인간의 직관에 부합하는 블록으로 자동 분할하는 경량 세그멘터를 학습(training)합니다. 둘째로, 고정된 풀 어텐션(full-attention) 교사 모델(teacher model)을 사용하여 블록 어텐션 학생 모델을 유도하는, 블록 미세조정보다 더 효율적인 학습 프레임워크인 블록 증류(block distillation)를 제안합니다. 이 프레임워크는 세 가지 새로운 구성 요소를 통합합니다. 블록 경계에서의 정보 손실을 완화하는 블록 싱크 토큰(token), 모든 블록의 학습 신호를 활용하는 블록 드롭아웃(dropout), 그리고 블록 어텐션에 민감한 토큰에 학습을 집중시키는 토큰 수준 손실(loss) 가중치 부여입니다. 여러 모델과 벤치마크에 걸친 실험은 제안된 세그멘터가 휴리스틱 및 통계적 베이스라인을 능가하며, 블록 증류가 블록 어텐션 하에서 풀 어텐션에 가까운 성능을 달성하여 블록 어텐션을 배포하기 위한 실용적이고 확장 가능한 경로를 구축함을 보여줍니다.

arXiv:2609.11242v1cs.CV

From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models

Video generation has advanced to produce visually compelling and temporally coherent results. Yet, whether these models can genuinely think with video--executing symbolic rules, respecting physical laws, and pursuing intentional goals--remains an open question. Existing benchmarks only partially address this, often conflating visual quality with cognitive correctness. We introduce VWG-Bench (Video World Generalist Benchmark), a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks. To enable precise diagnosis, we design a three-level VLM-as-Judge protocol that independently assesses video-level fluency, task-level rule adherence, and sample-level goal realization. Evaluations of leading models reveal a striking gap: while models achieve strong rendering scores, they consistently fail on logic-heavy and rule-constrained tasks. To address this, we propose Vid-PRE (Video Prompt Reasoner and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM. Trained via reinforcement learning with purely text-based rewards, Vid-PRE produces concise, constraint-aware prompts without the instability of video-level reward signals. Experiments show that Vid-PRE yields substantial reasoning improvements across multiple generators without architectural modifications. Together, VWG-Bench and Vid-PRE offer a rigorous diagnostic lens and a scalable path toward true think-with-video capabilities. All data and code are publicly available at https://huggingface.co/datasets/KlingTeam/VWG-Bench.

arXiv:2008.09228공식 코드 · Papers with Code

AWNet: Attentive Wavelet Network for Image ISP

As the revolutionary improvement being made on the performance of smartphones over the last decade, mobile photography becomes one of the most common practices among the majority of smartphone users. However, due to the limited size of camera sensors on phone, the photographed image is still visually distinct to the one taken by the digital single-lens reflex (DSLR) camera. To narrow this performance gap, one is to redesign the camera image signal processor (ISP) to improve the image quality. Owing to the rapid rise of deep learning, recent works resort to the deep convolutional neural network (CNN) to develop a sophisticated data-driven ISP that directly maps the phone-captured image to the DSLR-captured one. In this paper, we introduce a novel network that utilizes the attention mechanism and wavelet transform, dubbed AWNet, to tackle this learnable image ISP problem. By adding the wavelet transform, our proposed method enables us to restore favorable image details from RAW information and achieve a larger receptive field while remaining high efficiency in terms of computational cost. The global context block is adopted in our method to learn the non-local color mapping for the generation of appealing RGB images. More importantly, this block alleviates the influence of image misalignment occurred on the provided dataset. Experimental results indicate the advances of our design in both qualitative and quantitative measurements. The code is available publically.

arXiv:2012.03408공식 코드 · Papers with Code

PMP-Net: Point Cloud Completion by Learning Multi-step Point Moving Paths

The task of point cloud completion aims to predict the missing part for an incomplete 3D shape. A widely used strategy is to generate a complete point cloud from the incomplete one. However, the unordered nature of point clouds will degrade the generation of high-quality 3D shapes, as the detailed topology and structure of discrete points are hard to be captured by the generative process only using a latent code. In this paper, we address the above problem by reconsidering the completion task from a new perspective, where we formulate the prediction as a point cloud deformation process. Specifically, we design a novel neural network, named PMP-Net, to mimic the behavior of an earth mover. It moves each point of the incomplete input to complete the point cloud, where the total distance of point moving paths (PMP) should be shortest. Therefore, PMP-Net predicts a unique point moving path for each point according to the constraint of total point moving distances. As a result, the network learns a strict and unique correspondence on point-level, which can capture the detailed topology and structure relationships between the incomplete shape and the complete target, and thus improves the quality of the predicted complete shape. We conduct comprehensive experiments on Completion3D and PCN datasets, which demonstrate our advantages over the state-of-the-art point cloud completion methods.

arXiv:2103.04354

Feedback Refined Local-Global Network for Super-Resolution of Hyperspectral Imagery

With the development of deep learning technology, multi-spectral image super-resolution methods based on convolutional neural network have recently achieved great progress. However, the single hyperspectral image super-resolution remains a challenging problem due to the high-dimensional and complex spectral characteristics of hyperspectral data, which make it difficult to simultaneously capture spatial and spectral information. To deal with this issue, we propose a novel Feedback Refined Local-Global Network (FRLGN) for the super-resolution of hyperspectral image. To be specific, we develop a new Feedback Structure and a Local-Global Spectral Block to alleviate the difficulty in spatial and spectral feature extraction. The Feedback Structure can transfer the high-level information to guide the generation process of low-level feature, which is achieved by a recurrent structure with finite unfoldings. Furthermore, in order to effectively use the high-level information passed back, a Local-Global Spectral Block is constructed to handle the feedback connections. The Local-Global Spectral Block utilizes the feedback high-level information to correct the low-level feature from local spectral bands and generates powerful high-level representations among global spectral bands. By incorporating the Feedback Structure and Local-Global Spectral Block, the FRLGN can fully exploit spatial-spectral correlations among spectral bands and gradually reconstruct high-resolution hyperspectral images. The source code of FRLGN is available at https://github.com/tangzhenjie/FRLGN.

arXiv:2107.11252공식 코드 · Papers with Code

Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation

Language instruction plays an essential role in the natural language grounded navigation tasks. However, navigators trained with limited human-annotated instructions may have difficulties in accurately capturing key information from the complicated instruction at different timesteps, leading to poor navigation performance. In this paper, we exploit to train a more robust navigator which is capable of dynamically extracting crucial factors from the long instruction, by using an adversarial attacking paradigm. Specifically, we propose a Dynamic Reinforced Instruction Attacker (DR-Attacker), which learns to mislead the navigator to move to the wrong target by destroying the most instructive information in instructions at different timesteps. By formulating the perturbation generation as a Markov Decision Process, DR-Attacker is optimized by the reinforcement learning algorithm to generate perturbed instructions sequentially during the navigation, according to a learnable attack score. Then, the perturbed instructions, which serve as hard samples, are used for improving the robustness of the navigator with an effective adversarial training strategy and an auxiliary self-supervised reasoning task. Experimental results on both Vision-and-Language Navigation (VLN) and Navigation from Dialog History (NDH) tasks show the superiority of our proposed method over state-of-the-art methods. Moreover, the visualization analysis shows the effectiveness of the proposed DR-Attacker, which can successfully attack crucial information in the instructions at different timesteps. Code is available at https://github.com/expectorlin/DR-Attacker.

arXiv:2108.09022공식 코드 · Papers with Code

Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images

We present a method for creating 3D indoor scenes with a generative model learned from a collection of semantic-segmented depth images captured from different unknown scenes. Given a room with a specified size, our method automatically generates 3D objects in a room from a randomly sampled latent code. Different from existing methods that represent an indoor scene with the type, location, and other properties of objects in the room and learn the scene layout from a collection of complete 3D indoor scenes, our method models each indoor scene as a 3D semantic scene volume and learns a volumetric generative adversarial network (GAN) from a collection of 2.5D partial observations of 3D scenes. To this end, we apply a differentiable projection layer to project the generated 3D semantic scene volumes into semantic-segmented depth images and design a new multiple-view discriminator for learning the complete 3D scene volume from 2.5D semantic-segmented depth images. Compared to existing methods, our method not only efficiently reduces the workload of modeling and acquiring 3D scenes for training, but also produces better object shapes and their detailed layouts in the scene. We evaluate our method with different indoor scene datasets and demonstrate the advantages of our method. We also extend our method for generating 3D indoor scenes from semantic-segmented depth images inferred from RGB images of real scenes.

arXiv:2306.03424공식 코드 · Papers with Code

GCD-DDPM: A Generative Change Detection Model Based on Difference-Feature Guided DDPM

Deep learning (DL)-based methods have recently shown great promise in bitemporal change detection (CD). Existing discriminative methods based on Convolutional Neural Networks (CNNs) and Transformers rely on discriminative representation learning for change recognition while struggling with exploring local and long-range contextual dependencies. As a result, it is still challenging to obtain fine-grained and robust CD maps in diverse ground scenes. To cope with this challenge, this work proposes a generative change detection model called GCD-DDPM to directly generate CD maps by exploiting the Denoising Diffusion Probabilistic Model (DDPM), instead of classifying each pixel into changed or unchanged categories. Furthermore, the Difference Conditional Encoder (DCE), is designed to guide the generation of CD maps by exploiting multi-level difference features. Leveraging the variational inference (VI) procedure, GCD-DDPM can adaptively re-calibrate the CD results through an iterative inference process, while accurately distinguishing subtle and irregular changes in diverse scenes. Finally, a Noise Suppression-based Semantic Enhancer (NSSE) is specifically designed to mitigate noise in the current step's change-aware feature representations from the CD Encoder. This refinement, serving as an attention map, can guide subsequent iterations while enhancing CD accuracy. Extensive experiments on four high-resolution CD datasets confirm the superior performance of the proposed GCD-DDPM. The code for this work will be available at https://github.com/udrs/GCD.

arXiv:2306.15551

MyCrunchGPT: A chatGPT assisted framework for scientific machine learning

Scientific Machine Learning (SciML) has advanced recently across many different areas in computational science and engineering. The objective is to integrate data and physics seamlessly without the need of employing elaborate and computationally taxing data assimilation schemes. However, preprocessing, problem formulation, code generation, postprocessing and analysis are still time consuming and may prevent SciML from wide applicability in industrial applications and in digital twin frameworks. Here, we integrate the various stages of SciML under the umbrella of ChatGPT, to formulate MyCrunchGPT, which plays the role of a conductor orchestrating the entire workflow of SciML based on simple prompts by the user. Specifically, we present two examples that demonstrate the potential use of MyCrunchGPT in optimizing airfoils in aerodynamics, and in obtaining flow fields in various geometries in interactive mode, with emphasis on the validation stage. To demonstrate the flow of the MyCrunchGPT, and create an infrastructure that can facilitate a broader vision, we built a webapp based guided user interface, that includes options for a comprehensive summary report. The overall objective is to extend MyCrunchGPT to handle diverse problems in computational mechanics, design, optimization and controls, and general scientific computing tasks involved in SciML, hence using it as a research assistant tool but also as an educational tool. While here the examples focus in fluid mechanics, future versions will target solid mechanics and materials science, geophysics, systems biology and bioinformatics.

arXiv:2307.10236

Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models

The recent performance leap of Large Language Models (LLMs) opens up new opportunities across numerous industrial applications and domains. However, erroneous generations, such as false predictions, misinformation, and hallucination made by LLMs, have also raised severe concerns for the trustworthiness of LLMs', especially in safety-, security- and reliability-sensitive scenarios, potentially hindering real-world adoptions. While uncertainty estimation has shown its potential for interpreting the prediction risks made by general machine learning (ML) models, little is known about whether and to what extent it can help explore an LLM's capabilities and counteract its undesired behavior. To bridge the gap, in this paper, we initiate an exploratory study on the risk assessment of LLMs from the lens of uncertainty. In particular, we experiment with twelve uncertainty estimation methods and four LLMs on four prominent natural language processing (NLP) tasks to investigate to what extent uncertainty estimation techniques could help characterize the prediction risks of LLMs. Our findings validate the effectiveness of uncertainty estimation for revealing LLMs' uncertain/non-factual predictions. In addition to general NLP tasks, we extensively conduct experiments with four LLMs for code generation on two datasets. We find that uncertainty estimation can potentially uncover buggy programs generated by LLMs. Insights from our study shed light on future design and development for reliable LLMs, facilitating further research toward enhancing the trustworthiness of LLMs.

arXiv:2309.16211공식 코드 · Papers with Code

VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models

The role of data in building AI systems has recently been emphasized by the emerging concept of data-centric AI. Unfortunately, in the real-world, datasets may contain dirty samples, such as poisoned samples from backdoor attack, noisy labels in crowdsourcing, and even hybrids of them. The presence of such dirty samples makes the DNNs vunerable and unreliable.Hence, it is critical to detect dirty samples to improve the quality and realiability of dataset. Existing detectors only focus on detecting poisoned samples or noisy labels, that are often prone to weak generalization when dealing with dirty samples from other domains.In this paper, we find a commonality of various dirty samples is visual-linguistic inconsistency between images and associated labels. To capture the semantic inconsistency between modalities, we propose versatile data cleanser (VDC) leveraging the surpassing capabilities of multimodal large language models (MLLM) in cross-modal alignment and reasoning.It consists of three consecutive modules: the visual question generation module to generate insightful questions about the image; the visual question answering module to acquire the semantics of the visual content by answering the questions with MLLM; followed by the visual answer evaluation module to evaluate the inconsistency.Extensive experiments demonstrate its superior performance and generalization to various categories and types of dirty samples. The code is available at \url{https://github.com/zihao-ai/vdc}.

arXiv:2402.04050공식 코드 · Papers with Code

Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models

With the emergence of pretrained vision-language models (VLMs), considerable efforts have been devoted to fine-tuning them for downstream tasks. Despite the progress made in designing efficient fine-tuning methods, such methods require access to the model's parameters, which can be challenging as model owners often opt to provide their models as a black box to safeguard model ownership. This paper proposes a \textbf{C}ollabo\textbf{ra}tive \textbf{F}ine-\textbf{T}uning (\textbf{CraFT}) approach for fine-tuning black-box VLMs to downstream tasks, where one only has access to the input prompts and the output predictions of the model. CraFT comprises two modules, a prompt generation module for learning text prompts and a prediction refinement module for enhancing output predictions in residual style. Additionally, we introduce an auxiliary prediction-consistent loss to promote consistent optimization across these modules. These modules are optimized by a novel collaborative training algorithm. Extensive experiments on few-shot classification over 15 datasets demonstrate the superiority of CraFT. The results show that CraFT achieves a decent gain of about 12\% with 16-shot datasets and only 8,000 queries. Moreover, CraFT trains faster and uses only about 1/80 of the memory footprint for deployment, while sacrificing only 1.62\% compared to the white-box method. Our code is publicly available at https://github.com/mrflogs/CraFT .

arXiv:2403.12003공식 코드 · Papers with Code

GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learning

Self-supervised learning has achieved remarkable success in acquiring high-quality representations from unlabeled data. The widely adopted contrastive learning framework aims to learn invariant representations by minimizing the distance between positive views originating from the same image. However, existing techniques to construct positive views highly rely on manual transformations, resulting in limited diversity and potentially false positive pairs. To tackle these challenges, we present GenView, a controllable framework that augments the diversity of positive views leveraging the power of pretrained generative models while preserving semantics. We develop an adaptive view generation method that dynamically adjusts the noise level in sampling to ensure the preservation of essential semantic meaning while introducing variability. Additionally, we introduce a quality-driven contrastive loss, which assesses the quality of positive pairs by considering both foreground similarity and background diversity. This loss prioritizes the high-quality positive pairs we construct while reducing the influence of low-quality pairs, thereby mitigating potential semantic inconsistencies introduced by generative models and aggressive data augmentation. Thanks to the improved positive view quality and the quality-driven contrastive loss, GenView significantly improves self-supervised learning across various tasks. For instance, GenView improves MoCov2 performance by 2.5%/2.2% on ImageNet linear/semi-supervised classification. Moreover, GenView even performs much better than naively augmenting the ImageNet dataset with Laion400M or ImageNet21K. Code: https://github.com/xiaojieli0903/genview.

arXiv:2404.08917공식 코드 · Papers with Code

MAProtoNet: A Multi-scale Attentive Interpretable Prototypical Part Network for 3D Magnetic Resonance Imaging Brain Tumor Classification

Automated diagnosis with artificial intelligence has emerged as a promising area in the realm of medical imaging, while the interpretability of the introduced deep neural networks still remains an urgent concern. Although contemporary works, such as XProtoNet and MProtoNet, has sought to design interpretable prediction models for the issue, the localization precision of their resulting attribution maps can be further improved. To this end, we propose a Multi-scale Attentive Prototypical part Network, termed MAProtoNet, to provide more precise maps for attribution. Specifically, we introduce a concise multi-scale module to merge attentive features from quadruplet attention layers, and produces attribution maps. The proposed quadruplet attention layers can enhance the existing online class activation mapping loss via capturing interactions between the spatial and channel dimension, while the multi-scale module then fuses both fine-grained and coarse-grained information for precise maps generation. We also apply a novel multi-scale mapping loss for supervision on the proposed multi-scale module. Compared to existing interpretable prototypical part networks in medical imaging, MAProtoNet can achieve state-of-the-art performance in localization on brain tumor segmentation (BraTS) datasets, resulting in approximately 4% overall improvement on activation precision score (with a best score of 85.8%), without using additional annotated labels of segmentation. Our code will be released in https://github.com/TUAT-Novice/maprotonet.

arXiv:2404.10966공식 코드 · Papers with Code

Domain-Specific Block Selection and Paired-View Pseudo-Labeling for Online Test-Time Adaptation

Test-time adaptation (TTA) aims to adapt a pre-trained model to a new test domain without access to source data after deployment. Existing approaches typically rely on self-training with pseudo-labels since ground-truth cannot be obtained from test data. Although the quality of pseudo labels is important for stable and accurate long-term adaptation, it has not been previously addressed. In this work, we propose DPLOT, a simple yet effective TTA framework that consists of two components: (1) domain-specific block selection and (2) pseudo-label generation using paired-view images. Specifically, we select blocks that involve domain-specific feature extraction and train these blocks by entropy minimization. After blocks are adjusted for current test domain, we generate pseudo-labels by averaging given test images and corresponding flipped counterparts. By simply using flip augmentation, we prevent a decrease in the quality of the pseudo-labels, which can be caused by the domain gap resulting from strong augmentation. Our experimental results demonstrate that DPLOT outperforms previous TTA methods in CIFAR10-C, CIFAR100-C, and ImageNet-C benchmarks, reducing error by up to 5.4%, 9.1%, and 2.9%, respectively. Also, we provide an extensive analysis to demonstrate effectiveness of our framework. Code is available at https://github.com/gist-ailab/domain-specific-block-selection-and-paired-view-pseudo-labeling-for-online-TTA.

arXiv:2405.14377공식 코드 · Papers with Code

CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization

Training large AI models such as LLMs and DLRMs costs massive GPUs and computing time. The high training cost has become only affordable to big tech companies, meanwhile also causing increasing concerns about the environmental impact. This paper presents CoMERA, a Computing- and Memory-Efficient training method via Rank-Adaptive tensor optimization. CoMERA achieves rank-adaptive tensor-compressed (pre)-training via a multi-objective optimization formulation and improves the training to provide both a high compression ratio and excellent accuracy in the training process. Our optimized numerical computation (e.g., optimized tensorized embedding and tensor-network contractions) and GPU implementation eliminate part of the run-time overhead in the tensorized training on GPU. This leads to, for the first time, $2-3\times$ speedup per training epoch compared with standard training. CoMERA also outperforms the recent GaLore in terms of both memory and computing efficiency. Specifically, CoMERA is $2\times$ faster per training epoch and $9\times$ more memory-efficient than GaLore on a tested six-encoder transformer with single-batch training. Our method also shows $\sim 2\times$ speedup than standard pre-training on a BERT-like code-generation LLM while achieving $4.23\times$ compression ratio in pre-training. With further HPC optimization, CoMERA may reduce the pre-training cost of many other LLMs. An implementation of CoMERA is available at https://github.com/ziyangjoy/CoMERA.

arXiv:2406.13621공식 코드 · Papers with Code

Improving Visual Commonsense in Language Models via Multiple Image Generation

Commonsense reasoning is fundamentally based on multimodal knowledge. However, existing large language models (LLMs) are primarily trained using textual data only, limiting their ability to incorporate essential visual information. In contrast, Visual Language Models, which excel at visually-oriented tasks, often fail at non-visual tasks such as basic commonsense reasoning. This divergence highlights a critical challenge - the integration of robust visual understanding with foundational text-based language reasoning. To this end, we introduce a method aimed at enhancing LLMs' visual commonsense. Specifically, our method generates multiple images based on the input text prompt and integrates these into the model's decision-making process by mixing their prediction probabilities. To facilitate multimodal grounded language modeling, we employ a late-fusion layer that combines the projected visual features with the output of a pre-trained LLM conditioned on text only. This late-fusion layer enables predictions based on comprehensive image-text knowledge as well as text only when this is required. We evaluate our approach using several visual commonsense reasoning tasks together with traditional NLP tasks, including common sense reasoning and reading comprehension. Our experimental results demonstrate significant superiority over existing baselines. When applied to recent state-of-the-art LLMs (e.g., Llama3), we observe improvements not only in visual common sense but also in traditional NLP benchmarks. Code and models are available under https://github.com/guyyariv/vLMIG.