PyTorchKR Papers

“Code Generation”

총 10,163편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2605.30931v4cs.CL

MineExplorer: 마인크래프트에서 MLLM 에이전트의 오픈월드 탐험 능력 평가

MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft

멀티모달 대규모 언어 모델(MLLM)은 인지, 추론, 행동 생성에서 강력한 능력을 보여 왔습니다. 그러나 동적인 오픈월드에서 탐험(exploration)을 지속하는 능력은 여전히 불분명합니다. 기존의 체화(embodied) 및 게임 기반 벤치마크는 상호작용을 단기(short-horizon) 작업으로 압축하거나, 성공 여부를 도메인 특화 게임 메커니즘과 결부시키는 경우가 많습니다. 본 논문에서는 마인크래프트(Minecraft)에서 MLLM 에이전트의 오픈월드 탐험 능력을 평가하기 위한 MineExplorer 벤치마크를 소개합니다. 먼저 일반적인 오픈월드 추론을 더 잘 반영하기 위해, 해결 과정이 마인크래프트 특화 지식에 크게 의존하는 원자적(atomic) 작업을 걸러냅니다. 그런 다음 ReAct 스타일의 역량 정식화를 중심으로 벤치마크를 구성하고, 원자적 작업들을 암묵적 멀티홉(multi-hop) 작업으로 조합합니다. 신뢰할 수 있는 인스턴스를 추가로 구축하기 위해, MineExplorer는 작업 그래프, 샌드박스 장면, 규칙 기반 마일스톤 평가기를 함께 설계하는 다중 에이전트 합성 워크플로우를 사용합니다. 인간 평가 결과, 다중 에이전트 합성 워크플로우는 단일 에이전트 기준선(baseline)보다 유의미하게 더 신뢰할 수 있는 인스턴스를 생성하는 것으로 나타났습니다. 최신 MLLM 에이전트를 대상으로 한 실험에서, 강력한 모델은 많은 단일 홉 작업을 처리할 수 있지만 숨겨진 전제 조건을 더 긴 궤적에 걸쳐 조율해야 할 때는 성능이 급격히 저하되어, 오픈월드 탐험이 여전히 어려운 과제임을 확인했습니다. 추가 분석 결과, 작업 난이도는 에이전트의 작업 완료 정도와 함께 변하며, 더 큰 모델이나 사고(thinking) 모드가 항상 더 나은 성능으로 이어지는 것은 아님을 발견했습니다. 코드와 데이터셋은 https://github.com/meituan-longcat/MineExplorer 에서 확인할 수 있습니다.

arXiv:2602.24210v4cs.CL

누출되는 생각에서 비공개 추론으로: 대규모 추론 모델(LRM)이 스스로에게 말하는 내용 통제하기

From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves

대규모 추론 모델(LRM)은 흔히 민감한 정보를 포함하는 추론 흔적(RT)을 생성합니다. 이러한 누출되는 생각(leaky thoughts)은 제어하기 어렵고 명시적인 프라이버시 지침을 자주 위반합니다. RT는 프롬프트 주입 공격을 통해 노출될 수 있으므로, 이는 사용자에게 직접적인 프라이버시 위험이 됩니다. 본 연구에서는 이를 제어 가능성(controllability) 문제로 접근합니다. 즉, 프라이버시 지침 자체도 지시사항이므로, RT 내에서 인스트럭션 수행(IF) 능력을 향상시키는 것이 프라이버시 유출을 줄이는 직접적인 경로를 제공합니다. 이를 위해 모델이 추론 과정 전반에 걸쳐 일반적인 지시사항을 따르도록 학습시키는 SFT 데이터셋을 소개하고, 각 구성 요소의 IF를 극대화하기 위해 별도의 LoRA 어댑터를 사용하여 RT와 답변 생성을 분리하는 단순한 디코딩 전략인 단계별 디코딩(Staged Decoding)을 제안합니다. 두 개 제품군(1.7B-14B 매개변수)의 6개 모델을 대상으로 두 개의 IF 벤치마크와 두 개의 프라이버시 벤치마크에서 본 접근법을 평가합니다. 추론 성능과 IF 간의 트레이드오프로 인해 작업 효용성(task utility) 저하가 수반될 수 있으나, 본 연구의 방법은 IF에서 최대 20.9점, 프라이버시 벤치마크에서 51.9퍼센트포인트의 향상을 보이며 상당한 개선을 달성합니다. 연구 결과는 LRM의 IF 능력을 개선하는 것이 프라이버시를 크게 강화할 수 있음을 보여주며, 향후 프라이버시를 고려하는 LRM을 위한 유망한 방향을 제시합니다. 코드는 https://github.com/UKPLab/arxiv2026-controllable-reasoning-models 에서 확인할 수 있습니다.

arXiv:2610.04299v2cs.LG

질문에 질문 던지기: 추론 모델의 자기 진화 지속시키기

Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

자기 진화(self-evolving) 추론 모델은 스스로 생성한 질문으로부터 학습하지만, 반복적인 자기 훈련(self-training)은 성능 붕괴로 이어질 수 있습니다. 본 논문에서는 연속적인 라운드를 거치며 성능이 저하되는 이유와 자기 진화를 지속시키는 방법을 연구합니다. 분석 결과, 자체 생성된 질문에서 반복적으로 나타나는 두 가지 품질 문제를 확인했습니다. 바로 유효하지 않은 질문과 동일한 수학적 질문의 반복적인 변형입니다. 첫째, 라운드가 진행될수록 유효하지 않은 질문이 더 만연해지며, 답변 일관성 기반 필터링은 학습 데이터 내 이러한 질문의 비율을 더욱 증가시킵니다. 둘째, 어휘 유사도에 기반한 기존의 질문 다양성 제어 방식은 다른 방식으로 표현된 수학적으로 동등한 질문을 놓칠 수 있어, 후반부 학습 라운드에서 질문 다양성 붕괴를 초래합니다. 이러한 발견을 바탕으로, 질문 유효성 및 참신성 피드백을 활용하여 자기 진화를 유도하는 R-Quest를 소개합니다. 먼저 해결자(solver)가 유효하지 않은 질문을 인식하고 거부하도록 학습시킨 후, 그 판단 결과를 활용해 질문자(questioner)의 보상을 유도하고 해결자의 학습 데이터를 필터링합니다. 질문 반복을 방지하기 위해, 동결된 베이스 모델을 사용하여 샘플링된 질문 쌍을 비교하고 참신성 피드백을 제공합니다. 실험적으로, 제안하는 방법은 두 가지 모델 계열에 걸쳐 수학적 추론, 일반 영역 추론 및 코드 생성 분야의 12개 벤치마크에서 일관되게 가장 높은 평균 성능을 달성합니다. 또한 R-Quest는 10회의 자기 진화 라운드 동안 안정적인 성능 향상을 유지하여 최종 라운드에서 정점을 찍었으며, R-Zero 대비 17.32점 높은 성능을 기록했습니다.

arXiv:2001.02314공식 코드 · Papers with Code

Bridging Knowledge Graphs to Generate Scene Graphs

Scene graphs are powerful representations that parse images into their abstract semantic elements, i.e., objects and their interactions, which facilitates visual comprehension and explainable reasoning. On the other hand, commonsense knowledge graphs are rich repositories that encode how the world is structured, and how general concepts interact. In this paper, we present a unified formulation of these two constructs, where a scene graph is seen as an image-conditioned instantiation of a commonsense knowledge graph. Based on this new perspective, we re-formulate scene graph generation as the inference of a bridge between the scene and commonsense graphs, where each entity or predicate instance in the scene graph has to be linked to its corresponding entity or predicate class in the commonsense graph. To this end, we propose a novel graph-based neural network that iteratively propagates information between the two graphs, as well as within each of them, while gradually refining their bridge in each iteration. Our Graph Bridging Network, GB-Net, successively infers edges and nodes, allowing to simultaneously exploit and refine the rich, heterogeneous structure of the interconnected scene and commonsense graphs. Through extensive experimentation, we showcase the superior accuracy of GB-Net compared to the most recent methods, resulting in a new state of the art. We publicly release the source code of our method.

arXiv:2007.09609공식 코드 · Papers with Code

Symbiotic Adversarial Learning for Attribute-based Person Search

Attribute-based person search is in significant demand for applications where no detected query images are available, such as identifying a criminal from witness. However, the task itself is quite challenging because there is a huge modality gap between images and physical descriptions of attributes. Often, there may also be a large number of unseen categories (attribute combinations). The current state-of-the-art methods either focus on learning better cross-modal embeddings by mining only seen data, or they explicitly use generative adversarial networks (GANs) to synthesize unseen features. The former tends to produce poor embeddings due to insufficient data, while the latter does not preserve intra-class compactness during generation. In this paper, we present a symbiotic adversarial learning framework, called SAL.Two GANs sit at the base of the framework in a symbiotic learning scheme: one synthesizes features of unseen classes/categories, while the other optimizes the embedding and performs the cross-modal alignment on the common embedding space .Specifically, two different types of generative adversarial networks learn collaboratively throughout the training process and the interactions between the two mutually benefit each other. Extensive evaluations show SAL's superiority over nine state-of-the-art methods with two challenging pedestrian benchmarks, PETA and Market-1501. The code is publicly available at: https://github.com/ycao5602/SAL .

arXiv:2009.00149공식 코드 · Papers with Code

GIF: Generative Interpretable Faces

Photo-realistic visualization and animation of expressive human faces have been a long standing challenge. 3D face modeling methods provide parametric control but generates unrealistic images, on the other hand, generative 2D models like GANs (Generative Adversarial Networks) output photo-realistic face images, but lack explicit control. Recent methods gain partial control, either by attempting to disentangle different factors in an unsupervised manner, or by adding control post hoc to a pre-trained model. Unconditional GANs, however, may entangle factors that are hard to undo later. We condition our generative model on pre-defined control parameters to encourage disentanglement in the generation process. Specifically, we condition StyleGAN2 on FLAME, a generative 3D face model. While conditioning on FLAME parameters yields unsatisfactory results, we find that conditioning on rendered FLAME geometry and photometric details works well. This gives us a generative 2D face model named GIF (Generative Interpretable Faces) that offers FLAME's parametric control. Here, interpretable refers to the semantic meaning of different parameters. Given FLAME parameters for shape, pose, expressions, parameters for appearance, lighting, and an additional style vector, GIF outputs photo-realistic face images. We perform an AMT based perceptual study to quantitatively and qualitatively evaluate how well GIF follows its conditioning. The code, data, and trained model are publicly available for research purposes at http://gif.is.tue.mpg.de.

arXiv:2103.13859공식 코드 · Papers with Code

Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional Networks

In this paper, we propose an efficient saliency map generation method, called Group score-weighted Class Activation Mapping (Group-CAM), which adopts the "split-transform-merge" strategy to generate saliency maps. Specifically, for an input image, the class activations are firstly split into groups. In each group, the sub-activations are summed and de-noised as an initial mask. After that, the initial masks are transformed with meaningful perturbations and then applied to preserve sub-pixels of the input (i.e., masked inputs), which are then fed into the network to calculate the confidence scores. Finally, the initial masks are weighted summed to form the final saliency map, where the weights are confidence scores produced by the masked inputs. Group-CAM is efficient yet effective, which only requires dozens of queries to the network while producing target-related saliency maps. As a result, Group-CAM can be served as an effective data augment trick for fine-tuning the networks. We comprehensively evaluate the performance of Group-CAM on common-used benchmarks, including deletion and insertion tests on ImageNet-1k, and pointing game tests on COCO2017. Extensive experimental results demonstrate that Group-CAM achieves better visual performance than the current state-of-the-art explanation approaches. The code is available at https://github.com/wofmanaf/Group-CAM.

arXiv:2104.06393

Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding

Spoken language understanding (SLU) system usually consists of various pipeline components, where each component heavily relies on the results of its upstream ones. For example, Intent detection (ID), and slot filling (SF) require its upstream automatic speech recognition (ASR) to transform the voice into text. In this case, the upstream perturbations, e.g. ASR errors, environmental noise and careless user speaking, will propagate to the ID and SF models, thus deteriorating the system performance. Therefore, the well-performing SF and ID models are expected to be noise resistant to some extent. However, existing models are trained on clean data, which causes a \textit{gap between clean data training and real-world inference.} To bridge the gap, we propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Meanwhile, we design a denoising generation model to reduce the impact of the low-quality samples. Experiments on the widely-used dataset, i.e. Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment. The source code will be released.

arXiv:2106.07631공식 코드 · Papers with Code

Improved Transformer for High-Resolution GANs

Attention-based models, exemplified by the Transformer, can effectively model long range dependency, but suffer from the quadratic complexity of self-attention operation, making them difficult to be adopted for high-resolution image generation based on Generative Adversarial Networks (GANs). In this paper, we introduce two key ingredients to Transformer to address this challenge. First, in low-resolution stages of the generative process, standard global self-attention is replaced with the proposed multi-axis blocked self-attention which allows efficient mixing of local and global attention. Second, in high-resolution stages, we drop self-attention while only keeping multi-layer perceptrons reminiscent of the implicit neural function. To further improve the performance, we introduce an additional self-modulation component based on cross-attention. The resulting model, denoted as HiT, has a nearly linear computational complexity with respect to the image size and thus directly scales to synthesizing high definition images. We show in the experiments that the proposed HiT achieves state-of-the-art FID scores of 30.83 and 2.95 on unconditional ImageNet $128 \times 128$ and FFHQ $256 \times 256$, respectively, with a reasonable throughput. We believe the proposed HiT is an important milestone for generators in GANs which are completely free of convolutions. Our code is made publicly available at https://github.com/google-research/hit-gan

arXiv:2202.09507공식 코드 · Papers with Code

PMP-Net++: Point Cloud Completion by Transformer-Enhanced Multi-step Point Moving Paths

Point cloud completion concerns to predict missing part for incomplete 3D shapes. A common strategy is to generate complete shape according to incomplete input. However, unordered nature of point clouds will degrade generation of high-quality 3D shapes, as detailed topology and structure of unordered points are hard to be captured during the generative process using an extracted latent code. We address this problem by formulating completion as point cloud deformation process. Specifically, we design a novel neural network, named PMP-Net++, to mimic behavior of an earth mover. It moves each point of incomplete input to obtain a complete point cloud, where total distance of point moving paths (PMPs) should be the shortest. Therefore, PMP-Net++ predicts unique PMP for each point according to constraint of point moving distances. The network learns a strict and unique correspondence on point-level, and thus improves quality of predicted complete shape. Moreover, since moving points heavily relies on per-point features learned by network, we further introduce a transformer-enhanced representation learning network, which significantly improves completion performance of PMP-Net++. We conduct comprehensive experiments in shape completion, and further explore application on point cloud up-sampling, which demonstrate non-trivial improvement of PMP-Net++ over state-of-the-art point cloud completion/up-sampling methods.

arXiv:2205.06207공식 코드 · Papers with Code

CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision

Scientific extreme summarization (TLDR) aims to form ultra-short summaries of scientific papers. Previous efforts on curating scientific TLDR datasets failed to scale up due to the heavy human annotation and domain expertise required. In this paper, we propose a simple yet effective approach to automatically extracting TLDR summaries for scientific papers from their citation texts. Based on the proposed approach, we create a new benchmark CiteSum without human annotation, which is around 30 times larger than the previous human-curated dataset SciTLDR. We conduct a comprehensive analysis of CiteSum, examining its data characteristics and establishing strong baselines. We further demonstrate the usefulness of CiteSum by adapting models pre-trained on CiteSum (named CITES) to new tasks and domains with limited supervision. For scientific extreme summarization, CITES outperforms most fully-supervised methods on SciTLDR without any fine-tuning and obtains state-of-the-art results with only 128 examples. For news extreme summarization, CITES achieves significant gains on XSum over its base model (not pre-trained on CiteSum), e.g., +7.2 ROUGE-1 zero-shot performance and state-of-the-art few-shot performance. For news headline generation, CITES performs the best among unsupervised and zero-shot methods on Gigaword. Our dataset and code can be found at https://github.com/morningmoni/CiteSum.

arXiv:2210.06126공식 코드 · Papers with Code

Regularized Graph Structure Learning with Semantic Knowledge for Multi-variates Time-Series Forecasting

Multivariate time-series forecasting is a critical task for many applications, and graph time-series network is widely studied due to its capability to capture the spatial-temporal correlation simultaneously. However, most existing works focus more on learning with the explicit prior graph structure, while ignoring potential information from the implicit graph structure, yielding incomplete structure modeling. Some recent works attempt to learn the intrinsic or implicit graph structure directly while lacking a way to combine explicit prior structure with implicit structure together. In this paper, we propose Regularized Graph Structure Learning (RGSL) model to incorporate both explicit prior structure and implicit structure together, and learn the forecasting deep networks along with the graph structure. RGSL consists of two innovative modules. First, we derive an implicit dense similarity matrix through node embedding, and learn the sparse graph structure using the Regularized Graph Generation (RGG) based on the Gumbel Softmax trick. Second, we propose a Laplacian Matrix Mixed-up Module (LM3) to fuse the explicit graph and implicit graph together. We conduct experiments on three real-word datasets. Results show that the proposed RGSL model outperforms existing graph forecasting algorithms with a notable margin, while learning meaningful graph structure simultaneously. Our code and models are made publicly available at https://github.com/alipay/RGSL.git.

arXiv:2211.08658공식 코드 · Papers with Code

Consistent Direct Time-of-Flight Video Depth Super-Resolution

Direct time-of-flight (dToF) sensors are promising for next-generation on-device 3D sensing. However, limited by manufacturing capabilities in a compact module, the dToF data has a low spatial resolution (e.g., $\sim 20\times30$ for iPhone dToF), and it requires a super-resolution step before being passed to downstream tasks. In this paper, we solve this super-resolution problem by fusing the low-resolution dToF data with the corresponding high-resolution RGB guidance. Unlike the conventional RGB-guided depth enhancement approaches, which perform the fusion in a per-frame manner, we propose the first multi-frame fusion scheme to mitigate the spatial ambiguity resulting from the low-resolution dToF imaging. In addition, dToF sensors provide unique depth histogram information for each local patch, and we incorporate this dToF-specific feature in our network design to further alleviate spatial ambiguity. To evaluate our models on complex dynamic indoor environments and to provide a large-scale dToF sensor dataset, we introduce DyDToF, the first synthetic RGB-dToF video dataset that features dynamic objects and a realistic dToF simulator following the physical imaging process. We believe the methods and dataset are beneficial to a broad community as dToF depth sensing is becoming mainstream on mobile devices. Our code and data are publicly available: https://github.com/facebookresearch/DVSR/

arXiv:2307.10554공식 코드 · Papers with Code

EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization

Mixed-Precision Quantization~(MQ) can achieve a competitive accuracy-complexity trade-off for models. Conventional training-based search methods require time-consuming candidate training to search optimized per-layer bit-width configurations in MQ. Recently, some training-free approaches have presented various MQ proxies and significantly improve search efficiency. However, the correlation between these proxies and quantization accuracy is poorly understood. To address the gap, we first build the MQ-Bench-101, which involves different bit configurations and quantization results. Then, we observe that the existing training-free proxies perform weak correlations on the MQ-Bench-101. To efficiently seek superior proxies, we develop an automatic search of proxies framework for MQ via evolving algorithms. In particular, we devise an elaborate search space involving the existing proxies and perform an evolution search to discover the best correlated MQ proxy. We proposed a diversity-prompting selection strategy and compatibility screening protocol to avoid premature convergence and improve search efficiency. In this way, our Evolving proxies for Mixed-precision Quantization~(EMQ) framework allows the auto-generation of proxies without heavy tuning and expert knowledge. Extensive experiments on ImageNet with various ResNet and MobileNet families demonstrate that our EMQ obtains superior performance than state-of-the-art mixed-precision methods at a significantly reduced cost. The code will be released.

arXiv:2311.06383공식 코드 · Papers with Code

Distilling Large Language Models using Skill-Occupation Graph Context for HR-Related Tasks

Numerous HR applications are centered around resumes and job descriptions. While they can benefit from advancements in NLP, particularly large language models, their real-world adoption faces challenges due to absence of comprehensive benchmarks for various HR tasks, and lack of smaller models with competitive capabilities. In this paper, we aim to bridge this gap by introducing the Resume-Job Description Benchmark (RJDB). We meticulously craft this benchmark to cater to a wide array of HR tasks, including matching and explaining resumes to job descriptions, extracting skills and experiences from resumes, and editing resumes. To create this benchmark, we propose to distill domain-specific knowledge from a large language model (LLM). We rely on a curated skill-occupation graph to ensure diversity and provide context for LLMs generation. Our benchmark includes over 50 thousand triples of job descriptions, matched resumes and unmatched resumes. Using RJDB, we train multiple smaller student models. Our experiments reveal that the student models achieve near/better performance than the teacher model (GPT-4), affirming the effectiveness of the benchmark. Additionally, we explore the utility of RJDB on out-of-distribution data for skill extraction and resume-job description matching, in zero-shot and weak supervision manner. We release our datasets and code to foster further research and industry applications.

arXiv:2401.08727공식 코드 · Papers with Code

MA2GCN: Multi Adjacency relationship Attention Graph Convolutional Networks for Traffic Prediction using Trajectory data

The problem of traffic congestion not only causes a large amount of economic losses, but also seriously endangers the urban environment. Predicting traffic congestion has important practical significance. So far, most studies have been based on historical data from sensors placed on different roads to predict future traffic flow and speed, to analyze the traffic congestion conditions of a certain road segment. However, due to the fixed position of sensors, it is difficult to mine new information. On the other hand, vehicle trajectory data is more flexible and can extract traffic information as needed. Therefore, we proposed a new traffic congestion prediction model - Multi Adjacency relationship Attention Graph Convolutional Networks(MA2GCN). This model transformed vehicle trajectory data into graph structured data in grid form, and proposed a vehicle entry and exit matrix based on the mobility between different grids. At the same time, in order to improve the performance of the model, this paper also built a new adaptive adjacency matrix generation method and adjacency matrix attention module. This model mainly used gated temporal convolution and graph convolution to extract temporal and spatial information, respectively. Compared with multiple baselines, our model achieved the best performance on Shanghai taxi GPS trajectory dataset. The code is available at https://github.com/zachysun/Taxi_Traffic_Benchmark.

arXiv:2402.12654공식 코드 · Papers with Code

OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification

There has been an increasing interest in large speech models that can perform multiple tasks in a single model. Such models usually adopt an encoder-decoder or decoder-only architecture due to their popularity and good performance in many domains. However, autoregressive models can be slower during inference compared to non-autoregressive models and also have potential risks of hallucination. Though prior studies observed promising results of non-autoregressive models for certain tasks at small scales, it remains unclear if they can be scaled to speech-to-text generation in diverse languages and tasks. Inspired by the Open Whisper-style Speech Model (OWSM) project, we propose OWSM-CTC, a novel encoder-only speech foundation model based on Connectionist Temporal Classification (CTC). It is trained on 180k hours of public audio data for multilingual automatic speech recognition (ASR), speech translation (ST), and language identification (LID). Compared to encoder-decoder OWSM, our OWSM-CTC achieves competitive results on ASR and up to 24% relative improvement on ST, while it is more robust and 3 to 4 times faster for inference. OWSM-CTC also improves the long-form ASR result with 20x speed-up. We will publicly release our code, pre-trained model, and training logs to promote open science in speech foundation models.

arXiv:2403.02818

Are Dense Labels Always Necessary for 3D Object Detection from Point Cloud?

Current state-of-the-art (SOTA) 3D object detection methods often require a large amount of 3D bounding box annotations for training. However, collecting such large-scale densely-supervised datasets is notoriously costly. To reduce the cumbersome data annotation process, we propose a novel sparsely-annotated framework, in which we just annotate one 3D object per scene. Such a sparse annotation strategy could significantly reduce the heavy annotation burden, while inexact and incomplete sparse supervision may severely deteriorate the detection performance. To address this issue, we develop the SS3D++ method that alternatively improves 3D detector training and confident fully-annotated scene generation in a unified learning scheme. Using sparse annotations as seeds, we progressively generate confident fully-annotated scenes based on designing a missing-annotated instance mining module and reliable background mining module. Our proposed method produces competitive results when compared with SOTA weakly-supervised methods using the same or even more annotation costs. Besides, compared with SOTA fully-supervised methods, we achieve on-par or even better performance on the KITTI dataset with about 5x less annotation cost, and 90% of their performance on the Waymo dataset with about 15x less annotation cost. The additional unlabeled training scenes could further boost the performance. The code will be available at https://github.com/gaocq/SS3D2.

arXiv:2404.01959공식 코드 · Papers with Code

Bi-LORA: A Vision-Language Approach for Synthetic Image Detection

Advancements in deep image synthesis techniques, such as generative adversarial networks (GANs) and diffusion models (DMs), have ushered in an era of generating highly realistic images. While this technological progress has captured significant interest, it has also raised concerns about the potential difficulty in distinguishing real images from their synthetic counterparts. This paper takes inspiration from the potent convergence capabilities between vision and language, coupled with the zero-shot nature of vision-language models (VLMs). We introduce an innovative method called Bi-LORA that leverages VLMs, combined with low-rank adaptation (LORA) tuning techniques, to enhance the precision of synthetic image detection for unseen model-generated images. The pivotal conceptual shift in our methodology revolves around reframing binary classification as an image captioning task, leveraging the distinctive capabilities of cutting-edge VLM, notably bootstrapping language image pre-training (BLIP2). Rigorous and comprehensive experiments are conducted to validate the effectiveness of our proposed approach, particularly in detecting unseen diffusion-generated images from unknown diffusion-based generative models during training, showcasing robustness to noise, and demonstrating generalization capabilities to GANs. The obtained results showcase an impressive average accuracy of 93.41% in synthetic image detection on unseen generation models. The code and models associated with this research can be publicly accessed at https://github.com/Mamadou-Keita/VLM-DETECT.

arXiv:2405.00957공식 코드 · Papers with Code

IntraMix: Intra-Class Mixup Generation for Accurate Labels and Neighbors

Graph Neural Networks (GNNs) have shown great performance in various tasks, with the core idea of learning from data labels and aggregating messages within the neighborhood of nodes. However, the common challenges in graphs are twofold: insufficient accurate (high-quality) labels and limited neighbors for nodes, resulting in weak GNNs. Existing graph augmentation methods typically address only one of these challenges, often adding training costs or relying on oversimplified or knowledge-intensive strategies, limiting their generalization. To simultaneously address both challenges faced by graphs in a generalized way, we propose an elegant method called IntraMix. Considering the incompatibility of vanilla Mixup with the complex topology of graphs, IntraMix innovatively employs Mixup among inaccurate labeled data of the same class, generating high-quality labeled data at minimal cost. Additionally, it finds data with high confidence of being clustered into the same group as the generated data to serve as their neighbors, thereby enriching the neighborhoods of graphs. IntraMix efficiently tackles both issues faced by graphs and challenges the prior notion of the limited effectiveness of Mixup in node classification. IntraMix is a theoretically grounded plug-in-play method that can be readily applied to all GNNs. Extensive experiments demonstrate the effectiveness of IntraMix across various GNNs and datasets. Our code is available at: https://github.com/Zhengsh123/IntraMix.