PyTorchKR Papers

“Code Generation”

총 10,140편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2606.24623v1cs.CL

Privacy-Preserving RAG via Multi-Agent Semantic Rewriting: Achieving Confidentiality Without Compromising Contextual Fidelity

민감한 검색-증강 생성(RAG) 배포에서 악의적 프롬프트로 인한 개인정보 유출 위험을 줄이기 위해, 프라이버시 추출·의미 분석·재구성을 담당하는 세 에이전트가 검색 문서를 의미 재작성으로 정화하는 다중 에이전트 프레임워크를 제안한다. ChatDoctor와 Wiki-PII에서 LLaMA-3-8B의 표적 공격 노출을 144건에서 1건으로 낮추면서 BLEU-1 0.122로 문맥 충실도를 유지했고, 재작성은 오프라인 전처리로 온라인 추론 지연을 추가하지 않는다.

arXiv:2602.05143v2cs.AI

CausalRAG2: Hierarchical Causal Knowledge Graph Design for RAG

Retrieval augmented generation (RAG) has enhanced large language models by enabling access to external knowledge, with graph-based RAG emerging as a powerful paradigm for structured retrieval and reasoning. However, existing graph-based methods often over-rely on entity-centric node matching and lack explicit causal modeling, leading to unfaithful or spurious answers. Prior attempts to incorporate causality are typically limited to local or single-document contexts and also suffer from information isolation that arises from modular graph structures, which hinders scalability and cross-module causal reasoning. To address these challenges, we propose CausalRAG2, a framework that rethinks knowledge organization for graph-based RAG through causal gating across hierarchical modules. CausalRAG2 explicitly models causal relationships to suppress spurious correlations while enabling scalable reasoning over large-scale knowledge graphs. We also introduce HolisQA, a benchmark for holistic comprehension beyond entity-centric matching. Extensive experiments demonstrate that CausalRAG2 consistently outperforms competitive graph-based RAG baselines across multiple datasets and evaluation metrics. Our work establishes a principled foundation for structured, scalable, and causally grounded RAG systems. Our code and HolisQA benchmark are available at https://github.com/Pwnb/CausalRAG2.

arXiv:2604.20328v2cs.CV

HyLaR: 분리된 정책 최적화를 통한 하이브리드 잠재 추론

HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization

사고의 연쇄(Chain-of-Thought, CoT) 추론은 멀티모달 대규모 언어 모델(MLLM)의 복잡한 문제 해결 능력을 크게 향상시킨다. 그러나 비전에 CoT를 적용할 때는 일반적으로 LLM 입력에 맞추기 위해 신호를 이산화하며, 이로 인해 조기 의미 붕괴가 발생하고 세밀한 세부 정보가 버려진다. 외부 도구로 이를 완화할 수 있지만, 경직된 병목을 도입하여 추론을 사전 정의된 연산에 국한시킨다. 최근 잠재 추론(latent reasoning) 패러다임은 이러한 한계를 극복하기 위해 시각 상태를 내재화하지만, 그 결과로 생기는 하이브리드 이산-연속 행동 공간의 최적화는 여전히 어렵다. 본 연구에서는 이산 텍스트 생성과 연속적인 시각 잠재 표현을 원활하게 교차 배치하는 프레임워크인 HyLaR(Hybrid Latent Reasoning)을 제안한다. 구체적으로, 초기 콜드스타트 지도 미세조정(SFT) 이후, 이러한 하이브리드 공간에서 효과적인 강화학습을 가능하게 하는 DePO(Decoupled Policy Optimization)를 도입한다. DePO는 정책 그래디언트 목적을 분해하여 텍스트 구성 요소와 잠재 구성 요소에 각각 독립적인 신뢰 영역(trust-region) 제약을 적용하고, 정확한 폐형식 von Mises-Fisher(vMF) KL 정규화를 함께 사용한다. 광범위한 실험을 통해 HyLaR이 표준 MLLM 및 최첨단 잠재 추론 접근법보다 세밀한 인지와 일반 멀티모달 이해 벤치마크에서 우수함을 입증한다. 코드는 https://github.com/EthenCheng/HyLaR 에서 이용 가능하다.

arXiv:2511.03589v3cs.CV

모든 체형을 위한 인간 메시 모델링

Human Mesh Modeling for Anny Body

매개변수 기반 인체 모델(parametric body model)은 인간 중심의 다양한 작업에 구조적 기반을 제공하지만, 기존 모델들은 종종 고비용의 3D 스캔과 독점적이며 인구통계학적으로 편향된 학습된 형상 공간(shape space)에 의존합니다. 본 논문에서는 MakeHuman 커뮤니티의 인체 측정학적(anthropometric) 지식에 기반한 단순하고 완전 미분 가능하며 스캔이 필요 없는(scan-free) 인체 모델인 Anny를 소개합니다. Anny는 표현형 파라미터(예: 성별, 연령, 신장, 체중)가 연령대(유아부터 노인까지), 체형 및 신체 비율에 걸친 광범위한 인간 형태를 포괄하는 블렌드셰이프(blendshape)를 제어하는 연속적이고 해석 가능한 형상 공간을 정의합니다. 세계보건기구(WHO)의 인구 통계로 보정된 Anny는 단일 통합 모델 내에서 현실적이고 인구통계학적 기반을 갖춘 인체 형상 변형을 제공합니다. 저자들은 Anny 인체 모델과 코드를 Apache 2.0 라이선스로 공개합니다. 개방성과 의미론적 제어 능력 덕분에 Anny는 밀리미터 단위로 정밀한 스캔 피팅, 통제된 합성 데이터 생성, 인간 메시 복원(HMR, Human Mesh Recovery)을 지원하며 3D 인간 모델링을 위한 다목적 기반으로 기능합니다. 또한 Anny로 생성된 78만 장의 사실적인 이미지 컬렉션인 Anny-One을 소개하며, 모델의 단순함에도 불구하고 Anny로 학습된 HMR 모델이 스캔 기반 인체 모델로 학습된 모델의 성능에 필적할 수 있음을 보여줍니다.

arXiv:2608.14710v1cs.CV

Path2ST: Hierarchical Cell-Tissue Grounded Cross-Modal Translation for Spatial Transcriptomics

Predicting spatial gene expression from hematoxylin and eosin (H\&E)-stained images offers a cost-effective alternative to spatial transcriptomics (ST). However, existing methods treat H\&E images as generic visual inputs and ignore their intrinsic biological hierarchy, where spatially organized cell types collectively form functional tissue microenvironments that govern local gene expression programs. To bridge this gap, we formulate H\&E-to-ST prediction as a cross-modal semantic translation task and propose Path2ST, a hierarchically grounded autoregressive framework featuring three key components: (i) a Hierarchical Cell-Tissue Conditioning mechanism that fuses explicit and implicit cellular features with tissue-level semantic representations to construct hierarchical conditioning signals; (ii) a Scale-Adaptive Autoregressive Generation process over a hierarchical semantic vocabulary, enabling coarse-to-fine, biologically consistent expression synthesis; and (iii) SpectraLoss, a full-spectrum objective that jointly enforces ordinal fidelity, models transcriptional bursts, and aligns semantic structures with cell types. Extensive experiments on three datasets demonstrate state-of-the-art performance, validating that Path2ST generates highly accurate and spatially coherent transcriptomic profiles. The related code is released at https://github.com/RuochenLiu23/Path2ST.

arXiv:2608.18662v1cs.CV

Dynamic SpectraFormer for Ultra-High-Definition Underwater Image Enhancement

Underwater images suffer from color distortion, haze, and poor visibility due to light refraction and absorption in water. These challenges significantly impact the utilization of Autonomous Underwater Vehicles (AUVs) or marine robots. Typically, color and brightness distortions manifest at lower frequencies, while edge and texture distortions are prevalent at higher frequencies. Traditional methods struggle to concurrently rectify these mixed distortions as they primarily concentrate on the spatial domain. To address these issues, we introduce the Dynamic SpectraFormer, which enhances underwater images through a frequency domain transformer. The Dynamic SpectraFormer introduces an ultra-high-resolution sparse spectrum attention module, which could capture the long-term dependency without losing the universal approximating power. Additionally, we have developed a dynamic spectrum weight generation layer that serves as an adaptive spectrum band selector, accentuating critical frequency bands and suppressing less relevant ones. Consequently, this method significantly improves underwater image quality by addressing both high- and low-frequency distortions. Our extensive ablation studies and comparative evaluations consolidate the Dynamic SpectraFormer's efficacy across multiple underwater image enhancement benchmarks. The source code is available at https://github.com/arifence2024/DynamicSpectraFormer.git.

arXiv:2608.22872v2cs.CL

더 나은 검색, 더 낮은 견고성: 멀티홉 RAG가 상류 ASR 오류를 어떻게 증폭하는가

Better Retrieval, Worse Robustness: How Multi-hop RAG Amplifies Upstream ASR Errors

음성 기반 애플리케이션은 검색 모듈 이전에 음성 질의를 음성 인식(ASR)에 통과시키므로, ASR 오류는 고정된 상류 제약으로 파이프라인에 유입된다. 본 연구는 표준 검색-증강 생성(RAG)의 두 확장인 엔티티 그래프 링킹(entity-graph linking)과 반복적 재구성(iterative reformulation)이 이러한 오류를 흡수하는지 증폭하는지 실증적으로 검증한다. 신경망 TTS로 합성한 네 가지 영어 억양을 사용해, 깨끗한 텍스트 오라클 대비 세 가지 다중 홉(multi-hop) QA 벤치마크(HotpotQA, 2WikiMultiHopQA, MuSiQue)에서 네 가지 RAG 구성을 평가한다. 구조적으로 더 풍부한 구성은 ASR 입력에서도 일반적으로 더 높은 절대 F1을 유지하지만, 두 확장 모두 오류를 증폭한다. 깨끗한 텍스트에서 WER이 가장 높은 억양까지의 F1 격차는 세 벤치마크 모두에서 단순 밀집 검색(naive dense retrieval) 대비 두 확장의 조합에서 36–67% 더 크다. 지배적 실패 양상은 하나 이상의 질의 엔티티 손상이며, 네 가지 방법 모두 2WikiMultiHopQA에서 성능 저하 사례의 87–96%를 차지한다. 두 가지 경량 표면 형태(surface-form) 완화 기법은 격차의 대부분을 그대로 남기며, 이는 하류 검색 구조가 잔여 엔티티 오류를 증폭함을 시사한다. 코드와 데이터는 https://github.com/Continuum-AI-Corp/spoken-multihop-rag 에 공개한다.

arXiv:2610.03185v1cs.AI

온-정책 증류에서의 이득과 붕괴: 강화학습 관점

Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

온-정책(on-policy) 증류(OPD)는 언어 모델 사후 학습(post-training)의 중요한 접근법으로 자리 잡았습니다. 그러나 이러한 성능 향상에도 불구하고, OPD는 지나치게 길고 반복적인 생성으로 붕괴될 수 있으며, 이러한 상반된 결과를 초래하는 기본 메커니즘은 여전히 잘 이해되지 않고 있습니다. 본 연구에서는 강화학습(RL) 관점을 통해 이러한 결과를 설명합니다. 즉, 교사(teacher) 모델은 자신이 거의 보이지 않는 행동에 대해서도 학생(student)의 행동에 암묵적으로 보상을 부여합니다. 이러한 관점에서, 본 연구의 실험은 OPD가 학생 모델의 역량을 확장하지 않고도 성능을 향상시킴을 보여줍니다. 암묵적 보상 모델이 신뢰할 수 있을 때, OPD는 정답 반응을 더 쉽게 샘플링할 수 있도록 만듭니다. 반면, 선호도가 품질과 불일치할 때는 보상 해킹(reward hacking)이 발생합니다. 교사 모델은 스스로 그러한 텍스트를 거의 생성하지 않음에도 불구하고, 암묵적 보상 모델은 지나치게 길고 반복적인 학생 롤아웃(rollout)을 증폭시킵니다. 이러한 진단을 바탕으로, 학습 중 비정상적인 반응을 마스킹하고 지도 미세조정(SFT) 초기화를 사용하는 것이 각각 이러한 붕괴를 효과적으로 완화할 수 있음을 발견했습니다. 종합하면, 이러한 발견은 OPD가 교사의 암묵적 피드백이 선호하는 학생 행동을 증폭시킨다는 점을 보여주며, 초점을 교사가 얼마나 잘 생성하는지에서 학생 롤아웃을 얼마나 신뢰성 있게 평가하는지로 전환시킵니다. 코드는 https://github.com/HancCui/opd_hacking 에서 확인할 수 있습니다.

arXiv:2306.10073

Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses

This paper studies recent developments in large language models' (LLM) abilities to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. The emergence of ChatGPT resulted in heated debates of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming classes (e.g., cheating). Recent studies show that while the technology performs surprisingly well on diverse sets of assessment instruments employed in typical programming classes the performance is usually not sufficient to pass the courses. The release of GPT-4 largely emphasized notable improvements in the capabilities related to handling assessments originally designed for human test-takers. This study is the necessary analysis in the context of this ongoing transition towards mature generative AI systems. Specifically, we report the performance of GPT-4, comparing it to the previous generations of GPT models, on three Python courses with assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Additionally, we analyze the assessments that were not handled well by GPT-4 to understand the current limitations of the model, as well as its capabilities to leverage feedback provided by an auto-grader. We found that the GPT models evolved from completely failing the typical programming class' assessments (the original GPT-3) to confidently passing the courses with no human involvement (GPT-4). While we identified certain limitations in GPT-4's handling of MCQs and coding exercises, the rate of improvement across the recent generations of GPT models strongly suggests their potential to handle almost any type of assessment widely used in higher education programming courses. These findings could be leveraged by educators and institutions to adapt the design of programming assessments as well as to fuel the necessary discussions into how programming classes should be updated to reflect the recent technological developments. This study provides evidence that programming instructors need to prepare for a world in which there is an easy-to-use widely accessible technology that can be utilized by learners to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments.

arXiv:2107.00471공식 코드 · Papers with Code

SinGAN-Seg: Synthetic training data generation for medical image segmentation

Analyzing medical data to find abnormalities is a time-consuming and costly task, particularly for rare abnormalities, requiring tremendous efforts from medical experts. Artificial intelligence has become a popular tool for the automatic processing of medical data, acting as a supportive tool for doctors. However, the machine learning models used to build these tools are highly dependent on the data used to train them. Large amounts of data can be difficult to obtain in medicine due to privacy, expensive and time-consuming annotations, and a general lack of data samples for infrequent lesions. Here, we present a novel synthetic data generation pipeline, called SinGAN-Seg, to produce synthetic medical images with corresponding masks using a single training image. Our method is different from the traditional GANs because our model needs only a single image and the corresponding ground truth to train. Our method produces alternative artificial segmentation datasets with ground truth masks when real datasets are not allowed to share. The pipeline is evaluated using qualitative and quantitative comparisons between real and synthetic data to show that the style transfer technique used in our pipeline significantly improves the quality of the generated data and our method is better than other state-of-the-art GANs to prepare synthetic images when the size of training datasets are limited. By training UNet++ using both real and the synthetic data generated from the SinGAN-Seg pipeline, we show that models trained with synthetic data have very close performances to those trained on real data when the datasets have a considerable amount of data. In contrast, Synthetic data generated from the SinGAN-Seg pipeline can improve the performance of segmentation models when training datasets do not have a considerable amount of data. The code is available on GitHub.

arXiv:2411.09688공식 코드 · Papers with Code

Squeezed Attention: Accelerating Long Context Length LLM Inference

Emerging Large Language Model (LLM) applications require long input prompts to perform complex downstream tasks like document analysis and code generation. For these long context length applications, the length of the input prompt poses a significant challenge in terms of inference efficiency since the inference costs increase linearly with sequence length. However, for many of these applications, much of the context in the prompt is fixed across different user inputs, thereby providing the opportunity to perform offline optimizations to process user inputs quickly, as they are received. In this work, we propose Squeezed Attention as a mechanism to accelerate LLM applications where a large portion of the input prompt is fixed. We first leverage K-means clustering offline to group the keys for the fixed context based on semantic similarity and represent each cluster with a single centroid value. During inference, we compare query tokens from the user input with the centroids to predict which of the keys from the fixed context are semantically relevant and need to be loaded during inference. We then compute exact attention using only these important keys from the fixed context, thereby reducing bandwidth and computational costs. We also extend our method to use a hierarchical centroid lookup to identify important keys, which can reduce the complexity of attention from linear to logarithmic with respect to the context length. We implement optimized Triton kernels for centroid comparison and sparse FlashAttention with important keys, achieving more than 4x speedups during both the prefill and generation phases for long-context inference. Furthermore, we have extensively evaluated our method on various long-context benchmarks including LongBench, where it achieves a 3x reduction in KV cache budget without accuracy loss and up to an 8x reduction with <0.5 point accuracy gap for various models.

arXiv:2607.13164v1cs.CL

Text2Sign: 텍스트-수화언어 비디오 생성을 위한 단일 GPU 디퓨전 기준선

Text2Sign: A Single-GPU Diffusion Baseline for Text-to-Sign Language Video Generation

수화언어는 수백만 명의 농인(Deaf) 및 난청인(hard-of-hearing)을 위한 주요 의사소통 수단이지만, 비디오 디퓨전 모델의 학습과 평가 비용이 높아 텍스트-수화자(text-to-signer) 비디오 생성은 여전히 비용이 많이 든다. 본 논문은 단일 NVIDIA L4 GPU에서 실행되는 짧은 수화 클립용 텍스트 조건부 디퓨전 모델인 Text2Sign을 제안한다. 동결(frozen)된 비전-언어(vision-language) 텍스트 인코더와 3D 인코더-디코더, 그리고 분해(factorized) 시공간(spatiotemporal) 어텐션을 결합하여 전체 비디오 어텐션의 비용을 줄이면서 동작 일관성(motion coherence)을 유지한다. 합성곱 전용 및 트랜스포머 스타일 백본, 동결된 사전학습 및 과제별 텍스트 인코더, 분해 어텐션 대 전체(full) 어텐션을 비교한다. 수화자 비중복(signer-disjoint) How2Sign 분할에서, 최고의 단기 실행 ablation은 검증 손실 0.0648에 도달하고, 장기 실행 체크포인트는 0.00999에 도달한다. 간소화된 평가 슬라이스에서, 후자는 guidance scale 5.0의 8단계 DDIM 샘플링을 사용하여 SSIM $0.2403 \pm 0.0238$, PSNR $15.11 \pm 0.42$ dB, 시간적 일관성 $1.0000 \pm 0.0000$을 달성한다. 32프레임, $64 \times 64$ 클립을 12.60초, 즉 초당 2.54프레임으로 생성하며, 추론 메모리 피크는 3.12 GB이다. 홀드아웃(held-out) 디노이징 감사(audit)는 프롬프트 민감도가 약함을 보여준다: 텍스트를 제거하면 후기 타임스텝(late-timestep) 손실이 0.9875에서 0.9891로 증가하는 반면, 섞인(shuffled) 프롬프트는 올바른 프롬프트와 유사한 성능을 보인다. 따라서 동결된 텍스트 조건화는 짧은 예산의 검증 손실을 개선하지만, 프롬프트별 분리는 여전히 제한적이다. 이 시스템은 저해상도의 짧은 클립에 한정되며 전문 언어학적 평가가 부족하므로, 완전한 수화언어 프로덕션 시스템이라기보다 단일 GPU 연구 기준선으로 보아야 한다. 코드는 https://github.com/xiaruize0911/text2sign 에서 이용할 수 있다.

arXiv:1804.07755공식 코드 · Papers with Code

Phrase-Based & Neural Unsupervised Machine Translation

Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs. This work investigates how to learn to translate when having access to only large monolingual corpora in each language. We propose two model variants, a neural and a phrase-based model. Both versions leverage a careful initialization of the parameters, the denoising effect of language models and automatic generation of parallel data by iterative back-translation. These models are significantly better than methods from the literature, while being simpler and having fewer hyper-parameters. On the widely used WMT'14 English-French and WMT'16 German-English benchmarks, our models respectively obtain 28.1 and 25.2 BLEU points without using a single parallel sentence, outperforming the state of the art by more than 11 BLEU points. On low-resource languages like English-Urdu and English-Romanian, our methods achieve even better results than semi-supervised and supervised approaches leveraging the paucity of available bitexts. Our code for NMT and PBSMT is publicly available.

arXiv:2004.08692공식 코드 · Papers with Code

A Spatio-temporal Transformer for 3D Human Motion Prediction

We propose a novel Transformer-based architecture for the task of generative modelling of 3D human motion. Previous work commonly relies on RNN-based models considering shorter forecast horizons reaching a stationary and often implausible state quickly. Recent studies show that implicit temporal representations in the frequency domain are also effective in making predictions for a predetermined horizon. Our focus lies on learning spatio-temporal representations autoregressively and hence generation of plausible future developments over both short and long term. The proposed model learns high dimensional embeddings for skeletal joints and how to compose a temporally coherent pose via a decoupled temporal and spatial self-attention mechanism. Our dual attention concept allows the model to access current and past information directly and to capture both the structural and the temporal dependencies explicitly. We show empirically that this effectively learns the underlying motion dynamics and reduces error accumulation over time observed in auto-regressive models. Our model is able to make accurate short-term predictions and generate plausible motion sequences over long horizons. We make our code publicly available at https://github.com/eth-ait/motion-transformer.

arXiv:2006.08820

Quantitatively Assessing the Benefits of Model-driven Development in Agent-based Modeling and Simulation

The agent-based modeling and simulation (ABMS) paradigm has been used to analyze, reproduce, and predict phenomena related to many application areas. Although there are many agent-based platforms that support simulation development, they rely on programming languages that require extensive programming knowledge. Model-driven development (MDD) has been explored to facilitate simulation modeling, by means of high-level modeling languages that provide reusable building blocks that hide computational complexity, and code generation. However, there is still limited knowledge of how MDD approaches to ABMS contribute to increasing development productivity and quality. We thus in this paper present an empirical study that quantitatively compares the use of MDD and ABMS platforms mainly in terms of effort and developer mistakes. Our evaluation was performed using MDD4ABMS-an MDD approach with a core and extensions to two application areas, one of which developed for this study-and NetLogo, a widely used platform. The obtained results show that MDD4ABMS requires less effort to develop simulations with similar (sometimes better) design quality than NetLogo, giving evidence of the benefits that MDD can provide to ABMS.

arXiv:2010.14974공식 코드 · Papers with Code

Object Hider: Adversarial Patch Attack Against Object Detectors

Deep neural networks have been widely used in many computer vision tasks. However, it is proved that they are susceptible to small, imperceptible perturbations added to the input. Inputs with elaborately designed perturbations that can fool deep learning models are called adversarial examples, and they have drawn great concerns about the safety of deep neural networks. Object detection algorithms are designed to locate and classify objects in images or videos and they are the core of many computer vision tasks, which have great research value and wide applications. In this paper, we focus on adversarial attack on some state-of-the-art object detection models. As a practical alternative, we use adversarial patches for the attack. Two adversarial patch generation algorithms have been proposed: the heatmap-based algorithm and the consensus-based algorithm. The experiment results have shown that the proposed methods are highly effective, transferable and generic. Additionally, we have applied the proposed methods to competition "Adversarial Challenge on Object Detection" that is organized by Alibaba on the Tianchi platform and won top 7 in 1701 teams. Code is available at: https://github.com/FenHua/DetDak

arXiv:2210.06462공식 코드 · Papers with Code

Self-Guided Diffusion Models

Diffusion models have demonstrated remarkable progress in image generation quality, especially when guidance is used to control the generative process. However, guidance requires a large amount of image-annotation pairs for training and is thus dependent on their availability, correctness and unbiasedness. In this paper, we eliminate the need for such annotation by instead leveraging the flexibility of self-supervision signals to design a framework for self-guided diffusion models. By leveraging a feature extraction function and a self-annotation function, our method provides guidance signals at various image granularities: from the level of holistic images to object boxes and even segmentation masks. Our experiments on single-label and multi-label image datasets demonstrate that self-labeled guidance always outperforms diffusion models without guidance and may even surpass guidance based on ground-truth labels, especially on unbalanced data. When equipped with self-supervised box or mask proposals, our method further generates visually diverse yet semantically consistent images, without the need for any class, box, or segment label annotation. Self-guided diffusion is simple, flexible and expected to profit from deployment at scale. Source code will be at: https://taohu.me/sgdm/

arXiv:2212.03293공식 코드 · Papers with Code

Diffusion-SDF: Text-to-Shape via Voxelized Diffusion

With the rising industrial attention to 3D virtual modeling technology, generating novel 3D content based on specified conditions (e.g. text) has become a hot issue. In this paper, we propose a new generative 3D modeling framework called Diffusion-SDF for the challenging task of text-to-shape synthesis. Previous approaches lack flexibility in both 3D data representation and shape generation, thereby failing to generate highly diversified 3D shapes conforming to the given text descriptions. To address this, we propose a SDF autoencoder together with the Voxelized Diffusion model to learn and generate representations for voxelized signed distance fields (SDFs) of 3D shapes. Specifically, we design a novel UinU-Net architecture that implants a local-focused inner network inside the standard U-Net architecture, which enables better reconstruction of patch-independent SDF representations. We extend our approach to further text-to-shape tasks including text-conditioned shape completion and manipulation. Experimental results show that Diffusion-SDF generates both higher quality and more diversified 3D shapes that conform well to given text descriptions when compared to previous approaches. Code is available at: https://github.com/ttlmh/Diffusion-SDF

arXiv:2212.09746공식 코드 · Papers with Code

Evaluating Human-Language Model Interaction

Many real-world applications of language models (LMs), such as writing assistance and code autocomplete, involve human-LM interaction. However, most benchmarks are non-interactive in that a model produces output without human involvement. To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics. Compared to standard, non-interactive evaluation, HALIE captures (i) the interactive process, not only the final output; (ii) the first-person subjective experience, not just a third-party assessment; and (iii) notions of preference beyond quality (e.g., enjoyment and ownership). We then design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. With four state-of-the-art LMs (three variants of OpenAI's GPT-3 and AI21 Labs' Jurassic-1), we find that better non-interactive performance does not always translate to better human-LM interaction. In particular, we highlight three cases where the results from non-interactive and interactive metrics diverge and underscore the importance of human-LM interaction for LM evaluation.

arXiv:2301.09460

HRVQA: A Visual Question Answering Benchmark for High-Resolution Aerial Images

Visual question answering (VQA) is an important and challenging multimodal task in computer vision. Recently, a few efforts have been made to bring VQA task to aerial images, due to its potential real-world applications in disaster monitoring, urban planning, and digital earth product generation. However, not only the huge variation in the appearance, scale and orientation of the concepts in aerial images, but also the scarcity of the well-annotated datasets restricts the development of VQA in this domain. In this paper, we introduce a new dataset, HRVQA, which provides collected 53512 aerial images of 1024*1024 pixels and semi-automatically generated 1070240 QA pairs. To benchmark the understanding capability of VQA models for aerial images, we evaluate the relevant methods on HRVQA. Moreover, we propose a novel model, GFTransformer, with gated attention modules and a mutual fusion module. The experiments show that the proposed dataset is quite challenging, especially the specific attribute related questions. Our method achieves superior performance in comparison to the previous state-of-the-art approaches. The dataset and the source code will be released at https://hrvqa.nl/.