PyTorchKR Papers

“Code Generation”

총 10,163편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2503.23905공식 코드 · Papers with Code

Boosting MLLM Reasoning with Text-Debiased Hint-GRPO

MLLM reasoning has drawn widespread research for its excellent problem-solving capability. Current reasoning methods fall into two types: PRM, which supervises the intermediate reasoning steps, and ORM, which supervises the final results. Recently, DeepSeek-R1 has challenged the traditional view that PRM outperforms ORM, which demonstrates strong generalization performance using an ORM method (i.e., GRPO). However, current MLLM's GRPO algorithms still struggle to handle challenging and complex multimodal reasoning tasks (e.g., mathematical reasoning). In this work, we reveal two problems that impede the performance of GRPO on the MLLM: Low data utilization and Text-bias. Low data utilization refers to that GRPO cannot acquire positive rewards to update the MLLM on difficult samples, and text-bias is a phenomenon that the MLLM bypasses image condition and solely relies on text condition for generation after GRPO training. To tackle these problems, this work proposes Hint-GRPO that improves data utilization by adaptively providing hints for samples of varying difficulty, and text-bias calibration that mitigates text-bias by calibrating the token prediction logits with image condition in test-time. Experiment results on three base MLLMs across eleven datasets demonstrate that our proposed methods advance the reasoning capability of original MLLM by a large margin, exhibiting superior performance to existing MLLM reasoning methods. Our code is available at https://github.com/hqhQAQ/Hint-GRPO.

arXiv:2504.12961

QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning?

Credit assignment has remained a fundamental challenge in multi-agent reinforcement learning (MARL). Previous studies have primarily addressed this issue through value decomposition methods under the centralized training with decentralized execution paradigm, where neural networks are utilized to approximate the nonlinear relationship between individual Q-values and the global Q-value. Although these approaches have achieved considerable success in various benchmark tasks, they still suffer from several limitations, including imprecise attribution of contributions, limited interpretability, and poor scalability in high-dimensional state spaces. To address these challenges, we propose a novel algorithm, \textbf{QLLM}, which facilitates the automatic construction of credit assignment functions using large language models (LLMs). Specifically, the concept of \textbf{TFCAF} is introduced, wherein the credit allocation process is represented as a direct and expressive nonlinear functional formulation. A custom-designed \textit{coder-evaluator} framework is further employed to guide the generation, verification, and refinement of executable code by LLMs, significantly mitigating issues such as hallucination and shallow reasoning during inference. Extensive experiments conducted on several standard MARL benchmarks demonstrate that the proposed method consistently outperforms existing state-of-the-art baselines. Moreover, QLLM exhibits strong generalization capability and maintains compatibility with a wide range of MARL algorithms that utilize mixing networks, positioning it as a promising and versatile solution for complex multi-agent scenarios.

arXiv:2609.07108v3cs.LG

Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training

Speculative decoding accelerates rollout generation, which dominates the cost of reinforcement learning (RL) post-training. Online co-training keeps the draft aligned with the evolving policy, further increasing acceptance length and speedup. However, training large models on long contexts requires context parallelism (CP) and pipeline parallelism (PP), and co-training drafts under this layout poses two obstacles: (1) the draft's branch attention is unsupported by causal CP, and (2) the target features that the draft consumes are spread across PP stages. In this paper, we present the design and implementation of online draft co-training in NeMo-RL. For CP, we extend packed, load-balanced zigzag ring attention by merging rank-local branch attention with causal main-sequence attention. For PP, TapChannel transports intermediate target features across stages via a separate path, leaving the pipeline schedule unaffected. Experiments show that co-training keeps the policy's reward and accuracy close to the no-draft baseline while delivering $1.48$--$2.25\times$ rollout speedups and $1.19$--$1.88\times$ end-to-end training speedups across model scales up to 122B. Our CP design achieves strong scaling at 256K sequence length with significant memory savings over prior work, and our PP transport incurs modest overhead. Code can be found at https://github.com/NVIDIA-NeMo/RL/issues/3698.

arXiv:2603.07148v2cs.LG

Agentic Planning with Reasoning for Image Styling via Offline RL

Direct prompt-based editing often fails on complex transformations because vague and subjective prompts often require nuanced understanding of what should be changed in the image. Our core intuition is that leveraging compositional image editing tools rather than direct prompting profits from structured agent-level planning with explicit reasoning, leading to better results. This structured planning framework enables efficient offline RL post-training on quality-scored trajectories to improve performance. We present a tool-based agentic RL post-training framework that addresses this through structured planning with chain-of-thought reasoning. Our key contributions include: (1) A tool-based agentic planning methodology that combines a compositional library of orthogonal primitive transformations, structured context representation, and explicit per-step reasoning to decompose complex styling into interpretable tool sequences. (2) A synthetic data generation pipeline producing three large-scale datasets (each $\sim$10K trajectories) with reasoning chains, plans, and quality scores, as no existing datasets provide such supervision. Our datasets and code are publicly available at the HuggingFace repository. (3) Offline RL training methods for learning planners with reasoning as our core algorithmic contributions, which consistently improve over the Edit-Only baseline in visual quality and instruction following. (4) Comprehensive evaluation across 4B and 8B parameter Qwen3-VL models showing that our methods outperform other baselines in the majority of compositional tasks, validated by human evaluations.

arXiv:2609.33386v2cs.GR

Sivia: Learning to Draw, Learning to Learn

A paper's overview figure is often the first thing readers see: it must explain the method correctly and look polished. We present Sivia, an overview-figure system that reads the paper, plans its relations, adapts visual organization from recent top-conference papers, and composes a design for image generation. Sivia improves with use through human-guided recursive self-improvement (RSI) at two levels: learning to draw revises its drawing skill from feedback on accuracy and presentation, and learning to learn revises the updater that proposes those revisions. A personal profile adapts presentation to its user's aesthetic preferences while the scientific content stays fixed. A diagnostic study of 1,964 overviews for 445 papers (17,973 relation judgments) shows that most errors are omissions and that source-derived relation plans raise required-relation accuracy (RRA) by up to 19.85 points. On 48 test papers, learning both levels reaches 93.54\% RRA, 15.31, 9.38, and 8.33 points above the initial skill, the Fixed-updater, and the GEPA baselines, while readability and visual quality rise from 3.95 and 4.00 to 4.70 and 4.78 on five-point scales. Learned presentation profiles raise liking from 4.02 to 4.73 and adoption from 34 to 45 of 48 tasks. Code and released artifacts are available at https://github.com/exsinger-hub/Sivia

arXiv:2607.27788v1cs.AI

SpecCal: 적외선 분광 기반 분자 구조 재구성을 위한 모호성 인식 후보 보정

SpecCal: Ambiguity-Aware Candidate Calibration for Infrared Spectrum-Based Molecular Structure Reconstruction

적외선(IR) 스펙트럼으로부터 분자 구조를 추론하는 것은 근본적이면서도 까다로운 문제입니다. 핵심적인 어려움은 IR 스펙트럼이 제한된 구조 정보만을 제공한다는 점입니다. 서로 다른 분자라도 유사한 작용기와 국소 진동 패턴을 공유하여 매우 유사한 스펙트럼 반응을 나타낼 수 있습니다. 따라서 관측된 스펙트럼에 대응하는 고유한 기본 구조가 존재하더라도, 스펙트럼으로부터 이를 재구성하는 작업에는 여전히 모호성이 남습니다. 기존의 IR-to-molecule 모델들은 대개 순위가 매겨진 후보 분자 집합을 생성하지만, 이 집합은 모델의 학습된 생성 선호도에 의해 크게 좌우되며 관측된 스펙트럼 제약 조건을 가장 잘 만족하는 구조를 완전히 포착하지 못할 수 있습니다. 이러한 한계를 해결하기 위해, 본 논문에서는 IR-to-molecule 예측을 위한 학습 불필요(training-free) 후보 보정 프레임워크인 SpecCal을 제안합니다. SpecCal은 기존 베이스 모델의 후보 출력에 작용하며, 기존 후보들을 재순위화(re-ranking)하는 동시에 스펙트럼 일관성을 기반으로 구조적으로 타당한 대안 후보를 추가 도입하여 예측 집합을 개선합니다. 이 프레임워크는 플러그 앤 플레이(plug-and-play) 방식이자 모델 무관(model-agnostic) 방식으로, 다양한 베이스 모델과 통합할 때 별도의 매개변수 업데이트를 요구하지 않습니다. 여러 벤치마크에 대한 실험 결과, SpecCal은 다양한 베이스 모델에 걸쳐 SMILES 및 스캐폴드(scaffold) 수준 모두에서 top-k 재구성 성능을 일관되게 향상시키는 것으로 나타났습니다. 추가 분석을 통해 스펙트럼 모호성 하에서 후보 집합을 보정하는 것이 IR 스펙트럼으로부터의 분자 재구성을 개선하는 실용적인 방법임을 입증했습니다. 코드는 https://anonymous.4open.science/r/SpecCal-B18A 에서 확인할 수 있습니다.

arXiv:2608.28478v1cs.CL

장님과 코끼리: 롱테일 상충 지식 하에서 LLM의 인식론적 근시 탐색

Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge

사실 기반 질의응답(QA)은 보통 단 하나의 정답(canonical answer)을 가정하여, 대규모 언어 모델(LLM)이 롱테일 사실에 대해 서로 다른 서술을 얼마나 유지하는지는 가려진다. 이 공백을 메우기 위해 우리는 감사 가능한 그래프 기반 파이프라인으로 생성한 1,094개의 질문으로 구성된 클로즈드북 지식 프로브 ElephantBench를 제안한다. 이 파이프라인은 노출도가 낮은 웹 코퍼스에서 관련 문서를 검색하고, 자연스럽게 발생하는 불일치를 식별한 뒤 이를 다중 서술 QA 레코드로 변환한다. 각 답변은 원 출처 문서 및 권위 있는 공개 웹 소스에 대해 검증되며, 이후 사람 어노테이터가 검토한다. 32개 모델에 걸쳐, 가장 강한 모델조차 질문의 52.4%에서만 두 서술을 모두 회복하며, 나머지 거의 모든 질문에서는 한 서술은 떠올리지만 다른 하나는 누락한다. 모델 크기 스케일링과 추론 시점의 reasoning은 회상을 개선하지만 이러한 불완전성을 없애지는 못한다. 코퍼스 분석은 further 노출 불균형이 지배적 서술에 유리함을 보여 주며, 소수 측 노출이 더 클수록 더 완전한 회상과 연관된다. 이러한 결과는 ElephantBench를 파라메트릭 메모리에서 인식론적 근시를 진단하는 재현 가능한 지식 프로브로 자리매김한다. 더 넓게, 우리의 그래프 기반 벤치마크 구축 파이프라인은 롱테일 코퍼스를 출처 추적 가능한 지식 프로브로 바꾸는 효율적이고 확장 가능한 방법을 제공하여, 차세대 LLM의 인식론적 엄밀성(epistemic rigour)을 평가하고 향상시키려는 노력을 뒷받침한다. 코드는 https://github.com/Tencent/ElephantBench 에서 이용할 수 있다.

arXiv:2609.03047v1cs.CL

SHELF: 다중 태스크 서지 벤치마킹을 위한 합성 하네스

SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

도서관과 아카이브는 제한된 인력과 컴퓨팅 예산으로 대규모 컬렉션을 관리하지만, 기존의 일반적인 벤치마크는 이들의 서지(bibliographic) 작업을 체계적으로 테스트하지 못합니다. 따라서 이들에게는 어떤 방법이 자신들의 태스크에 적합하며, 해당 방법을 실행하는 데 무엇이 필요한지 파악하는 것이 필수적입니다. LLM 적합성 평가를 위한 합성 하네스인 SHELF(Synthetic Harness for Evaluating LLM Fitness)는 이러한 공백을 해결합니다. SHELF는 라벨이 지정된 분류 체계(taxonomy), 작성 규격, 생성 예산을 통제된 벤치마크 데이터 및 평가 태스크로 변환하는 Python 시스템입니다. 이번 첫 릴리스에는 미국 의회도서관(Library of Congress) 어휘를 기반으로 모델이 작성한 62,899개의 문서가 포함되어 있으며, 분류(classification), 군집화(clustering), 검색(retrieval), 쌍 분류(pair classification), 인스트럭션 검색(instruction retrieval) 태스크를 제공합니다. 본 연구에서는 TF, TF-IDF, BM25, 널리 쓰이는 인코더, 그리고 주제 분류에 한해 제로샷(zero-shot) 디코더를 비교하며, 각 방법은 이를 지원하는 태스크에만 적용됩니다. 실험 결과, 주제 분류는 0.8887에 도달한 반면 장르-형식 분류는 0.2605에 그쳤고, 일부 쌍 분류 및 군집화 태스크는 무작위 선택 확률(chance level) 근처에 머물렀습니다. 희소(sparse) 방식은 분류에서 여전히 경쟁력을 유지했으며, TF-IDF는 주제별 소요 시간 측정 실험에서 가장 빠른 성능을 보였습니다. 또한 SHELF는 서지 패싯(facet)을 독립적으로 변화시키며, 모델의 학습(training) 마감 시점(cutoff) 이후의 검증 가능하게 새로운 문서를 생성할 수 있습니다. LCSHBench 및 Project Gutenberg와의 비교는 절대적인 점수보다 모델 순위가 더 안정적으로 전이됨을 보여주지만, SHELF 점수가 실제 운영 카탈로그 데이터에 대한 정확도를 추정하지는 못한다는 한계도 나타냅니다. 본 연구의 모든 소스 코드와 데이터는 허용적 라이선스에 따라 GitHub와 Hugging Face에 공개됩니다.

arXiv:1707.09531공식 코드 · Papers with Code

Recurrent Scale Approximation for Object Detection in CNN

Since convolutional neural network (CNN) lacks an inherent mechanism to handle large scale variations, we always need to compute feature maps multiple times for multi-scale object detection, which has the bottleneck of computational cost in practice. To address this, we devise a recurrent scale approximation (RSA) to compute feature map once only, and only through this map can we approximate the rest maps on other levels. At the core of RSA is the recursive rolling out mechanism: given an initial map at a particular scale, it generates the prediction at a smaller scale that is half the size of input. To further increase efficiency and accuracy, we (a): design a scale-forecast network to globally predict potential scales in the image since there is no need to compute maps on all levels of the pyramid. (b): propose a landmark retracing network (LRN) to trace back locations of the regressed landmarks and generate a confidence score for each landmark; LRN can effectively alleviate false positives caused by the accumulated error in RSA. The whole system can be trained end-to-end in a unified CNN framework. Experiments demonstrate that our proposed algorithm is superior against state-of-the-art methods on face detection benchmarks and achieves comparable results for generic proposal generation. The source code of RSA is available at github.com/sciencefans/RSA-for-object-detection.

arXiv:2004.03064공식 코드 · Papers with Code

Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance

Gaze redirection aims at manipulating the gaze of a given face image with respect to a desired direction (i.e., a reference angle) and it can be applied to many real life scenarios, such as video-conferencing or taking group photos. However, previous work on this topic mainly suffers of two limitations: (1) Low-quality image generation and (2) Low redirection precision. In this paper, we propose to alleviate these problems by means of a novel gaze redirection framework which exploits both a numerical and a pictorial direction guidance, jointly with a coarse-to-fine learning strategy. Specifically, the coarse branch learns the spatial transformation which warps input image according to desired gaze. On the other hand, the fine-grained branch consists of a generator network with conditional residual image learning and a multi-task discriminator. This second branch reduces the gap between the previously warped image and the ground-truth image and recovers finer texture details. Moreover, we propose a numerical and pictorial guidance module~(NPG) which uses a pictorial gazemap description and numerical angles as an extra guide to further improve the precision of gaze redirection. Extensive experiments on a benchmark dataset show that the proposed method outperforms the state-of-the-art approaches in terms of both image quality and redirection precision. The code is available at https://github.com/jingjingchen777/CFGR

arXiv:2102.09754

VisuoSpatial Foresight for Physical Sequential Fabric Manipulation

Robotic fabric manipulation has applications in home robotics, textiles, senior care and surgery. Existing fabric manipulation techniques, however, are designed for specific tasks, making it difficult to generalize across different but related tasks. We build upon the Visual Foresight framework to learn fabric dynamics that can be efficiently reused to accomplish different sequential fabric manipulation tasks with a single goal-conditioned policy. We extend our earlier work on VisuoSpatial Foresight (VSF), which learns visual dynamics on domain randomized RGB images and depth maps simultaneously and completely in simulation. In this earlier work, we evaluated VSF on multi-step fabric smoothing and folding tasks against 5 baseline methods in simulation and on the da Vinci Research Kit (dVRK) surgical robot without any demonstrations at train or test time. A key finding was that depth sensing significantly improves performance: RGBD data yields an 80% improvement in fabric folding success rate in simulation over pure RGB data. In this work, we vary 4 components of VSF, including data generation, visual dynamics model, cost function, and optimization procedure. Results suggest that training visual dynamics models using longer, corner-based actions can improve the efficiency of fabric folding by 76% and enable a physical sequential fabric folding task that VSF could not previously perform with 90% reliability. Code, data, videos, and supplementary material are available at https://sites.google.com/view/fabric-vsf/.

arXiv:2106.06752

EDEN: A high-performance, general-purpose, NeuroML-based neural simulator

Modern neuroscience employs in silico experimentation on ever-increasing and more detailed neural networks. The high modelling detail goes hand in hand with the need for high model reproducibility, reusability and transparency. Besides, the size of the models and the long timescales under study mandate the use of a simulation system with high computational performance, so as to provide an acceptable time to result. In this work, we present EDEN (Extensible Dynamics Engine for Networks), a new general-purpose, NeuroML-based neural simulator that achieves both high model flexibility and high computational performance, through an innovative model-analysis and code-generation technique. The simulator runs NeuroML v2 models directly, eliminating the need for users to learn yet another simulator-specific, model-specification language. EDEN's functional correctness and computational performance were assessed through NeuroML models available on the NeuroML-DB and Open Source Brain model repositories. In qualitative experiments, the results produced by EDEN were verified against the established NEURON simulator, for a wide range of models. At the same time, computational-performance benchmarks reveal that EDEN runs up to 2 orders-of-magnitude faster than NEURON on a typical desktop computer, and does so without additional effort from the user. Finally, and without added user effort, EDEN has been built from scratch to scale seamlessly over multiple CPUs and across computer clusters, when available.

arXiv:2111.05011공식 코드 · Papers with Code

RAVE: A variational autoencoder for fast and high-quality neural audio synthesis

Deep generative models applied to audio have improved by a large margin the state-of-the-art in many speech and music related tasks. However, as raw waveform modelling remains an inherently difficult task, audio generative models are either computationally intensive, rely on low sampling rates, are complicated to control or restrict the nature of possible signals. Among those models, Variational AutoEncoders (VAE) give control over the generation by exposing latent variables, although they usually suffer from low synthesis quality. In this paper, we introduce a Realtime Audio Variational autoEncoder (RAVE) allowing both fast and high-quality audio waveform synthesis. We introduce a novel two-stage training procedure, namely representation learning and adversarial fine-tuning. We show that using a post-training analysis of the latent space allows a direct control between the reconstruction fidelity and the representation compactness. By leveraging a multi-band decomposition of the raw waveform, we show that our model is the first able to generate 48kHz audio signals, while simultaneously running 20 times faster than real-time on a standard laptop CPU. We evaluate synthesis quality using both quantitative and qualitative subjective experiments and show the superiority of our approach compared to existing models. Finally, we present applications of our model for timbre transfer and signal compression. All of our source code and audio examples are publicly available.

arXiv:2211.11039공식 코드 · Papers with Code

Deep Composite Face Image Attacks: Generation, Vulnerability and Detection

Face manipulation attacks have drawn the attention of biometric researchers because of their vulnerability to Face Recognition Systems (FRS). This paper proposes a novel scheme to generate Composite Face Image Attacks (CFIA) based on facial attributes using Generative Adversarial Networks (GANs). Given the face images corresponding to two unique data subjects, the proposed CFIA method will independently generate the segmented facial attributes, then blend them using transparent masks to generate the CFIA samples. We generate $526$ unique CFIA combinations of facial attributes for each pair of contributory data subjects. Extensive experiments are carried out on our newly generated CFIA dataset consisting of 1000 unique identities with 2000 bona fide samples and 526000 CFIA samples, thus resulting in an overall 528000 face image samples. {{We present a sequence of experiments to benchmark the attack potential of CFIA samples using four different automatic FRS}}. We introduced a new metric named Generalized Morphing Attack Potential (G-MAP) to benchmark the vulnerability of generated attacks on FRS effectively. Additional experiments are performed on the representative subset of the CFIA dataset to benchmark both perceptual quality and human observer response. Finally, the CFIA detection performance is benchmarked using three different single image based face Morphing Attack Detection (MAD) algorithms. The source code of the proposed method together with CFIA dataset will be made publicly available: \url{https://github.com/jagmohaniiit/LatentCompositionCode}

arXiv:2302.10887공식 코드 · Papers with Code

The configurable tree graph (CT-graph): measurable problems in partially observable and distal reward environments for lifelong reinforcement learning

This paper introduces a set of formally defined and transparent problems for reinforcement learning algorithms with the following characteristics: (1) variable degrees of observability (non-Markov observations), (2) distal and sparse rewards, (3) variable and hierarchical reward structure, (4) multiple-task generation, (5) variable problem complexity. The environment provides 1D or 2D categorical observations, and takes actions as input. The core structure of the CT-graph is a multi-branch tree graph with arbitrary branching factor, depth, and observation sets that can be varied to increase the dimensions of the problem in a controllable and measurable way. Two main categories of states, decision states and wait states, are devised to create a hierarchy of importance among observations, typical of real-world problems. A large observation set can produce a vast set of histories that impairs memory-augmented agents. Variable reward functions allow for the easy creation of multiple tasks and the ability of an agent to efficiently adapt in dynamic scenarios where tasks with controllable degrees of similarities are presented. Challenging complexity levels can be easily achieved due to the exponential growth of the graph. The problem formulation and accompanying code provide a fast, transparent, and mathematically defined set of configurable tests to compare the performance of reinforcement learning algorithms, in particular in lifelong learning settings.

arXiv:2304.03013

Tensor Slicing and Optimization for Multicore NPUs

Although code generation for Convolution Neural Network (CNN) models has been extensively studied, performing efficient data slicing and parallelization for highly-constrai\-ned Multicore Neural Processor Units (NPUs) is still a challenging problem. Given the size of convolutions' input/output tensors and the small footprint of NPU on-chip memories, minimizing memory transactions while maximizing parallelism and MAC utilization are central to any effective solution. This paper proposes a TensorFlow XLA/LLVM compiler optimization pass for Multicore NPUs, called Tensor Slicing Optimization (TSO), which: (a) maximizes convolution parallelism and memory usage across NPU cores; and (b) reduces data transfers between host and NPU on-chip memories by using DRAM memory burst time estimates to guide tensor slicing. To evaluate the proposed approach, a set of experiments was performed using the NeuroMorphic Processor (NMP), a multicore NPU containing 32 RISC-V cores extended with novel CNN instructions. Experimental results show that TSO is capable of identifying the best tensor slicing that minimizes execution time for a set of CNN models. Speed-ups of up to 21.7\% result when comparing the TSO burst-based technique to a no-burst data slicing approach. To validate the generality of the TSO approach, the algorithm was also ported to the Glow Machine Learning framework. The performance of the models were measured on both Glow and TensorFlow XLA/LLVM compilers, revealing similar results.

arXiv:2308.08724공식 코드 · Papers with Code

Long-Range Grouping Transformer for Multi-View 3D Reconstruction

Nowadays, transformer networks have demonstrated superior performance in many computer vision tasks. In a multi-view 3D reconstruction algorithm following this paradigm, self-attention processing has to deal with intricate image tokens including massive information when facing heavy amounts of view input. The curse of information content leads to the extreme difficulty of model learning. To alleviate this problem, recent methods compress the token number representing each view or discard the attention operations between the tokens from different views. Obviously, they give a negative impact on performance. Therefore, we propose long-range grouping attention (LGA) based on the divide-and-conquer principle. Tokens from all views are grouped for separate attention operations. The tokens in each group are sampled from all views and can provide macro representation for the resided view. The richness of feature learning is guaranteed by the diversity among different groups. An effective and efficient encoder can be established which connects inter-view features using LGA and extract intra-view features using the standard self-attention layer. Moreover, a novel progressive upsampling decoder is also designed for voxel generation with relatively high resolution. Hinging on the above, we construct a powerful transformer-based network, called LRGT. Experimental results on ShapeNet verify our method achieves SOTA accuracy in multi-view reconstruction. Code will be available at https://github.com/LiyingCV/Long-Range-Grouping-Transformer.

arXiv:2308.10408공식 코드 · Papers with Code

Turning a CLIP Model into a Scene Text Spotter

We exploit the potential of the large-scale Contrastive Language-Image Pretraining (CLIP) model to enhance scene text detection and spotting tasks, transforming it into a robust backbone, FastTCM-CR50. This backbone utilizes visual prompt learning and cross-attention in CLIP to extract image and text-based prior knowledge. Using predefined and learnable prompts, FastTCM-CR50 introduces an instance-language matching process to enhance the synergy between image and text embeddings, thereby refining text regions. Our Bimodal Similarity Matching (BSM) module facilitates dynamic language prompt generation, enabling offline computations and improving performance. FastTCM-CR50 offers several advantages: 1) It can enhance existing text detectors and spotters, improving performance by an average of 1.7% and 1.5%, respectively. 2) It outperforms the previous TCM-CR50 backbone, yielding an average improvement of 0.2% and 0.56% in text detection and spotting tasks, along with a 48.5% increase in inference speed. 3) It showcases robust few-shot training capabilities. Utilizing only 10% of the supervised data, FastTCM-CR50 improves performance by an average of 26.5% and 5.5% for text detection and spotting tasks, respectively. 4) It consistently enhances performance on out-of-distribution text detection and spotting datasets, particularly the NightTime-ArT subset from ICDAR2019-ArT and the DOTA dataset for oriented object detection. The code is available at https://github.com/wenwenyu/TCM.

arXiv:2309.04071공식 코드 · Papers with Code

Enhancing Hierarchical Transformers for Whole Brain Segmentation with Intracranial Measurements Integration

Whole brain segmentation with magnetic resonance imaging (MRI) enables the non-invasive measurement of brain regions, including total intracranial volume (TICV) and posterior fossa volume (PFV). Enhancing the existing whole brain segmentation methodology to incorporate intracranial measurements offers a heightened level of comprehensiveness in the analysis of brain structures. Despite its potential, the task of generalizing deep learning techniques for intracranial measurements faces data availability constraints due to limited manually annotated atlases encompassing whole brain and TICV/PFV labels. In this paper, we enhancing the hierarchical transformer UNesT for whole brain segmentation to achieve segmenting whole brain with 133 classes and TICV/PFV simultaneously. To address the problem of data scarcity, the model is first pretrained on 4859 T1-weighted (T1w) 3D volumes sourced from 8 different sites. These volumes are processed through a multi-atlas segmentation pipeline for label generation, while TICV/PFV labels are unavailable. Subsequently, the model is finetuned with 45 T1w 3D volumes from Open Access Series Imaging Studies (OASIS) where both 133 whole brain classes and TICV/PFV labels are available. We evaluate our method with Dice similarity coefficients(DSC). We show that our model is able to conduct precise TICV/PFV estimation while maintaining the 132 brain regions performance at a comparable level. Code and trained model are available at: https://github.com/MASILab/UNesT/tree/main/wholebrainSeg.

arXiv:2310.10967공식 코드 · Papers with Code

EXMODD: An EXplanatory Multimodal Open-Domain Dialogue dataset

The need for high-quality data has been a key issue hindering the research of dialogue tasks. Recent studies try to build datasets through manual, web crawling, and large pre-trained models. However, man-made data is expensive and data collected from the internet often includes generic responses, meaningless statements, and toxic dialogues. Automatic data generation through large models is a cost-effective method, but for open-domain multimodal dialogue tasks, there are still three drawbacks: 1) There is currently no open-source large model that can accept multimodal input; 2) The content generated by the model lacks interpretability; 3) The generated data is usually difficult to quality control and require extensive resource to collect. To alleviate the significant human and resource expenditure in data collection, we propose a Multimodal Data Construction Framework (MDCF). MDCF designs proper prompts to spur the large-scale pre-trained language model to generate well-formed and satisfactory content. Additionally, MDCF also automatically provides explanation for a given image and its corresponding dialogue, which can provide a certain degree of interpretability and facilitate manual follow-up quality inspection. Based on this, we release an Explanatory Multimodal Open-Domain dialogue dataset (EXMODD). Experiments indicate a positive correlation between the model's ability to generate accurate understandings and high-quality responses. Our code and data can be found at https://github.com/poplpr/EXMODD.