PyTorchKR Papers

“Code Generation”

총 10,186편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2607.19409v1cs.AI

FORCE-Bench: 기업 금융 분야 에이전트 AI를 위한 벤치마크, 데이터셋 및 평가 하네스

FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance

최근 대규모 언어 모델(LLM)의 발전으로 운영 금융(operational finance) 분야에서 에이전트 시스템의 배포가 가속화되고 있다. 기존 벤치마크는 일반 역량, 지시 따르기(instruction following), 또는 안전성(safety) 측정에 중점을 두지만, 에이전트 시스템이 자동화하도록 배포되고 있는 운영 금융 워크플로를 직접 다루는 벤치마크는 거의 없다. 금융 전문가들은 에이전트가 사실에 부합하고 적절히 그라운딩(grounding)된 정보를 제공할 뿐만 아니라, 이러한 정보가 검증 가능하고 운영 금융 도메인의 규칙과 제약을 일관되게 준수하는 것을 요구한다. 우리는 251개의 전문가 주석 쿼리를 포함하고, 운영 금융 도메인 요구사항에 맞게 보정된 루브릭 기반 프레임워크를 사용하여 응답을 정확성(accuracy), 인용(citations), 명확성(clarity), 깊이(depth), 그라운딩(groundedness), 최신성(recency), 관련성(relevance), 구조(structure)의 8개 차원에서 평가하는 FORCE-Bench를 제안한다. FORCE-Bench는 세 가지 작업 유형에서 에이전트 시스템을 평가한다: 재무 의무 조사(매출채권 및 매입채무 데이터를 위해 ERP 시스템을 조회), 금융 실체 성과 조사(공시 자료 및 시장 데이터로부터 시한이 정해진 질문에 답변), 비즈니스 브리프 생성(다중 출처의 기업 인텔리전스 보고서 종합). 실제 배포 조건을 반영하기 위해, 우리는 목적에 맞게 구축된 에이전트와 범용 에이전트 시스템 모두를 공통 도구 접근 및 지연 시간 제한(latency-bounded) 설정 하에서 평가한다. 결과는 범용 에이전트 시스템이 운영 제약 하에서 금융 도메인 품질 요구사항을 일관되게 충족하지 못하는 반면, Microsoft 365 Copilot을 위해 목적에 맞게 구축된 Finance Agent가 모든 차원에서 더 신뢰할 수 있음을 보여준다. 우리는 재현 가능한 비교와 다른 기업 금융 환경에의 적응을 지원하기 위해 데이터셋, 루브릭, 하네스, 분석 코드를 오픈소스로 공개한다.

arXiv:2608.13040v1cs.LG

잠재 온-정책 자기 증류

Latent On-Policy Self-Distillation

에이전트가 경험으로부터 학습하고 이를 정책에 내재화하도록 하는 것은 자기 진화형 AI의 핵심 과제가 되었습니다. 온-정책 자기 증류(OPSD)는 특권적 자기 교사(privileged self-teacher)를 활용하여 학생 에이전트 자신의 궤적에 대해 조밀한 지도(dense supervision)를 제공함으로써 효과적인 경로를 제시합니다. 그러나 기존 방법들은 설계자가 지정한 특권적 아티팩트(예: 정답, 피드백, 스킬, 궤적)에 여전히 크게 의존하므로, 지속적인 자기 개선에 필요한 엔드투엔드(end-to-end) 학습 가능성과 확장성을 제한합니다. 본 연구에서는 새로 규정된 형태의 특권적 컨텍스트를 갖는 또 다른 수작업 설계 OPSD 변형을 제안하는 대신, 교사의 특권적 컨텍스트 자체를 경험으로부터 엔드투엔드로 학습할 수 있도록 만드는 잠재 온-정책 자기 증류(LOPD)를 소개합니다. 기술적으로 LOPD는 관련 경험을 검색하여 이를 자기 교사의 조건으로 활용되는 연속적인 잠재 토큰으로 구성하며, 학생 에이전트는 과제 및 상호작용 기록으로부터 궤적을 생성하고 방문한 모든 접두 문맥에서 조밀한 토큰 수준 지도를 받습니다. 또한 잠재 컨텍스트의 학습을 안정화하고 조절하기 위해 특권적 마진 목적함수(privileged-margin objective)를 도입합니다. 실험적으로 LOPD는 (I) 에이전트 도구 사용과 코드 생성 모두에서 RLVR 및 OPSD, SDPO, Skill-SD를 포함한 대표적인 OPSD 방법들을 능가하는 강력한 성능을 보여주며, (II) 롤아웃 예산의 30% 미만만으로도 GRPO 및 Skill-SD를 넘어서는 높은 학습 효율성을 입증합니다. 절제 연구(ablation study)는 특권적 컨텍스트를 학습 가능하게 만드는 것이 이러한 성능 향상을 달성하는 데 필수적이라는 직접적인 증거를 추가로 제공합니다. 종합적으로 이러한 결과들은 LOPD가 에이전트 진화를 위한 보다 확장 가능하고 자기 주도적인 패러다임을 향한 한 걸음임을 보여줍니다.

arXiv:2610.07874v1cs.LG

부정 정책 롤아웃을 활용한 온-정책 증류

On-Policy Distillation with Negative-Policy Rollouts

온-정책 증류(On-policy distillation, OPD)는 학생 모델이 자체 롤아웃(rollout)에서 더 우수한 교사 모델로부터 토큰 수준의 감독을 받는 사후 학습(post-training) 방법으로 널리 연구되어 왔습니다. 최근 연구들은 증류의 목적이 교사 모델을 모방하는 데 집중된 상태를 유지하면서도, 대안적인 증류 보상(reward) 공식화 및 교사 모델 구성을 통해 OPD를 개선했습니다. 그러나 더 우수한 교사 모델이 학생 모델과 제한된 분포 중첩(distributional overlap)을 가질 경우, 이러한 긍정적 지침(positive guidance)은 불충분한 학습 신호를 제공할 수 있습니다. 본 연구에서는 성능과 능력이 더 낮은 부정 정책(negative policy)의 롤아웃으로 교사 감독을 보완하여, 해당 부정 정책이 학생 모델을 위한 부정적 기준점 역할을 하도록 하는 Negative-Policy OPD(NP-OPD)를 소개합니다. NP-OPD는 증류 보상 공식을 수정하는 대신 롤아웃 단계에 부정 정책을 도입하여, 교사 모델보다 부정 정책이 선호하는 토큰을 지속적으로 공급함으로써 학습 전반에 걸쳐 교사 감독에 계속 노출되도록 합니다. 이를 통해 OPD에 사용되는 긍정적 교사 감독을 유지하면서 부정 정책 롤아웃을 통한 명시적인 부정 신호를 제공합니다. 광범위한 실험을 통해 NP-OPD가 모델 규모, 생성 모드, 추론 도메인 및 다양한 OPD 변형 전반에 걸쳐 OPD 성능을 향상시킴을 보여줍니다. 나아가 분석 결과, NP-OPD가 교사 모델보다 부정 정책이 선호하는 토큰을 효과적으로 억제하고 학생 모델을 부정 정책으로부터 멀어지게 함을 확인했습니다. 이러한 결과는 부정 정책 롤아웃을 통해 부정적 신호를 도입하는 본 설계 방식을 뒷받침하며, OPD에서 롤아웃 정책의 역할에 대한 새로운 통찰을 제공합니다. 코드는 https://github.com/naver-ai/np-opd 에서 제공될 예정입니다.

arXiv:1512.02325공식 코드 · Papers with Code

SSD: Single Shot MultiBox Detector

We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. Our SSD model is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stage and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, MS COCO, and ILSVRC datasets confirm that SSD has comparable accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. Compared to other single stage methods, SSD has much better accuracy, even with a smaller input image size. For $300\times 300$ input, SSD achieves 72.1% mAP on VOC2007 test at 58 FPS on a Nvidia Titan X and for $500\times 500$ input, SSD achieves 75.1% mAP, outperforming a comparable state of the art Faster R-CNN model. Code is available at https://github.com/weiliu89/caffe/tree/ssd .

arXiv:1802.08251

SMAGEXP: a galaxy tool suite for transcriptomics data meta-analysis

Bakground: With the proliferation of available microarray and high throughput sequencing experiments in the public domain, the use of meta-analysis methods increases. In these experiments, where the sample size is often limited, meta-analysis offers the possibility to considerably enhance the statistical power and give more accurate results. For those purposes, it combines either effect sizes or results of single studies in a appropriate manner. R packages metaMA and metaRNASeq perform meta-analysis on microarray and NGS data, respectively. They are not interchangeable as they rely on statistical modeling specific to each technology. Results: SMAGEXP (Statistical Meta-Analysis for Gene EXPression) integrates metaMA and metaRNAseq packages into Galaxy. We aim to propose a unified way to carry out meta-analysis of gene expression data, while taking care of their specificities. We have developed this tool suite to analyse microarray data from Gene Expression Omnibus (GEO) database or custom data from affymetrix microarrays. These data are then combined to carry out meta-analysis using metaMA package. SMAGEXP also offers to combine raw read counts from Next Generation Sequencing (NGS) experiments using DESeq2 and metaRNASeq package. In both cases, key values, independent from the technology type, are reported to judge the quality of the meta-analysis. These tools are available on the Galaxy main tool shed. Source code, help and installation instructions are available on github. Conclusion: The use of Galaxy offers an easy-to-use gene expression meta-analysis tool suite based on the metaMA and metaRNASeq packages.

arXiv:2105.05873공식 코드 · Papers with Code

Out of the Box: Embodied Navigation in the Real World

The research field of Embodied AI has witnessed substantial progress in visual navigation and exploration thanks to powerful simulating platforms and the availability of 3D data of indoor and photorealistic environments. These two factors have opened the doors to a new generation of intelligent agents capable of achieving nearly perfect PointGoal Navigation. However, such architectures are commonly trained with millions, if not billions, of frames and tested in simulation. Together with great enthusiasm, these results yield a question: how many researchers will effectively benefit from these advances? In this work, we detail how to transfer the knowledge acquired in simulation into the real world. To that end, we describe the architectural discrepancies that damage the Sim2Real adaptation ability of models trained on the Habitat simulator and propose a novel solution tailored towards the deployment in real-world scenarios. We then deploy our models on a LoCoBot, a Low-Cost Robot equipped with a single Intel RealSense camera. Different from previous work, our testing scene is unavailable to the agent in simulation. The environment is also inaccessible to the agent beforehand, so it cannot count on scene-specific semantic priors. In this way, we reproduce a setting in which a research group (potentially from other fields) needs to employ the agent visual navigation capabilities as-a-Service. Our experiments indicate that it is possible to achieve satisfying results when deploying the obtained model in the real world. Our code and models are available at https://github.com/aimagelab/LoCoNav.

arXiv:2203.07618공식 코드 · Papers with Code

Do Language Models Plagiarize?

Past literature has illustrated that language models (LMs) often memorize parts of training instances and reproduce them in natural language generation (NLG) processes. However, it is unclear to what extent LMs "reuse" a training corpus. For instance, models can generate paraphrased sentences that are contextually similar to training samples. In this work, therefore, we study three types of plagiarism (i.e., verbatim, paraphrase, and idea) among GPT-2 generated texts, in comparison to its training data, and further analyze the plagiarism patterns of fine-tuned LMs with domain-specific corpora which are extensively used in practice. Our results suggest that (1) three types of plagiarism widely exist in LMs beyond memorization, (2) both size and decoding methods of LMs are strongly associated with the degrees of plagiarism they exhibit, and (3) fine-tuned LMs' plagiarism patterns vary based on their corpus similarity and homogeneity. Given that a majority of LMs' training data is scraped from the Web without informing content owners, their reiteration of words, phrases, and even core ideas from training sets into generated texts has ethical implications. Their patterns are likely to exacerbate as both the size of LMs and their training data increase, raising concerns about indiscriminately pursuing larger models with larger training corpora. Plagiarized content can also contain individuals' personal and sensitive information. These findings overall cast doubt on the practicality of current LMs in mission-critical writing tasks and urge more discussions around the observed phenomena. Data and source code are available at https://github.com/Brit7777/LM-plagiarism.

arXiv:2205.05920공식 코드 · Papers with Code

Group R-CNN for Weakly Semi-supervised Object Detection with Points

We study the problem of weakly semi-supervised object detection with points (WSSOD-P), where the training data is combined by a small set of fully annotated images with bounding boxes and a large set of weakly-labeled images with only a single point annotated for each instance. The core of this task is to train a point-to-box regressor on well-labeled images that can be used to predict credible bounding boxes for each point annotation. We challenge the prior belief that existing CNN-based detectors are not compatible with this task. Based on the classic R-CNN architecture, we propose an effective point-to-box regressor: Group R-CNN. Group R-CNN first uses instance-level proposal grouping to generate a group of proposals for each point annotation and thus can obtain a high recall rate. To better distinguish different instances and improve precision, we propose instance-level proposal assignment to replace the vanilla assignment strategy adopted in the original R-CNN methods. As naive instance-level assignment brings converging difficulty, we propose instance-aware representation learning which consists of instance-aware feature enhancement and instance-aware parameter generation to overcome this issue. Comprehensive experiments on the MS-COCO benchmark demonstrate the effectiveness of our method. Specifically, Group R-CNN significantly outperforms the prior method Point DETR by 3.9 mAP with 5% well-labeled images, which is the most challenging scenario. The source code can be found at https://github.com/jshilong/GroupRCNN

arXiv:2305.11769공식 코드 · Papers with Code

Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner

Large pre-trained multimodal models have demonstrated significant success in a range of downstream tasks, including image captioning, image-text retrieval, visual question answering (VQA), etc. However, many of these methods rely on image-text pairs collected from the web as pre-training data and unfortunately overlook the need for fine-grained feature alignment between vision and language modalities, which requires detailed understanding of images and language expressions. While integrating VQA and dense captioning (DC) into pre-training can address this issue, acquiring image-question-answer as well as image-location-caption triplets is challenging and time-consuming. Additionally, publicly available datasets for VQA and dense captioning are typically limited in scale due to manual data collection and labeling efforts. In this paper, we propose a novel method called Joint QA and DC GEneration (JADE), which utilizes a pre-trained multimodal model and easily-crawled image-text pairs to automatically generate and filter large-scale VQA and dense captioning datasets. We apply this method to the Conceptual Caption (CC3M) dataset to generate a new dataset called CC3M-QA-DC. Experiments show that when used for pre-training in a multi-task manner, CC3M-QA-DC can improve the performance with various backbones on various downstream tasks. Furthermore, our generated CC3M-QA-DC can be combined with larger image-text datasets (e.g., CC15M) and achieve competitive results compared with models using much more data. Code and dataset are available at https://github.com/johncaged/OPT_Questioner.

arXiv:2308.09708공식 코드 · Papers with Code

Training with Product Digital Twins for AutoRetail Checkout

Automating the checkout process is important in smart retail, where users effortlessly pass products by hand through a camera, triggering automatic product detection, tracking, and counting. In this emerging area, due to the lack of annotated training data, we introduce a dataset comprised of product 3D models, which allows for fast, flexible, and large-scale training data generation through graphic engine rendering. Within this context, we discern an intriguing facet, because of the user "hands-on" approach, bias in user behavior leads to distinct patterns in the real checkout process. The existence of such patterns would compromise training effectiveness if training data fail to reflect the same. To address this user bias problem, we propose a training data optimization framework, i.e., training with digital twins (DtTrain). Specifically, we leverage the product 3D models and optimize their rendering viewpoint and illumination to generate "digital twins" that visually resemble representative user images. These digital twins, inherit product labels and, when augmented, form the Digital Twin training set (DT set). Because the digital twins individually mimic user bias, the resulting DT training set better reflects the characteristics of the target scenario and allows us to train more effective product detection and tracking models. In our experiment, we show that DT set outperforms training sets created by existing dataset synthesis methods in terms of counting accuracy. Moreover, by combining DT set with pseudo-labeled real checkout data, further improvement is observed. The code is available at https://github.com/yorkeyao/Automated-Retail-Checkout.

arXiv:2310.09192공식 코드 · Papers with Code

Does Graph Distillation See Like Vision Dataset Counterpart?

Training on large-scale graphs has achieved remarkable results in graph representation learning, but its cost and storage have attracted increasing concerns. Existing graph condensation methods primarily focus on optimizing the feature matrices of condensed graphs while overlooking the impact of the structure information from the original graphs. To investigate the impact of the structure information, we conduct analysis from the spectral domain and empirically identify substantial Laplacian Energy Distribution (LED) shifts in previous works. Such shifts lead to poor performance in cross-architecture generalization and specific tasks, including anomaly detection and link prediction. In this paper, we propose a novel Structure-broadcasting Graph Dataset Distillation (SGDD) scheme for broadcasting the original structure information to the generation of the synthetic one, which explicitly prevents overlooking the original structure information. Theoretically, the synthetic graphs by SGDD are expected to have smaller LED shifts than previous works, leading to superior performance in both cross-architecture settings and specific tasks. We validate the proposed SGDD across 9 datasets and achieve state-of-the-art results on all of them: for example, on the YelpChi dataset, our approach maintains 98.6% test accuracy of training on the original graph dataset with 1,000 times saving on the scale of the graph. Moreover, we empirically evaluate there exist 17.6% ~ 31.4% reductions in LED shift crossing 9 datasets. Extensive experiments and analysis verify the effectiveness and necessity of the proposed designs. The code is available in the GitHub repository: https://github.com/RingBDStack/SGDD.

arXiv:2406.12072공식 코드 · Papers with Code

DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphs

Dynamic text-attributed graphs (DyTAGs) are prevalent in various real-world scenarios, where each node and edge are associated with text descriptions, and both the graph structure and text descriptions evolve over time. Despite their broad applicability, there is a notable scarcity of benchmark datasets tailored to DyTAGs, which hinders the potential advancement in many research fields. To address this gap, we introduce Dynamic Text-attributed Graph Benchmark (DTGB), a collection of large-scale, time-evolving graphs from diverse domains, with nodes and edges enriched by dynamically changing text attributes and categories. To facilitate the use of DTGB, we design standardized evaluation procedures based on four real-world use cases: future link prediction, destination node retrieval, edge classification, and textual relation generation. These tasks require models to understand both dynamic graph structures and natural language, highlighting the unique challenges posed by DyTAGs. Moreover, we conduct extensive benchmark experiments on DTGB, evaluating 7 popular dynamic graph learning algorithms and their variants of adapting to text attributes with LLM embeddings, along with 6 powerful large language models (LLMs). Our results show the limitations of existing models in handling DyTAGs. Our analysis also demonstrates the utility of DTGB in investigating the incorporation of structural and textual dynamics. The proposed DTGB fosters research on DyTAGs and their broad applications. It offers a comprehensive benchmark for evaluating and advancing models to handle the interplay between dynamic graph structures and natural language. The dataset and source code are available at https://github.com/zjs123/DTGB.

arXiv:2406.16218

Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs

We study a class of optimization problems motivated by automating the design and update of AI systems like coding assistants, robots, and copilots. AutoDiff frameworks, like PyTorch, enable efficient end-to-end optimization of differentiable systems. However, general computational workflows can be non-differentiable and involve rich feedback (e.g. console output or user's responses), heterogeneous parameters (e.g. prompts, codes), and intricate objectives (beyond maximizing a score). We investigate end-to-end generative optimization -- using generative models such as LLMs within the optimizer for automatic updating of general computational workflows. We discover that workflow execution traces are akin to back-propagated gradients in AutoDiff and can provide key information to interpret feedback for efficient optimization. Formally, we frame a new mathematical setup, Optimization with Trace Oracle (OPTO). In OPTO, an optimizer receives an execution trace along with feedback on the computed output and updates parameters iteratively. We provide a Python library, Trace, that efficiently converts a workflow optimization problem into an OPTO instance using PyTorch-like syntax. Using Trace, we develop a general LLM-based generative optimizer called OptoPrime. In empirical studies, we find that OptoPrime is capable of first-order numerical optimization, prompt optimization, hyper-parameter tuning, robot controller design, code debugging, etc., and is often competitive with specialized optimizers for each domain. We envision Trace as an open research platform for devising novel generative optimizers and developing the next generation of interactive learning agents. Website: https://microsoft.github.io/Trace/.

arXiv:2407.11398

Animate3D: Animating Any 3D Model with Multi-view Video Diffusion

Recent advances in 4D generation mainly focus on generating 4D content by distilling pre-trained text or single-view image-conditioned models. It is inconvenient for them to take advantage of various off-the-shelf 3D assets with multi-view attributes, and their results suffer from spatiotemporal inconsistency owing to the inherent ambiguity in the supervision signals. In this work, we present Animate3D, a novel framework for animating any static 3D model. The core idea is two-fold: 1) We propose a novel multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, which is trained on our presented large-scale multi-view video dataset (MV-Video). 2) Based on MV-VDM, we introduce a framework combining reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the multi-view video diffusion priors for animating 3D objects. Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. Additionally, we leverage the static 3D model's multi-view renderings as conditions to preserve its identity. For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct motions directly from generated multi-view videos, followed by the introduced 4D-SDS to refine both appearance and motion. Benefiting from accurate motion learning, we could achieve straightforward mesh animation. Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches. Data, code, and models will be open-released.

arXiv:2407.14500공식 코드 · Papers with Code

ViLLa: Video Reasoning Segmentation with Large Language Model

Although video perception models have made remarkable advancements in recent years, they still heavily rely on explicit text descriptions or pre-defined categories to identify target instances before executing video perception tasks. These models, however, fail to proactively comprehend and reason the user's intentions via textual input. Even though previous works attempt to investigate solutions to incorporate reasoning with image segmentation, they fail to reason with videos due to the video's complexity in object motion. To bridge the gap between image and video, in this work, we propose a new video segmentation task - video reasoning segmentation. The task is designed to output tracklets of segmentation masks given a complex input text query. What's more, to promote research in this unexplored area, we construct a reasoning video segmentation benchmark. Finally, we present ViLLa: Video reasoning segmentation with a Large Language Model, which incorporates the language generation capabilities of multimodal Large Language Models (LLMs) while retaining the capabilities of detecting, segmenting, and tracking multiple instances. We use a temporal-aware context aggregation module to incorporate contextual visual cues to text embeddings and propose a video-frame decoder to build temporal correlations across segmentation tokens. Remarkably, our ViLLa demonstrates capability in handling complex reasoning and referring video segmentation. Also, our model shows impressive ability in different temporal understanding benchmarks. Both quantitative and qualitative experiments show our method effectively unlocks new video reasoning segmentation capabilities for multimodal LLMs. The code and dataset will be available at https://github.com/rkzheng99/ViLLa.

arXiv:2408.00550공식 코드 · Papers with Code

Mitigating Multilingual Hallucination in Large Vision-Language Models

While Large Vision-Language Models (LVLMs) have exhibited remarkable capabilities across a wide range of tasks, they suffer from hallucination problems, where models generate plausible yet incorrect answers given the input image-query pair. This hallucination phenomenon is even more severe when querying the image in non-English languages, while existing methods for mitigating hallucinations in LVLMs only consider the English scenarios. In this paper, we make the first attempt to mitigate this important multilingual hallucination in LVLMs. With thorough experiment analysis, we found that multilingual hallucination in LVLMs is a systemic problem that could arise from deficiencies in multilingual capabilities or inadequate multimodal abilities. To this end, we propose a two-stage Multilingual Hallucination Removal (MHR) framework for LVLMs, aiming to improve resistance to hallucination for both high-resource and low-resource languages. Instead of relying on the intricate manual annotations of multilingual resources, we fully leverage the inherent capabilities of the LVLM and propose a novel cross-lingual alignment method, which generates multiple responses for each image-query input and then identifies the hallucination-aware pairs for each language. These data pairs are finally used for direct preference optimization to prompt the LVLMs to favor non-hallucinating responses. Experimental results show that our MHR achieves a substantial reduction in hallucination generation for LVLMs. Notably, on our extended multilingual POPE benchmark, our framework delivers an average increase of 19.0% in accuracy across 13 different languages. Our code and model weights are available at https://github.com/ssmisya/MHR

arXiv:2409.05122공식 코드 · Papers with Code

PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation

Semi-supervised learning has emerged as a widely adopted technique in the field of medical image segmentation. The existing works either focuses on the construction of consistency constraints or the generation of pseudo labels to provide high-quality supervisory signals, whose main challenge mainly comes from how to keep the continuous improvement of model capabilities. In this paper, we propose a simple yet effective semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation, whose goal is to generate high-fidelity pseudo labels by learning robust and diverse features in the training process. Specifically, our PMT employs a standard mean teacher to penalize the consistency of the current state and utilizes two sets of MT architectures for co-training. The two sets of MT architectures are individually updated for prolonged periods to maintain stable model diversity established through performance gaps generated by iteration differences. Additionally, a difference-driven alignment regularizer is employed to expedite the alignment of lagging models with the representation capabilities of leading models. Furthermore, a simple yet effective pseudo-label filtering algorithm is employed for facile evaluation of models and selection of high-fidelity pseudo-labels outputted when models are operating at high performance for co-training purposes. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches across various dimensions. The code is available at https://github.com/Axi404/PMT.

arXiv:2409.12448공식 코드 · Papers with Code

Infrared Small Target Detection in Satellite Videos: A New Dataset and A Novel Recurrent Feature Refinement Framework

Multi-frame infrared small target (MIRST) detection in satellite videos is a long-standing, fundamental yet challenging task for decades, and the challenges can be summarized as: First, extremely small target size, highly complex clutters & noises, various satellite motions result in limited feature representation, high false alarms, and difficult motion analyses. Second, the lack of large-scale public available MIRST dataset in satellite videos greatly hinders the algorithm development. To address the aforementioned challenges, in this paper, we first build a large-scale dataset for MIRST detection in satellite videos (namely IRSatVideo-LEO), and then develop a recurrent feature refinement (RFR) framework as the baseline method. Specifically, IRSatVideo-LEO is a semi-simulated dataset with synthesized satellite motion, target appearance, trajectory and intensity, which can provide a standard toolbox for satellite video generation and a reliable evaluation platform to facilitate the algorithm development. For baseline method, RFR is proposed to be equipped with existing powerful CNN-based methods for long-term temporal dependency exploitation and integrated motion compensation & MIRST detection. Specifically, a pyramid deformable alignment (PDA) module and a temporal-spatial-frequency modulation (TSFM) module are proposed to achieve effective and efficient feature alignment, propagation, aggregation and refinement. Extensive experiments have been conducted to demonstrate the effectiveness and superiority of our scheme. The comparative results show that ResUNet equipped with RFR outperforms the state-of-the-art MIRST detection methods. Dataset and code are released at https://github.com/XinyiYing/RFR.

arXiv:2410.20843공식 코드 · Papers with Code

Generative Simulations of The Solar Corona Evolution With Denoising Diffusion : Proof of Concept

The solar magnetized corona is responsible for various manifestations with a space weather impact, such as flares, coronal mass ejections (CMEs) and, naturally, the solar wind. Modeling the corona's dynamics and evolution is therefore critical for improving our ability to predict space weather In this work, we demonstrate that generative deep learning methods, such as Denoising Diffusion Probabilistic Models (DDPM), can be successfully applied to simulate future evolutions of the corona as observed in Extreme Ultraviolet (EUV) wavelengths. Our model takes a 12-hour video of an Active Region (AR) as input and simulate the potential evolution of the AR over the subsequent 12 hours, with a time-resolution of two hours. We propose a light UNet backbone architecture adapted to our problem by adding 1D temporal convolutions after each classical 2D spatial ones, and spatio-temporal attention in the bottleneck part. The model not only produce visually realistic outputs but also captures the inherent stochasticity of the system's evolution. Notably, the simulations enable the generation of reliable confidence intervals for key predictive metrics such as the EUV peak flux and fluence of the ARs, paving the way for probabilistic and interpretable space weather forecasting. Future studies will focus on shorter forecasting horizons with increased spatial and temporal resolution, aiming at reducing the uncertainty of the simulations and providing practical applications for space weather forecasting. The code used for this study is available at the following link: https://github.com/gfrancisco20/video_diffusion

arXiv:2411.13901

Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel KAN Adapter for Enhanced Feature Adaptation

Specialized datasets that capture the fashion industry's rich language and styling elements can boost progress in AI-driven fashion design. We present FLORA (Fashion Language Outfit Representation for Apparel Generation), the first comprehensive dataset containing 4,330 curated pairs of fashion outfits and corresponding textual descriptions. Each description utilizes industry-specific terminology and jargon commonly used by professional fashion designers, providing precise and detailed insights into the outfits. Hence, the dataset captures the delicate features and subtle stylistic elements necessary to create high-fidelity fashion designs. We demonstrate that fine-tuning generative models on the FLORA dataset significantly enhances their capability to generate accurate and stylistically rich images from textual descriptions of fashion sketches. FLORA will catalyze the creation of advanced AI models capable of comprehending and producing subtle, stylistically rich fashion designs. It will also help fashion designers and end-users to bring their ideas to life. As a second orthogonal contribution, we introduce KAN Adapters, which leverage Kolmogorov-Arnold Networks (KAN) as adaptive modules. They serve as replacements for traditional MLP-based LoRA adapters. With learnable spline-based activations, KAN Adapters excel in modeling complex, non-linear relationships, achieving superior fidelity, faster convergence and semantic alignment. Extensive experiments and ablation studies on our proposed FLORA dataset validate the superiority of KAN Adapters over LoRA adapters. To foster further research and collaboration, we will open-source both the FLORA and our implementation code.