PyTorchKR Papers

“Code Generation”

총 10,186편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2611.02654v1cs.CV

VerdictAD: 판정 기반 정제를 통한 소수 샷 이상 탐지

VerdictAD: Few-Shot Anomaly Detection via Decision-Guided Refinement

실제 산업 검사에서 결함을 탐지하려면 범주당 소수의 정상 이미지만으로 학습해야 하는 경우가 많다. 최근의 소수 샷 이상 탐지 방법은 질의 이미지와 정상 참조 이미지 간의 특징 차이를 측정하여 강력한 성능을 달성하였다. 그러나 시각적 차이가 항상 진짜 결함을 나타내는 것은 아니며, 정상적인 외형 변이도 높은 이상 점수를 낳을 수 있다. 반대로 구조적 또는 논리적 이상은 강한 국소 특징 차이를 만들지 않아 탐지하기 어려울 수 있다. 이러한 한계를 해결하기 위해 우리는 시각적 판정을 통해 기존 이상 탐지기를 정제하는 학습이 필요 없는 프레임워크인 VerdictAD를 소개한다. VerdictAD는 기존 탐지기를 대체하는 대신, 유형화된 판정 모델을 채택하여 소수의 정상 참조 이미지에 대해 질의 이미지와 그 국소 영역을 평가한다. 텍스트 생성 없이 구조화된 판정 확률을 직접 산출함으로써, 이 모델은 서로 다른 공간 규모에서 이상 응답을 정제하기 위한 보완적 증거를 제공한다. 이를 바탕으로 두 가지 모듈을 도입한다. 전역 판정 정제(GVR)는 이미지 수준 판정을 사용하여 전체 이상 점수를 정제하고, 국소 판정 정제(LVR)는 영역별 판정을 집계하여 이상 맵을 정제한다. 간단한 DINOv2 기반 탐지기와 결합했을 때 VerdictAD는 원샷 및 소수 샷 이상 탐지에서 경쟁력 있는 성능을 달성한다. 더 중요하게는, 플러그 앤 플레이 프레임워크로서 MVTec AD, VisA, MVTec LOCO AD 전반에서 기존 이상 탐지기를 일관되게 개선하며 원샷 및 소수 샷 설정에서 강력한 결과를 달성한다. 코드와 실험 로그는 https://github.com/sgstq0811/VerdictAD-Jev-AD 에 공개되어 있다.

arXiv:2611.04498v1cs.CV

3DGS-VBench+와 3DGSV-Assessor: 3D 가우시안 스플래팅 영상 품질 평가를 위한 다차원 주관적 벤치마크와 참조 유연형 객관적 모델

3DGS-VBench+ and 3DGSV-Assessor: A Multi-Dimensional Subjective Benchmark and a Reference-Flexible Objective Model for 3D Gaussian Splatting Video Quality Assessment

3D 가우시안 스플래팅(3DGS)은 실시간 새로운 시점 합성(NVS)을 가능하게 하지만, 저장 용량이 크기 때문에 압축이 필수적이며 학습과 압축 모두 표현 방식에 특유한 왜곡을 유발한다. 많은 방법에서 생성적 압축은 기하 속성과 색상 속성을 서로 다른 매개변수로 처리하므로 차원별 왜곡을 일으킬 수 있으며, 이는 다차원 평가의 필요성을 시사한다. 또한 실제 촬영 데이터에는 합성된 연속 카메라 궤적에 정렬된 정답 영상이 일반적으로 없어, 희소하게 촬영된 시점만이 참조로 남는다. 정렬된 참조가 부족한 3DGS 영상의 다차원 품질 평가를 가능하게 하기 위해, 우리는 먼저 3DGS-VBench+를 제시한다. 이는 대규모 주관적 영상 품질 평가(VQA) 벤치마크로, 실제 장면 11개와 합성 장면 22개에 걸쳐 3{,}036개의 3DGS 재구성을 포함하고 각각에 렌더링된 영상이 짝지어져 있다. 이 재구성은 6가지 대표적 압축 알고리즘을 체계적으로 설계한 수준에서 적용하여 생성되었으며, 전체, 기하, 색상 품질에 대한 9{,}108개의 MOS가 주석으로 달려 있다. 3DGS-VBench+를 바탕으로 우리는 3DGSV-Assessor를 제안한다. 이는 참조 유연형 대규모 멀티모달 프레임워크로, 세 가지 차원을 한 번의 생성 과정으로 모두 점수화하고, 순위 기반 참조 단서를 제한된 게이트를 통해 고정된 회귀기에 융합하여 감축 참조(RR)와 무참조(NR) 평가를 모두 지원한다. 광범위한 실험을 통해 3DGS-VBench+에서 인간 지각과의 최첨단 상관성을 보이고, 다른 NVS 벤치마크에서도 강한 제로샷 일반화 성능을 보임을 입증한다. 데이터셋과 코드는 https://github.com/YukeXing/3DGSV-Assessor 에서 공개될 예정이다.

arXiv:2611.08918v1cs.LG

구성 가능한 인과성: 체계적인 시계열 인과 발견 및 처치 효과 벤치마킹을 위한 툴킷

Composable Causality: A Toolkit for Systematic Time-Series Causal Discovery and Treatment-Effect Benchmarking

실제 다변량 시계열은 검증된 정답 인과 구조를 갖추고 있는 경우가 드물어, 시계열 인과 발견과 처치 효과 추정 연구는 합성 데이터에 의존한다. 기존 도구는 공개 후 수정할 수 없는 고정 데이터셋이거나, 각 스크립트가 메커니즘, 교락, 결측의 한 가지 조합을 고정하는 스크립트 기반 생성기이다. 어느 쪽도 희소성, 노이즈 분포, 결측 메커니즘과 같은 단일 속성만 변화시키고 나머지는 고정하는 통제된 실험을 지원하지 않는다. 반사실 평가에도 같은 공백이 있는데, 기존 시뮬레이터는 논문마다 하드코딩된 메커니즘으로 만들어지거나 관측 데이터만 생성하기 때문이다. 우리는 정답 동시점 및 지연 인접 행렬을 가진 다변량 시계열을 생성하는 데이터 생성 라이브러리 DTCD(Datasets and Toolkit for Causal Discovery)를 소개한다. 위상, 메커니즘, 노이즈, 관측 마스크, 개입 정책은 독립적인 구성요소이며 설정만으로 조합된다. 사실 롤아웃 중 확률적 실현을 캐시하고 개입 롤아웃 중 다시 주입하여 만든 쌍을 이루는 반사실 쌍둥이는 개별 처치 효과를 수치 정밀도로 복원한다. 81개 구성에 대한 8개의 발견 알고리즘과 18개 구성에 대한 4개의 처치 효과 추정기로 이루어진 참조 벤치마크는 집계 지표가 가리는 실패 양상을 드러낸다. 여기에는 결측 메커니즘이 정보성을 갖는지와 무관하게 표준 발견 방법이 결측값을 처리하지 못하는 점, 그리고 우리가 측정하고 표준화로 제거한 주변 분산 지름길(varsortability)이 포함된다. 코드와 데이터: https://github.com/hferdous/Composable_Causality

arXiv:2607.08681v1cs.AI

SolarChain-Eval: 분산 에너지 시장에서 신뢰할 수 있는 경제 에이전트를 위한 물리 제약 벤치마크

SolarChain-Eval: A Physics-Constrained Benchmark for Trustworthy Economic Agents in Decentralized Energy Markets

분산 에너지 시장에서 강화학습(RL) 에이전트가 물리적 제약을 위반하거나 인위적 유동성을 생성하는 신뢰성 문제를 해결하기 위해, SolarChain-Eval이라는 물리 제약 기반 벤치마크를 제안하며 시장 효용, 물리적 안전성, 공정성, 감사 가능성 등 다차원 평가를 Gymnasium 호환 마르코프 결정 과정(MDP)으로 정식화한다. LLM 기반 Planner/Auditor 레이어를 통합한 실험 결과, RL 에이전트는 시장 효용을 개선하지만 물리 페널티 제거 시 유효하지 않은 발전량을 악용하며, LLM 감사자는 감사 가능성을 높이지만 잘못 설계된 보상 함수를 완전히 보완하지는 못함을 보였다.

arXiv:2510.20535v2cs.CL

ARC-Encoder: learning compressed text representations for large language models

Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs. Context compression techniques can reduce these costs, but the most effective approaches require fine-tuning the target model or even modifying its architecture. This can degrade its general abilities when not used for this specific purpose. Here we explore an alternative approach: an encoder that compresses the context into continuous representations which replace token embeddings in decoder LLMs. First, we perform a systematic study of training strategies and architecture choices for the encoder. Our findings led to the design of an Adaptable text Representations Compressor, named ARC-Encoder, which outputs $x$-times fewer continuous representations (typically $x\!\in\!\{4,8\}$) than text tokens. We evaluate ARC-Encoder across a variety of LLM usage scenarios, ranging from in-context learning to context window extension, on both instruct and base decoders. Results show that ARC-Encoder achieves state-of-the-art performance on several benchmarks while improving computational efficiency at inference. Finally, we demonstrate that our models can be adapted to multiple decoders simultaneously, allowing a single encoder to generalize across different decoder LLMs. This makes ARC-Encoder a flexible and efficient solution for portable encoders that work seamlessly with multiple LLMs. We release a training code at https://github.com/kyutai-labs/ARC-Encoder , fine-tuning dataset and pretrained models are available at https://huggingface.co/collections/kyutai/arc-encoders-68ee18787301407d60a57047 .

arXiv:2608.00107v1cs.LG

MetaRoute-Bench: 에이전트 워크플로우 라우팅을 위한 메타 의사결정 정책 평가

MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing

에이전트(agent) 시스템은 직접 응답할지, 작업을 분해할지, 도구를 호출할지, 코드를 실행할지, 전문가에게 위임할지, 중간 결과를 검증할지, 아니면 실패로부터 복구할지를 반복적으로 결정해야 합니다. 이러한 메타 의사결정은 작업 성공뿐만 아니라 운영 비용 및 지연 시간에도 영향을 미치지만, 종종 오케스트레이션 프레임워크 내부에 내장되어 총체적인 작업 정확도를 통해서만 평가되곤 합니다. 본 논문에서는 공유 실행 모델 하에서 메타 의사결정 정책을 비교하기 위한 공개적이고 검사 가능한 프레임워크인 MetaRoute-Bench를 제시합니다. 초기 벤치마크는 데이터 분석, 연구, 문서 처리에 걸친 180개의 합성 작업 프로필, 8개의 라우팅 정책, 30개의 대응 무작위 시드(seed)를 포함합니다. 43,200개의 추적 전반에서, 작업 인식 조합 정책은 강력한 워크로드 특정 정적 정책의 76.7%, 원샷 작업 라우팅의 67.4%, 직접 응답의 52.9%에 비해 79.4%의 성공률을 달성합니다. 정적 정책 대비 이는 4.7% 더 높은 평균 비용과 6.4% 더 높은 지연 시간에서, 대응 95% 신뢰구간(CI) ±2.0%포인트를 동반한 2.7%포인트의 향상입니다. 절제 연구(ablation) 결과, 경로 조합이 단일 연산으로 제한되거나 검증이 제거될 때 가장 큰 성능 저하가 나타났습니다. 이러한 결과는 실제 배포 환경이 아닌 시드가 지정된 오프라인 실행 모델에 의해 생성되었으므로, 본 연구의 주요 기여는 프로덕션 환경에서의 효과성에 대한 증거가 아니라 재현 가능한 평가 방법론 및 라우팅 정책의 트레이드오프 분석입니다. 저자들은 라이브 시스템 검증을 지원하기 위해 작업 생성, 정책, 추적, 테스트 및 분석 결과물을 공개합니다.

arXiv:2608.21234v1math.NA

응용을 아우른 고급 선형대수 - 제1부 (PDE, 기계학습, 데이터 동화를 위한 수치 선형대수)

Advanced Linear Algebra with Applications - Part I (Numerical linear algebra for PDEs, machine learning, and data assimilation)

이 강의 노트는 고급 수치 선형대수에 관한 석사 수준 강좌의 첫 번째 부분을 구성한다. 목적은 고전적 알고리즘을 소개하는 데 그치지 않고, 이 주제가 한 세대 전보다 훨씬 더 중심적이 된 이유를 보여 주는 것이다. 수치 선형대수는 편미분방정식(PDE)의 수치 해법과 함께 발전해 왔으며, 오랫동안 그곳이 대규모 희소 시스템의 출처였다. 네트워크 노드의 순위 매기기, 관측을 일기 예보에 동화하기, 대규모 잡음이 있는 데이터셋에 모델을 적합시키기 같은 문제들이 이제 같은 종류의 문제로 이어진다. 즉, 인수분해하기에는 너무 크고, 구조를 갖으며, 행렬-벡터 곱(matrix-vector product)을 통해서만 접근 가능하다. 이들 모두에 필요한 아이디어는 놀랄 만큼 적다. 따라서 각 장은 표준 주제를 전개한 뒤, 원래 맥락 밖에서 이를 적용한다. 규범(norm), 분해(factorisation), 조건수(conditioning), 부동소수점 연산; 유한 차분, 그래프, 기계학습에서 기인하는 희소 행렬; 정적 반복(stationary iteration)과 smoothing property; conjugate gradient와 Lanczos 방법, 스펙트럴 클러스터링(spectral clustering) 및 조기 종료(early stopping)에 의한 정규화(regularisation); Arnoldi와 GMRES, PageRank 및 대규모 최소제곱(least squares); 마지막으로 preconditioning, Schwarz 영역 분할(domain decomposition), multigrid를 다룬다. 선형대수 입문 과정을 선수 지식으로 가정한다. 각 절은 기억해야 할 내용의 요약으로, 각 장은 연습문제로 마무리되며, 그중 여러 문제는 과거 시험에서 가져왔다. 함께 제공되는 Python 코드가 수치 예시를 재현한다.

arXiv:2608.23663v1cs.SE

자신 있게 틀리고, 조용히: 배포된 온디바이스 언어 모델의 탐지 불가능한 실패 감사

Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model

배포된 언어 모델의 정렬(alignment)은 출력을 언제 신뢰할 수 있는지 아는 것을 요구하지만, 온디바이스 모델은 이제 서버 측 검열 없이 수억 대의 기기에 탑재되며, 개발자가 실제로 배포할 수 있는 구성은 독립적으로 감사되는 경우가 드뭅니다. 우리는 감독(oversight) 질문으로 프레이밍한, 개발자가 접근 가능한 온디바이스 기반(foundation) 모델에 대한 재현 가능한 신뢰성 감사를 제시합니다. 사용자나 자원이 제한된 개발자가 모델이 틀렸을 때 이를 알 수 있는가? 캘리브레이션(calibration), 거짓 전제(false-premise) 질문에서의 자신 있게 하는 날조(confident confabulation), 양성(benign) 프롬프트에 대한 과도한 거부(over-refusal)에 대해 레드팀 테스팅(red-teaming)한 결과, \emph{작업 비대칭적 miscalibration}을 발견합니다. 작업마다 가드레일(guardrails)이 상반된 방향으로 실패합니다(거짓 전제의 69\%에서 날조하면서 완전히 양성인 입력의 18\%는 거부). 이는 포화되어 변별력이 없는 자기 보고 신뢰도(self-reported confidence) 위에 놓여 있습니다(AUROC 0.47; ECE 70, 유사한 소형 모델 중 최악). 결정적으로, 자신 있게 맞은 출력과 자신 있게 틀린 출력은 \emph{표면상 구분 불가}(surface-indistinguishable)합니다. 15개의 사용자에게 보이는 특징에 대한 분류기가 AUROC 0.55로만 구분하며(동등성 확인됨), 추론 시점에 감독을 위한 신호를 남기지 않습니다. 값싼 단일 생성(single-generation) 신호는 이러한 실패를 표시하지 못합니다($\le$0.68 AUROC). 반면 모델 접근이 필요 없는 블랙박스 일관성 래퍼(black-box consistency wrapper)는 조정 가능한 비용으로 신뢰성을 회복합니다(자신 있게 하는 날조 75\%$\to$3\%; 선택적 정확도(selective accuracy) 43\%$\to$83\%). 우리는 모델 불가지론적(model-agnostic) 감사 프로토콜, 표면 구분 불가능성(surface-indistinguishability) 테스트, 그리고 배포된 모델 감사를 위한 재사용 가능한 인프라로 공개된 코드와 고정(frozen) 평가 항목을 기여합니다.

arXiv:1911.01346

CloudifierNet -- Deep Vision Models for Artificial Image Processing

Today, more and more, it is necessary that most applications and documents developed in previous or current technologies to be accessible online on cloud-based infrastructures. That is why the migration of legacy systems including their hosts of documents to new technologies and online infrastructures, using modern Artificial Intelligence techniques, is absolutely necessary. With the advancement of Artificial Intelligence and Deep Learning with its multitude of applications, a new area of research is emerging - that of automated systems development and maintenance. The underlying work objective that led to this paper aims to research and develop truly intelligent systems able to analyze user interfaces from various sources and generate real and usable inferences ranging from architecture analysis to actual code generation. One key element of such systems is that of artificial scene detection and analysis based on deep learning computer vision systems. Computer vision models and particularly deep directed acyclic graphs based on convolutional modules are generally constructed and trained based on natural images datasets. Due to this fact, the models will develop during the training process natural image feature detectors apart from the base graph modules that will learn basic primitive features. In the current paper, we will present the base principles of a deep neural pipeline for computer vision applied to artificial scenes (scenes generated by user interfaces or similar). Finally, we will present the conclusions based on experimental development and benchmarking against state-of-the-art transfer-learning implemented deep vision models.

arXiv:2006.06325공식 코드 · Papers with Code

CoMIR: Contrastive Multimodal Image Representation for Registration

We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations). CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures. CoMIRs reduce the multimodal registration problem to a monomodal one, in which general intensity-based, as well as feature-based, registration algorithms can be applied. The method involves training one neural network per modality on aligned images, using a contrastive loss based on noise-contrastive estimation (InfoNCE). Unlike other contrastive coding methods, used for, e.g., classification, our approach generates image-like representations that contain the information shared between modalities. We introduce a novel, hyperparameter-free modification to InfoNCE, to enforce rotational equivariance of the learnt representations, a property essential to the registration task. We assess the extent of achieved rotational equivariance and the stability of the representations with respect to weight initialization, training set, and hyperparameter settings, on a remote sensing dataset of RGB and near-infrared images. We evaluate the learnt representations through registration of a biomedical dataset of bright-field and second-harmonic generation microscopy images; two modalities with very little apparent correlation. The proposed approach based on CoMIRs significantly outperforms registration of representations created by GAN-based image-to-image translation, as well as a state-of-the-art, application-specific method which takes additional knowledge about the data into account. Code is available at: https://github.com/MIDA-group/CoMIR.

arXiv:2012.11552공식 코드 · Papers with Code

OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning

Learning image representations without human supervision is an important and active research field. Several recent approaches have successfully leveraged the idea of making such a representation invariant under different types of perturbations, especially via contrastive-based instance discrimination training. Although effective visual representations should indeed exhibit such invariances, there are other important characteristics, such as encoding contextual reasoning skills, for which alternative reconstruction-based approaches might be better suited. With this in mind, we propose a teacher-student scheme to learn representations by training a convolutional net to reconstruct a bag-of-visual-words (BoW) representation of an image, given as input a perturbed version of that same image. Our strategy performs an online training of both the teacher network (whose role is to generate the BoW targets) and the student network (whose role is to learn representations), along with an online update of the visual-words vocabulary (used for the BoW targets). This idea effectively enables fully online BoW-guided unsupervised learning. Extensive experiments demonstrate the interest of our BoW-based strategy which surpasses previous state-of-the-art methods (including contrastive-based ones) in several applications. For instance, in downstream tasks such Pascal object detection, Pascal classification and Places205 classification, our method improves over all prior unsupervised approaches, thus establishing new state-of-the-art results that are also significantly better even than those of supervised pre-training. We provide the implementation code at https://github.com/valeoai/obow.

arXiv:2306.03310

LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

Lifelong learning offers a promising paradigm of building a generalist agent that learns and adapts over its lifespan. Unlike traditional lifelong learning problems in image and text domains, which primarily involve the transfer of declarative knowledge of entities and concepts, lifelong learning in decision-making (LLDM) also necessitates the transfer of procedural knowledge, such as actions and behaviors. To advance research in LLDM, we introduce LIBERO, a novel benchmark of lifelong learning for robot manipulation. Specifically, LIBERO highlights five key research topics in LLDM: 1) how to efficiently transfer declarative knowledge, procedural knowledge, or the mixture of both; 2) how to design effective policy architectures and 3) effective algorithms for LLDM; 4) the robustness of a lifelong learner with respect to task ordering; and 5) the effect of model pretraining for LLDM. We develop an extendible procedural generation pipeline that can in principle generate infinitely many tasks. For benchmarking purpose, we create four task suites (130 tasks in total) that we use to investigate the above-mentioned research topics. To support sample-efficient learning, we provide high-quality human-teleoperated demonstration data for all tasks. Our extensive experiments present several insightful or even unexpected discoveries: sequential finetuning outperforms existing lifelong learning methods in forward transfer, no single visual encoder architecture excels at all types of knowledge transfer, and naive supervised pretraining can hinder agents' performance in the subsequent LLDM. Check the website at https://libero-project.github.io for the code and the datasets.

arXiv:2312.11242공식 코드 · Papers with Code

MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL

Recent LLM-based Text-to-SQL methods usually suffer from significant performance degradation on "huge" databases and complex user questions that require multi-step reasoning. Moreover, most existing methods neglect the crucial significance of LLMs utilizing external tools and model collaboration. To address these challenges, we introduce MAC-SQL, a novel LLM-based multi-agent collaborative framework. Our framework comprises a core decomposer agent for Text-to-SQL generation with few-shot chain-of-thought reasoning, accompanied by two auxiliary agents that utilize external tools or models to acquire smaller sub-databases and refine erroneous SQL queries. The decomposer agent collaborates with auxiliary agents, which are activated as needed and can be expanded to accommodate new features or tools for effective Text-to-SQL parsing. In our framework, We initially leverage GPT-4 as the strong backbone LLM for all agent tasks to determine the upper bound of our framework. We then fine-tune an open-sourced instruction-followed model, SQL-Llama, by leveraging Code Llama 7B, to accomplish all tasks as GPT-4 does. Experiments show that SQL-Llama achieves a comparable execution accuracy of 43.94, compared to the baseline accuracy of 46.35 for vanilla GPT-4. At the time of writing, MAC-SQL+GPT-4 achieves an execution accuracy of 59.59 when evaluated on the BIRD benchmark, establishing a new state-of-the-art (SOTA) on its holdout test set (https://github.com/wbbeyourself/MAC-SQL).

arXiv:2404.10688공식 코드 · Papers with Code

Efficient Conditional Diffusion Model with Probability Flow Sampling for Image Super-resolution

Image super-resolution is a fundamentally ill-posed problem because multiple valid high-resolution images exist for one low-resolution image. Super-resolution methods based on diffusion probabilistic models can deal with the ill-posed nature by learning the distribution of high-resolution images conditioned on low-resolution images, avoiding the problem of blurry images in PSNR-oriented methods. However, existing diffusion-based super-resolution methods have high time consumption with the use of iterative sampling, while the quality and consistency of generated images are less than ideal due to problems like color shifting. In this paper, we propose Efficient Conditional Diffusion Model with Probability Flow Sampling (ECDP) for image super-resolution. To reduce the time consumption, we design a continuous-time conditional diffusion model for image super-resolution, which enables the use of probability flow sampling for efficient generation. Additionally, to improve the consistency of generated images, we propose a hybrid parametrization for the denoiser network, which interpolates between the data-predicting parametrization and the noise-predicting parametrization for different noise scales. Moreover, we design an image quality loss as a complement to the score matching loss of diffusion models, further improving the consistency and quality of super-resolution. Extensive experiments on DIV2K, ImageNet, and CelebA demonstrate that our method achieves higher super-resolution quality than existing diffusion-based image super-resolution methods while having lower time consumption. Our code is available at https://github.com/Yuan-Yutao/ECDP.

arXiv:2404.17357공식 코드 · Papers with Code

Simultaneous Tri-Modal Medical Image Fusion and Super-Resolution using Conditional Diffusion Model

In clinical practice, tri-modal medical image fusion, compared to the existing dual-modal technique, can provide a more comprehensive view of the lesions, aiding physicians in evaluating the disease's shape, location, and biological activity. However, due to the limitations of imaging equipment and considerations for patient safety, the quality of medical images is usually limited, leading to sub-optimal fusion performance, and affecting the depth of image analysis by the physician. Thus, there is an urgent need for a technology that can both enhance image resolution and integrate multi-modal information. Although current image processing methods can effectively address image fusion and super-resolution individually, solving both problems synchronously remains extremely challenging. In this paper, we propose TFS-Diff, a simultaneously realize tri-modal medical image fusion and super-resolution model. Specially, TFS-Diff is based on the diffusion model generation of a random iterative denoising process. We also develop a simple objective function and the proposed fusion super-resolution loss, effectively evaluates the uncertainty in the fusion and ensures the stability of the optimization process. And the channel attention module is proposed to effectively integrate key information from different modalities for clinical diagnosis, avoiding information loss caused by multiple image processing. Extensive experiments on public Harvard datasets show that TFS-Diff significantly surpass the existing state-of-the-art methods in both quantitative and visual evaluations. Code is available at https://github.com/XylonXu01/TFS-Diff.

arXiv:2405.19716공식 코드 · Papers with Code

Enhancing Large Vision Language Models with Self-Training on Image Comprehension

Large vision language models (LVLMs) integrate large language models (LLMs) with pre-trained vision encoders, thereby activating the perception capability of the model to understand image inputs for different queries and conduct subsequent reasoning. Improving this capability requires high-quality vision-language data, which is costly and labor-intensive to acquire. Self-training approaches have been effective in single-modal settings to alleviate the need for labeled data by leveraging model's own generation. However, effective self-training remains a challenge regarding the unique visual perception and reasoning capability of LVLMs. To address this, we introduce Self-Training on Image Comprehension (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference dataset for image descriptions using unlabeled images. Preferred responses are generated through a step-by-step prompt, while dis-preferred responses are generated from either corrupted images or misleading prompts. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data and append its self-generated image descriptions to the prompts. We validate the effectiveness of STIC across seven different benchmarks, demonstrating substantial performance gains of 4.0% on average while using 70% less supervised fine-tuning data than the current method. Further studies investigate various components of STIC and highlight its potential to leverage vast quantities of unlabeled images for self-training. Code and data are made publicly available.

arXiv:2406.00834

End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model

Hybrid refractive-diffractive lenses combine the light efficiency of refractive lenses with the information encoding power of diffractive optical elements (DOE), showing great potential as the next generation of imaging systems. However, accurately simulating such hybrid designs is generally difficult, and in particular, there are no existing differentiable image formation models for hybrid lenses with sufficient accuracy. In this work, we propose a new hybrid ray-tracing and wave-propagation (ray-wave) model for accurate simulation of both optical aberrations and diffractive phase modulation, where the DOE is placed between the last refractive surface and the image sensor, i.e. away from the Fourier plane that is often used as a DOE position. The proposed ray-wave model is fully differentiable, enabling gradient back-propagation for end-to-end co-design of refractive-diffractive lens optimization and the image reconstruction network. We validate the accuracy of the proposed model by comparing the simulated point spread functions (PSFs) with theoretical results, as well as simulation experiments that show our model to be more accurate than solutions implemented in commercial software packages like Zemax. We demonstrate the effectiveness of the proposed model through real-world experiments and show significant improvements in both aberration correction and extended depth-of-field (EDoF) imaging. We believe the proposed model will motivate further investigation into a wide range of applications in computational imaging, computational photography, and advanced optical design. Code will be released upon publication.

arXiv:2408.10945공식 코드 · Papers with Code

HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models

High-resolution Vision-Language Models (VLMs) are widely used in multimodal tasks to enhance accuracy by preserving detailed image information. However, these models often generate an excessive number of visual tokens due to the need to encode multiple partitions of a high-resolution image input. Processing such a large number of visual tokens through multiple transformer networks poses significant computational challenges, particularly for resource-constrained commodity GPUs. To address this challenge, we propose High-Resolution Early Dropping (HiRED), a plug-and-play token-dropping method designed to operate within a fixed token budget. HiRED leverages the attention of CLS token in the vision transformer (ViT) to assess the visual content of the image partitions and allocate an optimal token budget for each partition accordingly. The most informative visual tokens from each partition within the allocated budget are then selected and passed to the subsequent Large Language Model (LLM). We showed that HiRED achieves superior accuracy and performance, compared to existing token-dropping methods. Empirically, HiRED-20% (i.e., a 20% token budget) on LLaVA-Next-7B achieves a 4.7x increase in token generation throughput, reduces response latency by 78%, and saves 14% of GPU memory for single inference on an NVIDIA TESLA P40 (24 GB). For larger batch sizes (e.g., 4), HiRED-20% prevents out-of-memory errors by cutting memory usage by 30%, while preserving throughput and latency benefits. Code - https://github.com/hasanar1f/HiRED

arXiv:2408.16314공식 코드 · Papers with Code

ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding

Visual grounding aims to localize the object referred to in an image based on a natural language query. Although progress has been made recently, accurately localizing target objects within multiple-instance distractions (multiple objects of the same category as the target) remains a significant challenge. Existing methods demonstrate a significant performance drop when there are multiple distractions in an image, indicating an insufficient understanding of the fine-grained semantics and spatial relationships between objects. In this paper, we propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ResVG) model, to address this issue. Firstly, we enhance the model's understanding of fine-grained semantics by injecting semantic prior information derived from text queries into the model. This is achieved by leveraging text-to-image generation models to produce images representing the semantic attributes of target objects described in queries. Secondly, we tackle the lack of training samples with multiple distractions by introducing a relation-sensitive data augmentation method. This method generates additional training data by synthesizing images containing multiple objects of the same category and pseudo queries based on their spatial relationships. The proposed ReSVG model significantly improves the model's ability to comprehend both object semantics and spatial relations, leading to enhanced performance in visual grounding tasks, particularly in scenarios with multiple-instance distractions. We conduct extensive experiments to validate the effectiveness of our methods on five datasets. Code is available at https://github.com/minghangz/ResVG.

arXiv:2409.03062공식 코드 · Papers with Code

MobileUNETR: A Lightweight End-To-End Hybrid Vision Transformer For Efficient Medical Image Segmentation

Skin cancer segmentation poses a significant challenge in medical image analysis. Numerous existing solutions, predominantly CNN-based, face issues related to a lack of global contextual understanding. Alternatively, some approaches resort to large-scale Transformer models to bridge the global contextual gaps, but at the expense of model size and computational complexity. Finally many Transformer based approaches rely primarily on CNN based decoders overlooking the benefits of Transformer based decoding models. Recognizing these limitations, we address the need efficient lightweight solutions by introducing MobileUNETR, which aims to overcome the performance constraints associated with both CNNs and Transformers while minimizing model size, presenting a promising stride towards efficient image segmentation. MobileUNETR has 3 main features. 1) MobileUNETR comprises of a lightweight hybrid CNN-Transformer encoder to help balance local and global contextual feature extraction in an efficient manner; 2) A novel hybrid decoder that simultaneously utilizes low-level and global features at different resolutions within the decoding stage for accurate mask generation; 3) surpassing large and complex architectures, MobileUNETR achieves superior performance with 3 million parameters and a computational complexity of 1.3 GFLOP resulting in 10x and 23x reduction in parameters and FLOPS, respectively. Extensive experiments have been conducted to validate the effectiveness of our proposed method on four publicly available skin lesion segmentation datasets, including ISIC 2016, ISIC 2017, ISIC 2018, and PH2 datasets. The code will be publicly available at: https://github.com/OSUPCVLab/MobileUNETR.git