In this paper, we present DM-Calib, a diffusion-based approach for estimating pinhole camera intrinsic parameters from a single input image. Monocular camera calibration is essential for many 3D vision tasks. However, most existing methods depend on handcrafted assumptions or are constrained by limited training data, resulting in poor generalization across diverse real-world images. Recent advancements in stable diffusion models, trained on massive data, have shown the ability to generate high-quality images with varied characteristics. Emerging evidence indicates that these models implicitly capture the relationship between camera focal length and image content. Building on this insight, we explore how to leverage the powerful priors of diffusion models for monocular pinhole camera calibration. Specifically, we introduce a new image-based representation, termed Camera Image, which losslessly encodes the numerical camera intrinsics and integrates seamlessly with the diffusion framework. Using this representation, we reformulate the problem of estimating camera intrinsics as the generation of a dense Camera Image conditioned on an input image. By fine-tuning a stable diffusion model to generate a Camera Image from a single RGB input, we can extract camera intrinsics via a RANSAC operation. We further demonstrate that our monocular calibration method enhances performance across various 3D tasks, including zero-shot metric depth estimation, 3D metrology, pose estimation and sparse-view reconstruction. Extensive experiments on multiple public datasets show that our approach significantly outperforms baselines and provides broad benefits to 3D vision tasks. Code is available at https://github.com/JunyuanDeng/DM-Calib.
Motion planning is a crucial component in autonomous driving. State-of-the-art motion planners are trained on meticulously curated datasets, which are not only expensive to annotate but also insufficient in capturing rarely seen critical scenarios. Failing to account for such scenarios poses a significant risk to motion planners and may lead to incidents during testing. An intuitive solution is to manually compose such scenarios by programming and executing a simulator (e.g., CARLA). However, this approach incurs substantial human costs. Motivated by this, we propose an inexpensive method for generating diverse critical traffic scenarios to train more robust motion planners. First, we represent traffic scenarios as scripts, which are then used by the simulator to generate traffic scenarios. Next, we develop a method that accepts user-specified text descriptions, which a Large Language Model translates into scripts using in-context learning. The output scripts are sent to the simulator that produces the corresponding traffic scenarios. As our method can generate abundant safety-critical traffic scenarios, we use them as synthetic training data for motion planners. To demonstrate the value of generated scenarios, we train existing motion planners on our synthetic data, real-world datasets, and a combination of both. Our experiments show that motion planners trained with our data significantly outperform those trained solely on real-world data, showing the usefulness of our synthetic data and the effectiveness of our data generation method. Our source code is available at https://ezharjan.github.io/AutoSceneGen.
The conversion of natural language into SQL language for querying databases (Text-to-SQL) has broad application prospects and has attracted widespread attention. At present, the mainstream Text-to-SQL methods are mainly divided into in-context learning (ICL) based methods and supervised fine-tuning (SFT) based methods. ICL-based methods can achieve relatively good results thanks to the use of the most advanced closed-source models. However, in real-world application scenarios, factors such as data privacy, SQL generation efficiency and cost need to be considered. SFT-based methods have certain advantages. At present, methods based on fine-tuning of open source models lack easy-to-implement and effective (cost-effective) baseline methods. We propose a pipeline-based method using open source model fine-tuning, referred to as BASE-SQL, which includes four components: Schema Linking, Candidate SQL Generate, SQL Revision and SQL Merge Revision. Experimental results show that BASE-SQL uses the open source model Qwen2.5-Coder-32B-Instruct, and achieves an accuracy of 67.47% on the BIRD development set and 88.9% on the Spider test set, which is significantly better than other methods using open source models, and even exceeds several methods using the GPT-4o closed-source model. At the same time, BASE-SQL is easy to implement and highly efficient (on average, only five calls to the large language model are required to generate SQL once). The code will be open sourced at https://github.com/CycloneBoy/base_sql.
Predicting the near-term behavior of a reactive agent is crucial in many robotic scenarios, yet remains challenging when observations of that agent are sparse or intermittent. Vision-Language Models (VLMs) offer a promising avenue by integrating textual domain knowledge with visual cues, but their one-shot predictions often miss important edge cases and unusual maneuvers. Our key insight is that iterative, counterfactual exploration--where a dedicated module probes each proposed behavior hypothesis, explicitly represented as a plausible trajectory, for overlooked possibilities--can significantly enhance VLM-based behavioral forecasting. We present TRACE (Tree-of-thought Reasoning And Counterfactual Exploration), an inference framework that couples tree-of-thought generation with domain-aware feedback to refine behavior hypotheses over multiple rounds. Concretely, a VLM first proposes candidate trajectories for the agent; a counterfactual critic then suggests edge-case variations consistent with partial observations, prompting the VLM to expand or adjust its hypotheses in the next iteration. This creates a self-improving cycle where the VLM progressively internalizes edge cases from previous rounds, systematically uncovering not only typical behaviors but also rare or borderline maneuvers, ultimately yielding more robust trajectory predictions from minimal sensor data. We validate TRACE on both ground-vehicle simulations and real-world marine autonomous surface vehicles. Experimental results show that our method consistently outperforms standard VLM-driven and purely model-based baselines, capturing a broader range of feasible agent behaviors despite sparse sensing. Evaluation videos and code are available at trace-robotics.github.io.
Although large visual-language models (LVLMs) have demonstrated strong performance in multimodal tasks, errors may occasionally arise due to biases during the reasoning process. Recently, reward models (RMs) have become increasingly pivotal in the reasoning process. Specifically, process RMs evaluate each reasoning step, outcome RMs focus on the assessment of reasoning results, and critique RMs perform error analysis on the entire reasoning process, followed by corrections. However, existing benchmarks for vision-language RMs (VLRMs) typically assess only a single aspect of their capabilities (e.g., distinguishing between two answers), thus limiting the all-round evaluation and restricting the development of RMs in the visual-language domain. To address this gap, we propose a comprehensive and challenging benchmark, dubbed as VLRMBench, encompassing 12,634 questions. VLRMBench is constructed based on three distinct types of datasets, covering mathematical reasoning, hallucination understanding, and multi-image understanding. We design 12 tasks across three major categories, focusing on evaluating VLRMs in the aspects of process understanding, outcome judgment, and critique generation. Extensive experiments are conducted on 21 open-source models and 5 advanced closed-source models, highlighting the challenges posed by VLRMBench. For instance, in the `Forecasting Future', a binary classification task, the advanced GPT-4o achieves only a 76.0% accuracy. Additionally, we perform comprehensive analytical studies, offering valuable insights for the future development of VLRMs. We anticipate that VLRMBench will serve as a pivotal benchmark in advancing VLRMs. Code and datasets will be available at https://github.com/JCruan519/VLRMBench.
Accurate and efficient 3D mapping of large-scale outdoor environments from LiDAR measurements is a fundamental challenge in robotics, particularly towards ensuring smooth and artifact-free surface reconstructions. Although the state-of-the-art methods focus on memory-efficient neural representations for high-fidelity surface generation, they often fail to produce artifact-free manifolds, with artifacts arising due to noisy and sparse inputs. To address this issue, we frame surface mapping as a physics-informed energy optimization problem, enforcing surface smoothness by optimizing an energy functional that penalizes sharp surface ridges. Specifically, we propose a deep learning based approach that learns the signed distance field (SDF) of the surface manifold from raw LiDAR point clouds using a physics-informed loss function that optimizes the $L_2$-Hessian energy of the surface. Our learning framework includes a hierarchical octree based input feature encoding and a multi-scale neural network to iteratively refine the signed distance field at different scales of resolution. Lastly, we introduce a test-time refinement strategy to correct topological inconsistencies and edge distortions that can arise in the generated mesh. We propose a \texttt{CUDA}-accelerated least-squares optimization that locally adjusts vertex positions to enforce feature-preserving smoothing. We evaluate our approach on large-scale outdoor datasets and demonstrate that our approach outperforms current state-of-the-art methods in terms of improved accuracy and smoothness. Our code is available at \href{https://github.com/HrishikeshVish/HessianForge/}{https://github.com/HrishikeshVish/HessianForge/}
Data scarcity is a long-standing challenge in the Vision-Language Navigation (VLN) field, which extremely hinders the generalization of agents to unseen environments. Previous works primarily rely on additional simulator data or web-collected images/videos to improve the generalization. However, the simulator environments still face limited diversity, and the web-collected data often requires extensive labor to remove the noise. In this paper, we propose a Rewriting-driven AugMentation (RAM) paradigm for VLN, which directly creates the unseen observation-instruction pairs via rewriting human-annotated training data. Benefiting from our rewriting mechanism, new observation-instruction can be obtained in both simulator-free and labor-saving manners to promote generalization. Specifically, we first introduce Object-Enriched Observation Rewriting, where we combine Vision-Language Models (VLMs) and Large Language Models (LLMs) to derive rewritten object-enriched scene descriptions, enabling observation synthesis with diverse objects and spatial layouts via Text-to-Image Generation Models (T2IMs). Then, we propose Observation-Contrast Instruction Rewriting, which generates observation-aligned rewritten instructions by requiring LLMs to reason the difference between original and new observations. We further develop a mixing-then-focusing training strategy with a random observation cropping scheme, effectively enhancing data distribution diversity while suppressing augmentation data noise during training. Experiments on both the discrete environments (R2R, REVERIE, and R4R datasets) and continuous environments (R2R-CE dataset) show the superior performance and impressive generalization ability of our method. Code is available at https://github.com/SaDil13/VLN-RAM.
With the recent rise of large language models, vision-language models, and other general foundation models, there is growing potential for multimodal, multi-task robotics that can operate in diverse environments given natural language input. One such application is indoor navigation using natural language instructions. However, despite recent progress, this problem remains challenging due to the 3D spatial reasoning and semantic understanding required. Additionally, the language used may be imperfect or misaligned with the scene, further complicating the task. To address this challenge, we curate a benchmark dataset, IRef-VLA, for Interactive Referential Vision and Language-guided Action in 3D Scenes with imperfect references. IRef-VLA is the largest real-world dataset for the referential grounding task, consisting of over 11.5K scanned 3D rooms from existing datasets, 7.6M heuristically generated semantic relations, and 4.7M referential statements. Our dataset also contains semantic object and room annotations, scene graphs, navigable free space annotations, and is augmented with statements where the language has imperfections or ambiguities. We verify the generalizability of our dataset by evaluating with state-of-the-art models to obtain a performance baseline and also develop a graph-search baseline to demonstrate the performance bound and generation of alternatives using scene-graph knowledge. With this benchmark, we aim to provide a resource for 3D scene understanding that aids the development of robust, interactive navigation systems. The dataset and all source code is publicly released at https://github.com/HaochenZ11/IRef-VLA.
Millions of users leverage generative pretrained transformer (GPT)-based language models developed by leading model providers for a wide range of tasks. To support enhanced user interaction and customization, many platforms-such as OpenAI-now enable developers to create and publish tailored model instances, known as custom GPTs, via dedicated repositories or application stores. These custom GPTs empower users to browse and interact with specialized applications designed to meet specific needs. However, as custom GPTs see growing adoption, concerns regarding their security vulnerabilities have intensified. Existing research on these vulnerabilities remains largely theoretical, often lacking empirical, large-scale, and statistically rigorous assessments of associated risks. In this study, we analyze 14,904 custom GPTs to assess their susceptibility to seven exploitable threats, such as roleplay-based attacks, system prompt leakage, phishing content generation, and malicious code synthesis, across various categories and popularity tiers within the OpenAI marketplace. We introduce a multi-metric ranking system to examine the relationship between a custom GPT's popularity and its associated security risks. Our findings reveal that over 95% of custom GPTs lack adequate security protections. The most prevalent vulnerabilities include roleplay-based vulnerabilities (96.51%), system prompt leakage (92.20%), and phishing (91.22%). Furthermore, we demonstrate that OpenAI's foundational models exhibit inherent security weaknesses, which are often inherited or amplified in custom GPTs. These results highlight the urgent need for enhanced security measures and stricter content moderation to ensure the safe deployment of GPT-based applications.
생성형 이미지 초해상도(ISR) 방법은 일반적으로 표준 가우시안 사전분포에서 샘플링하여 이를 데이터 분포로 이동시키며, 이때 저품질(LQ) 입력은 외부 조건으로만 사용된다. 그러나 LQ 이미지는 목표 고품질(HQ) 이미지에 대한 풍부한 구조 정보를 이미 담고 있으므로, LQ 이미지에 근사하는 분포가 더 적합한 출발점이 되며, 이 아이디어는 일련의 확산 기반 복원 방법에서 검증되었다. 반면 최근의 플로 매칭 기반 ISR 방법은 여전히 표준 가우시안 사전분포를 채택하여 이러한 구조적 사전 정보를 버린다. 더욱이 증류나 일관성 훈련과 같은 기존의 단일 단계 가속 기법은 추가 교사 모델에 의존하거나 모델의 다단계 추론 능력을 희생한다. 본 논문에서는 소스 분포를 LQ 잠재 표현을 중심으로 하는 가우시안으로 정의하여 플로가 순수 노이즈가 아닌 LQ 이미지에서 시작하도록 하고 입력에 이미 존재하는 구조적 사전 정보를 보존하는 이미지 초해상도를 위한 잔차 플로 매칭(RFMSR)을 제안한다. 고품질 단일 단계 생성을 달성하기 위해, 단일 단계 출력에 대한 지도 손실과 조건부 플로 매칭(CFM) 손실을 결합한 간단하면서도 효과적인 훈련 전략을 추가로 제안하며, 이를 통해 증류나 기타 가속 기법에 의존하지 않고도 임의의 단계 수로 고품질 복원이 가능하다. 폭넓은 실험을 통해 RFMSR이 최신 방법들과 경쟁력 있는 결과를 달성함을 보인다. 소스 코드는 https://github.com/Faze-Hsw/RFMSR 에서 이용할 수 있다.
Apple M4 Max GPU의 Metal 4.1 텐서 연산 경로를 역설계한 Rigel은 fp8(E4M3) matmul2d가 하드웨어 가속 없이 에뮬레이션됨을 밝혀내고, 해당 연산이 GPU 셰이더 코어에서 수행됨을 입증했다. 이러한 특성 분석을 통해 최적화된 커널이 기존 분해된 경로보다 최대 12.9% 높은 성능을 달성함을 보여주었다.
폐루프 대규모 언어 모델(LLM) 기반 채널 구성 탐색에서 얻은 유망한 초기 결과는, 실행 가능한 코드 생성과 정확도 피드백을 통해 신경망의 폭을 직접 최적화할 수 있음을 보여주었다. 그러나 이러한 결과는 상대적으로 희소한 유효 평가 집합에서 얻은 것이므로, 관찰된 최적화 행동이 더 조밀한 샘플링 체제로 전이되는지, 그리고 더 많은 생성 네트워크가 평가될 때 추가적인 아키텍처 규칙성이 나타나는지는 열려 있는 문제로 남아 있다. 이를 검증하기 위해, 동일한 탐색 설정을 미세조정 주기당 250개의 후보 네트워크로 확장하였다. 분석은 8개의 완전한 주기에서 생성된 2000개의 후보를 다루며, 작업 및 메타데이터 필터링 후 462개의 검증된 CIFAR-100 평가를 산출한다. 주기별 평균 정확도는 기울기 9.87e-4(p=0.043)의 양의 선형 추세를 보이며, 고성능 프론티어는 더 강하게 개선된다. 관찰된 최고 정확도는 0.3144에서 0.3676으로 증가했고, 상위 5개 및 상위 10개 주기별 평균 모두 양의 추세를 나타낸다. 확장된 실행은 또한 매개변수 효율의 개선을 드러낸다. 최고 모델은 11.8M 매개변수로 0.3676에 도달하는 반면, 초기 고성능 모델은 166.5M 매개변수로 0.3144를 기록했다. 정확도를 넘어, 더 큰 표본은 희소한 관측으로는 평가하기 어려웠던 아키텍처 규칙성을 드러낸다. 2의 거듭제곱이 아닌 채널 폭은 검증된 후보의 41.8%에서 나타나며, 가장 강력한 모델들은 초기 폭은 적당하고 중간 또는 후반 블록을 확장한 구조화된 채널 할당 패턴을 공유한다. 이러한 발견은 초기 연구에서 관찰된 채널 탐색 신호가 전이됨을 시사한다
텍스트-이미지 확산 모델(text-to-image diffusion model)은 일반적으로 강력한 프롬프트 수행 능력을 보이지만, 객체만 다른 소수의 프롬프트가 동일한 생성 설정에서도 동일한 목표 개념을 일관되게 구현하지 못하는, 지속적이면서도 그동안 충분히 탐구되지 않은 실패 패턴을 확인하였다. 우리는 이 현상을 객체 의존적 개념 취약성(object-dependent concept brittleness)이라 명명한다. 이러한 사례는 무작위 샘플링 노이즈가 아니라 모델 내부의 체계적인 사각지대가 존재함을 시사한다. 본 논문에서는 이러한 실패를 감사(audit)하고 최소한으로 교정하기 위한 해석 가능성 중심의 프레임워크를 제시한다. 핵심 아이디어는 단계별 희소 오토인코더(sparse autoencoder, SAE) 공간에서 디노이징 궤적을 분석하는 것으로, 이 공간에서는 추상적인 스타일 및 속성 개념이 원래의 디노이징 표현에서보다 더 잘 분리된다. 이 희소 공간을 통해 성공한 생성과 실패한 생성을 비교하고, 근거가 누락되었거나 약해졌거나 시간적으로 지연된 개념 차원을 식별하며, 신뢰할 수 있고 클래스 일관성이 있는 샘플로부터 클래스 수준의 개념 프로토타입을 구성할 수 있다. 이러한 감사 과정을 바탕으로, SAE 공간에서 디노이징 특징을 해당 프로토타입 쪽으로 보간하는 경량의 추론 시점 교정 전략을 도입한다. 이 개입은 특정 과제를 위한 재학습 방법이 아니라, 진단된 개념 결핍에 대한 검증 역할을 한다. 우리는 여러 확산 백본에 걸친 스타일 및 속성 실패 사례에서 제안한 프레임워크를 평가하였으며, 개념 일관성, 텍스트 충실도, 교정 성공률이 크게 향상되었다. 추가 분석 결과, 더 깊은 디노이징 표현일수록 더 명확한 개념 구조를 제공하고, 초기 단계의 개입이 가장 강력한 교정 효과를 낸다는 점이 나타났다. 코드는 https://github.com/Metecade/Object-Dependent-Concept-Brittleness 에서 확인할 수 있다.
LLM 기반 멀티 에이전트 시스템(MAS)은 다양한 도메인에 걸친 복잡한 문제 해결에서 강력한 능력을 보여주었다. 최근에는 에이전트 시스템의 동적 오케스트레이션이 중요한 연구 방향으로 떠오르고 있다. 그러나 기존 방법들은 제한된 구성, 어긋난 의존성, 경직된 규모라는 한계를 지니고 있어, 실행 중 추론 요구사항에 적응하는 능력이 제약된다. 이러한 한계를 해결하기 위해, 우리는 MAS 설계를 부분적으로 관측 가능한 마르코프 결정 과정(partially observable Markov decision process)으로 재구성하며, 이 과정에서 MAS의 구성과 규모가 모두 동적으로 결정된다. 우리는 Planner, Worker, Generator라는 세 모듈을 조율하여 질의(query)와 진화하는 실행 피드백을 바탕으로 동적 계층적 협업 그래프를 처음부터 점진적으로 구축하는 새로운 프레임워크인 DHCG를 제안한다. 각 단계에서 피드백의 안내를 받아, Planner는 현재 추론 요구에 맞춘 서로 구별되면서도 상호 보완적인 역할 집합을 생성하고, 관련 정보를 각 역할에 선택적으로 라우팅한다. 또한 Planner는 계층적 협업 그래프를 조기에 확정하거나, 추가 추론이 필요할 때 점진적으로 확장할 수 있다. 우리는 나아가 Planner가 계층적 협업 그래프를 구축할 때 더 효과적인 결정을 내리도록 학습시키기 위한 행동 인지 선호 최적화(action-aware preference optimization)를 도입한다. 우리는 코드 생성, 수학적 추론, 도메인 특화 추론 벤치마크에 걸쳐 DHCG를 체계적으로 평가한다. DHCG는 비교 대상 방법들 중 최고 수준의 평균 성능을 달성하며, 단일 에이전트 기준선(baseline) 대비 13.06점 향상되고, 정적 및 동적 MAS 기준선 모두를 2.77~8.02점 차이로 능가한다. 추가 실험은 서로 다른 Planner 백본과 미학습 Worker 모델에 걸친 일반화 능력을 더욱 입증한다.
Generating high-quality text with sufficient diversity is essential for a wide range of Natural Language Generation (NLG) tasks. Maximum-Likelihood (MLE) models trained with teacher forcing have consistently been reported as weak baselines, where poor performance is attributed to exposure bias (Bengio et al., 2015; Ranzato et al., 2015); at inference time, the model is fed its own prediction instead of a ground-truth token, which can lead to accumulating errors and poor samples. This line of reasoning has led to an outbreak of adversarial based approaches for NLG, on the account that GANs do not suffer from exposure bias. In this work, we make several surprising observations which contradict common beliefs. First, we revisit the canonical evaluation framework for NLG, and point out fundamental flaws with quality-only evaluation: we show that one can outperform such metrics using a simple, well-known temperature parameter to artificially reduce the entropy of the model's conditional distributions. Second, we leverage the control over the quality / diversity trade-off given by this parameter to evaluate models over the whole quality-diversity spectrum and find MLE models constantly outperform the proposed GAN variants over the whole quality-diversity space. Our results have several implications: 1) The impact of exposure bias on sample quality is less severe than previously thought, 2) temperature tuning provides a better quality / diversity trade-off than adversarial training while being easier to train, easier to cross-validate, and less computationally expensive. Code to reproduce the experiments is available at github.com/pclucas14/GansFallingShort
The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question "can we automate scientific reviewing?", discussing the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. Arguably the most difficult part of this is defining what a "good" review is in the first place, so we first discuss possible evaluation measures for such reviews. We then collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews, but the generated text can suffer from lower constructiveness for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. We finally summarize eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research on this subject. We make all code, and the dataset publicly available: https://github.com/neulab/ReviewAdvisor, as well as a ReviewAdvisor system: http://review.nlpedia.ai/.
In image generation, generative models can be evaluated naturally by visually inspecting model outputs. However, this is not always the case for graph generative models (GGMs), making their evaluation challenging. Currently, the standard process for evaluating GGMs suffers from three critical limitations: i) it does not produce a single score which makes model selection challenging, ii) in many cases it fails to consider underlying edge and node features, and iii) it is prohibitively slow to perform. In this work, we mitigate these issues by searching for scalar, domain-agnostic, and scalable metrics for evaluating and ranking GGMs. To this end, we study existing GGM metrics and neural-network-based metrics emerging from generative models of images that use embeddings extracted from a task-specific network. Motivated by the power of certain Graph Neural Networks (GNNs) to extract meaningful graph representations without any training, we introduce several metrics based on the features extracted by an untrained random GNN. We design experiments to thoroughly test metrics on their ability to measure the diversity and fidelity of generated graphs, as well as their sample and computational efficiency. Depending on the quantity of samples, we recommend one of two random-GNN-based metrics that we show to be more expressive than pre-existing metrics. While we focus on applying these metrics to GGM evaluation, in practice this enables the ability to easily compute the dissimilarity between any two sets of graphs regardless of domain. Our code is released at: https://github.com/uoguelph-mlrg/GGM-metrics.
Solving the domain shift problem during inference is essential in medical imaging, as most deep-learning based solutions suffer from it. In practice, domain shifts are tackled by performing Unsupervised Domain Adaptation (UDA), where a model is adapted to an unlabelled target domain by leveraging the labelled source data. In medical scenarios, the data comes with huge privacy concerns making it difficult to apply standard UDA techniques. Hence, a closer clinical setting is Source-Free UDA (SFUDA), where we have access to source-trained model but not the source data during adaptation. Existing SFUDA methods rely on pseudo-label based self-training techniques to address the domain shift. However, these pseudo-labels often have high entropy due to domain shift and adapting the source model with noisy pseudo-labels leads to sub-optimal performance. To overcome this limitation, we propose a systematic two-stage approach for SFUDA comprising of target-specific adaptation followed by task-specific adaptation. In target-specific adaptation, we enhance the pseudo-label generation by minimizing high entropy regions using the proposed ensemble entropy minimization loss and a selective voting strategy. In task-specific adaptation, we exploit the enhanced pseudo-labels using a student-teacher framework to effectively learn segmentation on the target domain. We evaluate our proposed method on 2D fundus datasets and 3D MRI volumes across 7 different domain shifts where we perform better than existing UDA and SFUDA methods for medical image segmentation. Code is available at https://github.com/Vibashan/tt-sfuda.
Deep convolutional neural network (CNN) based models are vulnerable to the adversarial attacks. One of the possible reasons is that the embedding space of CNN based model is sparse, resulting in a large space for the generation of adversarial samples. In this study, we propose a method, denoted as Dynamic Feature Aggregation, to compress the embedding space with a novel regularization. Particularly, the convex combination between two samples are regarded as the pivot for aggregation. In the embedding space, the selected samples are guided to be similar to the representation of the pivot. On the other side, to mitigate the trivial solution of such regularization, the last fully-connected layer of the model is replaced by an orthogonal classifier, in which the embedding codes for different classes are processed orthogonally and separately. With the regularization and orthogonal classifier, a more compact embedding space can be obtained, which accordingly improves the model robustness against adversarial attacks. An averaging accuracy of 56.91% is achieved by our method on CIFAR-10 against various attack methods, which significantly surpasses a solid baseline (Mixup) by a margin of 37.31%. More surprisingly, empirical results show that, the proposed method can also achieve the state-of-the-art performance for out-of-distribution (OOD) detection, due to the learned compact feature space. An F1 score of 0.937 is achieved by the proposed method, when adopting CIFAR-10 as in-distribution (ID) dataset and LSUN as OOD dataset. Code is available at https://github.com/HaozheLiu-ST/DynamicFeatureAggregation.
Generating images from graph-structured inputs, such as scene graphs, is uniquely challenging due to the difficulty of aligning nodes and connections in graphs with objects and their relations in images. Most existing methods address this challenge by using scene layouts, which are image-like representations of scene graphs designed to capture the coarse structures of scene images. Because scene layouts are manually crafted, the alignment with images may not be fully optimized, causing suboptimal compliance between the generated images and the original scene graphs. To tackle this issue, we propose to learn scene graph embeddings by directly optimizing their alignment with images. Specifically, we pre-train an encoder to extract both global and local information from scene graphs that are predictive of the corresponding images, relying on two loss functions: masked autoencoding loss and contrastive loss. The former trains embeddings by reconstructing randomly masked image regions, while the latter trains embeddings to discriminate between compliant and non-compliant images according to the scene graph. Given these embeddings, we build a latent diffusion model to generate images from scene graphs. The resulting method, called SGDiff, allows for the semantic manipulation of generated images by modifying scene graph nodes and connections. On the Visual Genome and COCO-Stuff datasets, we demonstrate that SGDiff outperforms state-of-the-art methods, as measured by both the Inception Score and Fr\'echet Inception Distance (FID) metrics. We will release our source code and trained models at https://github.com/YangLing0818/SGDiff.