Synthesizing natural interactions between virtual humans and their 3D environments is critical for numerous applications, such as computer games and AR/VR experiences. Our goal is to synthesize humans interacting with a given 3D scene controlled by high-level semantic specifications as pairs of action categories and object instances, e.g., "sit on the chair". The key challenge of incorporating interaction semantics into the generation framework is to learn a joint representation that effectively captures heterogeneous information, including human body articulation, 3D object geometry, and the intent of the interaction. To address this challenge, we design a novel transformer-based generative model, in which the articulated 3D human body surface points and 3D objects are jointly encoded in a unified latent space, and the semantics of the interaction between the human and objects are embedded via positional encoding. Furthermore, inspired by the compositional nature of interactions that humans can simultaneously interact with multiple objects, we define interaction semantics as the composition of varying numbers of atomic action-object pairs. Our proposed generative model can naturally incorporate varying numbers of atomic interactions, which enables synthesizing compositional human-scene interactions without requiring composite interaction data. We extend the PROX dataset with interaction semantic labels and scene instance segmentation to evaluate our method and demonstrate that our method can generate realistic human-scene interactions with semantic control. Our perceptual study shows that our synthesized virtual humans can naturally interact with 3D scenes, considerably outperforming existing methods. We name our method COINS, for COmpositional INteraction Synthesis with Semantic Control. Code and data are available at https://github.com/zkf1997/COINS.
Pseudo-labels are widely employed in weakly supervised 3D segmentation tasks where only sparse ground-truth labels are available for learning. Existing methods often rely on empirical label selection strategies, such as confidence thresholding, to generate beneficial pseudo-labels for model training. This approach may, however, hinder the comprehensive exploitation of unlabeled data points. We hypothesize that this selective usage arises from the noise in pseudo-labels generated on unlabeled data. The noise in pseudo-labels may result in significant discrepancies between pseudo-labels and model predictions, thus confusing and affecting the model training greatly. To address this issue, we propose a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions. More specifically, our method introduces an Entropy Regularization loss and a Distribution Alignment loss for weakly supervised learning in 3D segmentation tasks, resulting in an ERDA learning strategy. Interestingly, by using KL distance to formulate the distribution alignment loss, it reduces to a deceptively simple cross-entropy-based loss which optimizes both the pseudo-label generation network and the 3D segmentation network simultaneously. Despite the simplicity, our method promisingly improves the performance. We validate the effectiveness through extensive experiments on various baselines and large-scale datasets. Results show that ERDA effectively enables the effective usage of all unlabeled data points for learning and achieves state-of-the-art performance under different settings. Remarkably, our method can outperform fully-supervised baselines using only 1% of true annotations. Code and model will be made publicly available at https://github.com/LiyaoTang/ERDA.
Edge computing is a distributed computing paradigm that collects and processes data at or near the source of data generation. The on-device learning at edge relies on device-to-device wireless communication to facilitate real-time data sharing and collaborative decision-making among multiple devices. This significantly improves the adaptability of the edge computing system to the changing environments. However, as the scale of the edge computing system is getting larger, communication among devices is becoming the bottleneck because of the limited bandwidth of wireless communication leads to large data transfer latency. To reduce the amount of device-to-device data transmission and accelerate on-device learning, in this paper, we propose Residual-INR, a fog computing-based communication-efficient on-device learning framework by utilizing implicit neural representation (INR) to compress images/videos into neural network weights. Residual-INR enhances data transfer efficiency by collecting JPEG images from edge devices, compressing them into INR format at the fog node, and redistributing them for on-device learning. By using a smaller INR for full image encoding and a separate object INR for high-quality object region reconstruction through residual encoding, our technique can reduce the encoding redundancy while maintaining the object quality. Residual-INR is a promising solution for edge on-device learning because it reduces data transmission by up to 5.16 x across a network of 10 edge devices. It also facilitates CPU-free accelerated on-device learning, achieving up to 2.9 x speedup without sacrificing accuracy. Our code is available at: https://github.com/sharc-lab/Residual-INR.
With the growing deployment of LLMs in daily applications like chatbots and content generation, efforts to ensure outputs align with human values and avoid harmful content have intensified. However, increasingly sophisticated jailbreak attacks threaten this alignment, aiming to induce unsafe outputs. Current defense efforts either focus on prompt rewriting or detection, which are limited in effectiveness due to the various design of jailbreak prompts, or on output control and detection, which are computationally expensive as they require LLM inference. Therefore, designing a pre-inference defense method that resists diverse jailbreak prompts is crucial for preventing LLM jailbreak attacks. We observe that jailbreak attacks, safe queries, and harmful queries exhibit different clustering patterns within the LLM's hidden state representation space. This suggests that by leveraging the LLM's hidden state representational capabilities, we can analyze the LLM's forthcoming behavior and proactively intervene for defense. In this paper, we propose a jailbreak attack defense strategy based on a Hidden State Filter (HSF), a lossless architectural defense mechanism that enables the model to preemptively identify and reject adversarial inputs before the inference process begins. We activate its defensive potential through an additional plugin module, effectively framing the defense task as a classification problem. Experimental results on two benchmark datasets, utilizing three different LLMs, show that HSF significantly enhances resilience against six cutting-edge jailbreak attacks. It significantly reduces the success rate of jailbreak attacks while minimally impacting responses to benign user queries, with negligible inference overhead, and outperforming defense baselines.Our code and data are available at https://anonymous.4open.science/r/Hidden-State-Filtering-8652/
Goal-oriented planning, or anticipating a series of actions that transition an agent from its current state to a predefined objective, is crucial for developing intelligent assistants aiding users in daily procedural tasks. The problem presents significant challenges due to the need for comprehensive knowledge of temporal and hierarchical task structures, as well as strong capabilities in reasoning and planning. To achieve this, prior work typically relies on extensive training on the target dataset, which often results in significant dataset bias and a lack of generalization to unseen tasks. In this work, we introduce VidAssist, an integrated framework designed for zero/few-shot goal-oriented planning in instructional videos. VidAssist leverages large language models (LLMs) as both the knowledge base and the assessment tool for generating and evaluating action plans, thus overcoming the challenges of acquiring procedural knowledge from small-scale, low-diversity datasets. Moreover, VidAssist employs a breadth-first search algorithm for optimal plan generation, in which a composite of value functions designed for goal-oriented planning is utilized to assess the predicted actions at each step. Extensive experiments demonstrate that VidAssist offers a unified framework for different goal-oriented planning setups, e.g., visual planning for assistance (VPA) and procedural planning (PP), and achieves remarkable performance in zero-shot and few-shot setups. Specifically, our few-shot model outperforms the prior fully supervised state-of-the-art method by +7.7% in VPA and +4.81% PP task on the COIN dataset while predicting 4 future actions. Code, and models are publicly available at https://sites.google.com/view/vidassist.
Interactive portrait matting refers to extracting the soft portrait from a given image that best meets the user's intent through their inputs. Existing methods often underperform in complex scenarios, mainly due to three factors. (1) Most works apply a tightly coupled network that directly predicts matting results, lacking interpretability and resulting in inadequate modeling. (2) Existing works are limited to a single type of user input, which is ineffective for intention understanding and also inefficient for user operation. (3) The multi-round characteristics have been under-explored, which is crucial for user interaction. To alleviate these limitations, we propose DFIMat, a decoupled framework that enables flexible interactive matting. Specifically, we first decouple the task into 2 sub-ones: localizing target instances by understanding scene semantics and the flexible user inputs, and conducting refinement for instance-level matting. We observe a clear performance gain from decoupling, as it makes sub-tasks easier to learn, and the flexible multi-type input further enhances both effectiveness and efficiency. DFIMat also considers the multi-round interaction property, where a contrastive reasoning module is designed to enhance cross-round refinement. Another limitation for multi-person matting task is the lack of training data. We address this by introducing a new synthetic data generation pipeline that can generate much more realistic samples than previous arts. A new large-scale dataset SMPMat is subsequently established. Experiments verify the significant superiority of DFIMat. With it, we also investigate the roles of different input types, providing valuable principles for users. Our code and dataset can be found at https://github.com/JiaoSiyi/DFIMat.
Recent advances in large-scale text-to-image (T2I) diffusion models have enabled a variety of downstream applications, including style customization, subject-driven personalization, and conditional generation. As T2I models require extensive data and computational resources for training, they constitute highly valued intellectual property (IP) for their legitimate owners, yet making them incentive targets for unauthorized fine-tuning by adversaries seeking to leverage these models for customized, usually profitable applications. Existing IP protection methods for diffusion models generally involve embedding watermark patterns and then verifying ownership through generated outputs examination, or inspecting the model's feature space. However, these techniques are inherently ineffective in practical scenarios when the watermarked model undergoes fine-tuning, and the feature space is inaccessible during verification ((i.e., black-box setting). The model is prone to forgetting the previously learned watermark knowledge when it adapts to a new task. To address this challenge, we propose SleeperMark, a novel framework designed to embed resilient watermarks into T2I diffusion models. SleeperMark explicitly guides the model to disentangle the watermark information from the semantic concepts it learns, allowing the model to retain the embedded watermark while continuing to be adapted to new downstream tasks. Our extensive experiments demonstrate the effectiveness of SleeperMark across various types of diffusion models, including latent diffusion models (e.g., Stable Diffusion) and pixel diffusion models (e.g., DeepFloyd-IF), showing robustness against downstream fine-tuning and various attacks at both the image and model levels, with minimal impact on the model's generative capability. The code is available at https://github.com/taco-group/SleeperMark.
The exceptional generative capability of text-to-image models has raised substantial safety concerns regarding the generation of Not-Safe-For-Work (NSFW) content and potential copyright infringement. To address these concerns, previous methods safeguard the models by eliminating inappropriate concepts. Nonetheless, these models alter the parameters of the backbone network and exert considerable influences on the structural (low-frequency) components of the image, which undermines the model's ability to retain non-target concepts. In this work, we propose our Dual encoder Modulation network (DuMo), which achieves precise erasure of inappropriate target concepts with minimum impairment to non-target concepts. In contrast to previous methods, DuMo employs the Eraser with PRior Knowledge (EPR) module which modifies the skip connection features of the U-NET and primarily achieves concept erasure on details (high-frequency) components of the image. To minimize the damage to non-target concepts during erasure, the parameters of the backbone U-NET are frozen and the prior knowledge from the original skip connection features is introduced to the erasure process. Meanwhile, the phenomenon is observed that distinct erasing preferences for the image structure and details are demonstrated by the EPR at different timesteps and layers. Therefore, we adopt a novel Time-Layer MOdulation process (TLMO) that adjusts the erasure scale of EPR module's outputs across different layers and timesteps, automatically balancing the erasure effects and model's generative ability. Our method achieves state-of-the-art performance on Explicit Content Erasure, Cartoon Concept Removal and Artistic Style Erasure, clearly outperforming alternative methods. Code is available at https://github.com/Maplebb/DuMo
Following the gaze of other people and analyzing the target they are looking at can help us understand what they are thinking, and doing, and predict the actions that may follow. Existing methods for gaze following struggle to perform well in natural scenes with diverse objects, and focus on gaze points rather than objects, making it difficult to deliver clear semantics and accurate scope of the targets. To address this shortcoming, we propose a novel gaze target prediction solution named GazeSeg, that can fully utilize the spatial visual field of the person as guiding information and lead to a progressively coarse-to-fine gaze target segmentation and recognition process. Specifically, a prompt-based visual foundation model serves as the encoder, working in conjunction with three distinct decoding modules (e.g. FoV perception, heatmap generation, and segmentation) to form the framework for gaze target prediction. Then, with the head bounding box performed as an initial prompt, GazeSeg obtains the FoV map, heatmap, and segmentation map progressively, leading to a unified framework for multiple tasks (e.g. direction estimation, gaze target segmentation, and recognition). In particular, to facilitate this research, we construct and release a new dataset, comprising 72k images with pixel-level annotations and 270 categories of gaze targets, built upon the GazeFollow dataset. The quantitative evaluation shows that our approach achieves the Dice of 0.325 in gaze target segmentation and 71.7% top-5 recognition. Meanwhile, our approach also outperforms previous state-of-the-art methods, achieving 0.953 in AUC on the gaze-following task. The dataset and code will be released.
As language models improve and become capable of performing more complex tasks across modalities, evaluating them automatically becomes increasingly challenging. Developing strong and robust task-specific automatic metrics gets harder, and human-annotated test sets -- which are expensive to create -- saturate more quickly. A compelling alternative is to design reliable strategies to automate the creation of test data and evaluation, but previous attempts either rely on pre-existing data, or focus solely on individual tasks. We present Zero-shot Benchmarking (ZSB), a framework for creating high-quality benchmarks for any task by leveraging language models for both synthetic test data creation and evaluation. ZSB is simple and flexible: it requires only the creation of a prompt for data generation and one for evaluation; it is scalable to tasks and languages where collecting real-world data is costly or impractical; it is model-agnostic, allowing the creation of increasingly challenging benchmarks as models improve. To assess the effectiveness of our framework, we create benchmarks for five text-only tasks and a multi-modal one: general capabilities in four languages (English, Chinese, French, and Korean), translation, and general vision-language capabilities in English. We then rank a broad range of open and closed systems on our benchmarks. ZSB rankings consistently correlate strongly with human rankings, outperforming widely-adopted standard benchmarks. Through ablations, we find that strong benchmarks can be created with open models, and that judge model size and dataset variety are crucial drivers of performance. We release all our benchmarks, and code to reproduce our experiments and to produce new benchmarks.
Recent studies empirically reveal that large reasoning models (LRMs) can automatically allocate more reasoning strengths (i.e., the number of reasoning tokens) for harder problems, exhibiting difficulty-awareness for better task performance. While this automatic reasoning strength allocation phenomenon has been widely observed, its underlying mechanism remains largely unexplored. To this end, we provide explanations for this phenomenon from the perspective of model activations. We find evidence that LRMs pre-plan the reasoning strengths in their activations even before generation, with this reasoning strength causally controlled by the magnitude of a pre-allocated directional vector. Specifically, we show that the number of reasoning tokens is predictable solely based on the question activations using linear probes, indicating that LRMs estimate the required reasoning strength in advance. We then uncover that LRMs encode this reasoning strength through a pre-allocated directional vector embedded in the activations of the model, where the vector's magnitude modulates the reasoning strength. Subtracting this vector can lead to reduced reasoning token number and performance, while adding this vector can lead to increased reasoning token number and even improved performance. We further reveal that this direction vector consistently yields positive reasoning length prediction, and it modifies the logits of end-of-reasoning token </think> to affect the reasoning length. Finally, we demonstrate two potential applications of our findings: overthinking behavior detection and enabling efficient reasoning on simple problems. Our work provides new insights into the internal mechanisms of reasoning in LRMs and offers practical tools for controlling their reasoning behaviors. Our code is available at https://github.com/AlphaLab-USTC/LRM-plans-CoT.
Large Language Models excel at reasoning and generation but their autoregressive, token-by-token inference makes them unnecessarily slow, expensive, and poorly calibrated for the many routine decisions (routing, gating, scoring, classification) that modern AI systems require at sub-second latency. A growing class of models addresses this gap: System 1 decision models, architectures that map an input directly to a typed, structured decision in a single forward pass. This guide provides a comprehensive, practitioner-oriented treatment of the field. We propose a five-family taxonomy covering encoder-based classifiers, reward models, non-autoregressive decision engines, contrastive decision models, and extraction models, and we provide in-depth coverage of each family's architecture, training methodology, and deployment trade-offs. A central theme is calibration: we develop the theory of proper scoring rules, analyze Reinforcement Learning for Calibrated Decisions (RLCD), and examine recently discovered phenomena including IIA violations, ordinal-scale bias, and few-shot recalibration under distribution shift. On the integration side, we describe how System 1 models serve as decision infrastructure within agentic workflows and System 1 + System 2 cascades that route between fast and slow models based on calibrated confidence. The guide covers 18 application domains, maps the rapidly growing open-source ecosystem (over 70 model reproductions in weeks), provides runnable code throughout, and includes a practitioner's checklist, an open-source model zoo, and 22 self-assessment questions with detailed answers. Spanning 190 references, it is intended as a unified entry point for researchers and engineers working at the intersection of fast decision-making, calibration, and production AI systems.
과학적 발견은 도메인 지식과 추론을 통해 해석된 정확한 관찰 위에 구축된다. 과학적 3D 데이터의 경우 이는 단백질 접촉에서 해부학적 구성에 이르기까지 구조와 공간적 관계를 이해하는 것을 요구한다. 기존 멀티모달 벤치마크는 대체로 텍스트와 2D 이미지를 평가하며, 과학적 3D 평가는 분야마다 표현과 프로토콜이 달라 분야 간 비교가 제한된다. 이러한 기초 역량을 평가하기 위해 우리는 전문가 수준의 시각 질의응답(VQA)을 통한 과학적 3D 이해와 추론을 위한 분야 횡단 벤치마크인 ScienceVision을 소개한다. 우리는 생물학, 화학, 의료 영상, 기계 CAD, 건축에서 각 50개씩 총 250개의 질문을, 합성 질문 생성과 도메인 전문가의 검토 및 주석을 결합한 파이프라인으로 구축한다. 각 과제는 원본 3D 산출물, 과학적 질문, 지정된 응답 형식, 과제별 참조 답안을 짝지어 제시하며, 기하학적 또는 위상적 증거에 도메인 지식을 적용할 것을 요구한다. 우리는 12개 모델을 동일한 질문에 대해 세 가지 설정에서 평가한다. 첫째는 에이전트 하니스의 파일 및 코드 실행 도구를 통해 원본 3D 산출물에 접근하는 시각 질의응답이고, 둘째는 동일한 워크플로에 Model Context Protocol(MCP)로 제공되는 전문가에서 착안한 뷰어 검사 및 측정 도구를 추가한 것이며, 셋째는 에이전트 하니스를 사용해 지정된 렌더링 2D 이미지로 수행하는 시각 질의응답이다. 모든 설정은 동일한 과제별 결정론적 최종 답안 채점을 사용한다. 검사 환경은 시점 변경, 대상 분리, 슬라이스 검사를 지원한다. 이 주요 비교에서 최고 평균 답안 점수는 1점 만점에 0.53이다. 추가 MCP 도구가 없는 원본 산출물 워크플로의 평균 점수는 모든 모델에서 렌더링 이미지 워크플로보다 높았으며, 평균 17.2 퍼센트포인트 차이를 보였다.
Real-world detectors for autonomous driving, surveillance, and robotics must handle domain-shifts under strict latency and memory constraints, yet existing source-free object detection (SFOD) methods rely on heavyweight architectures that prioritize accuracy alone. We show this trade-off is unnecessary: building on YOLOv10, an NMS-free dual-head detector, we achieve state-of-the-art adaptation accuracy while being faster and more compact. We observe that directly applying vanilla mean-teacher self-training to dual-head detectors leads to suboptimal adaptation performance due to two key factors. First, simple pseudo-label generation strategies, such as using a single head or directly combining high-confidence predictions from both heads, yield suboptimal supervision under domain-shift. We propose DHF (Dual-Head Pseudo-Label Fusion) which selectively admits one-to-one (O2O) and one-to-many (O2M) head predictions, preserving precision and recovering missed objects. Second, we observe domain-shift collapses multi-scale feature discriminability. We propose the use of our MARD (Multi-scale Adaptive Representation Diversification) loss which mitigates this by enforcing detection-aware variance and covariance constraints on multi-scale feature maps. Both modules are training-time only, leaving inference unchanged. Across domain-shift benchmarks, our method, RT-SFOD yields 1.4 to 3.5\% mAP gains, 1.3$\times$ higher throughput, with $\sim$2$\times$ fewer parameters than prior state-of-the-art SFOD methods, thus advancing the Pareto frontier of the speed-accuracy-model size trade-off. We report main results with YOLOv10, and demonstrate generalizability with additional YOLO- and DETR-based dual-head detectors. Code is available here: https://github.com/Sairam13001/RT-SFOD/
멀티모달 연속 지시 미세조정(Multimodal Continual Instruction Tuning, MCIT)은 멀티모달 대규모 언어 모델이 이전에 학습한 기능을 유지하면서 새로운 작업을 순차적으로 습득할 수 있도록 합니다. 최근의 많은 방법은 작업별 LoRA 전문가(expert)를 유지하고 추론(inference) 시 각 입력을 하나 이상의 전문가로 라우팅합니다. 그러나 전문가 라우팅의 기반이 되는 작업 식별 문제는 여전히 충분히 연구되지 않았습니다. 본 연구에서는 널리 사용되는 MCIT 벤치마크에서 라우팅이 거의 포화 상태임을 보입니다. 작업의 정체를 유출하는 텍스트 핑거프린트와 경쟁하는 전문가 수가 적은 짧은 4~10개 작업 시퀀스가 결합하여 장기적(long-horizon) 라우팅 문제를 가리고 있습니다. 이러한 과제를 드러내기 위해, 본 연구에서는 텍스트 핑거프린트를 완화한 34개 작업 규모의 장기적 MCIT 벤치마크인 FLEX(Fingerprint-reduced Long-horizon Expert eXamination)를 소개합니다. FLEX는 유사한 지시문 및 답변 형식을 가지면서도 시각 및 지식 도메인이 다양한 작업들을 그룹화하고, 외부 템플릿을 정규화하며, 훨씬 더 큰 전문가 풀에 대해 라우팅을 평가합니다. 결정적으로, 본 논문은 점진적 LoRA 라우팅을 소프트 작업-클래스 멀티모달 클래스 증분 학습(Multimodal Class-Incremental Learning, MCIL)으로 공식화합니다. 즉, 각 작업이 증분 라우팅 클래스를 정의하며, 이의 완전한 점수 분포가 LoRA 혼합 가중치를 제공하고 하드 라우팅은 이산적인 특수 사례가 됩니다. FLEX는 확장되는 작업 식별 과제를 드러내는 한편, MCIL 공식화는 CIL 방법을 전문가 라우팅으로 전이하기 위한 원칙적인 인터페이스를 제공합니다. 본 연구는 통제된 기준선(baseline)으로서 PureLoRA를 구현하고, LoRA 전문가나 생성 파이프라인을 수정하지 않은 채 4개의 CIL 방법을 4개의 MCIT 프레임워크에 적용합니다. 이러한 플러그인 라우터는 엄격한 LoRA 매칭을 최대 16.3%포인트 향상시키고 전체 MacroScore를 최대 4.6포인트 개선합니다. 코드는 https://github.com/RINC-CL/FLEX 에서 확인할 수 있습니다.
대규모 언어 모델(LLM)의 급속한 발전으로 서베이(survey) 작성은 수개월에 걸치던 수작업에서 자동화된 과정으로 바뀌었다. 생성 규모가 커질수록 신뢰할 수 있는 평가가 병목이 되며, LLM이 서베이 평가자로 점점 더 많이 쓰인다. 그러나 기존 접근은 대부분 인간 리뷰어와의 체계적 정렬 없이 상용 LLM-as-a-judge 방식에 의존하며, 인간 리뷰어와의 정렬을 정량화하는 체계적 프레임워크가 부족하다. 이 공백을 메우기 위해 우리는 서베이 평가를 위한 리뷰어 정렬 다차원 벤치마크 및 데이터셋인 SurveyReview를 제안한다. 675편의 서베이 논문과 1,630건의 리뷰 보고서를 수집·주석했다. 자유 형식 코멘트를 네 가지 차원 점수(가독성 Readability, 비판성 Criticalness, 포괄성 Comprehensiveness, 구조 Structure)와 뒷받침 근거로 변환해 실제 동료 심사(peer-review) 보고서를 구조화했다. 또한 자동 평가자와 인간 리뷰어 간 정렬을 측정하는 표준화된 train/test 분할과 평가 프로토콜을 공개한다. 벤치마크 검증을 위해 주석 데이터에 LoRA로 Qwen3-32B를 미세조정하고, 지식 집약적 차원에는 외부 지식을 보강한 강력한 기준선 평가자 SurveyAlign을 개발했다. 테스트 세트에서 SurveyAlign은 GPT-5.2 기반 프롬프트 심사 대비 리뷰어 정렬을 크게 개선하여, 네 차원 전체에서 평균 MSE를 2.28에서 1.38로, MAE를 1.15에서 0.69로 낮췄다. 기여는 두 가지다: (1) 서베이 심사를 위한 최초의 다차원·리뷰어 정렬 데이터셋과 재현 가능한 평가 프레임워크를 수립했고, (2) 인간 리뷰어와의 정렬을 크게 높인 강력한 기준선 평가자를 개발해 향후 연구의 경쟁적 참조점을 제공한다. 코드와 데이터는 https://surveyreview.github.io 에서 이용할 수 있다.
NV-Reason-CT: CT 분석을 위한 3D 비전-언어 모델
{"title_ko": "NV-Reason-CT: CT 분석을 위한 3D 비전-언어 모델", "abstract_ko": "본 논문은 흉부 및 복부 CT를 위한 생성형 비전-언어 모델인 NV-Reason-CT를 제안한다. 이 모델은 네이티브 3D 시각 인코딩과 방사선 전문의가 안내한 추론(reasoning)을 결합한다. 네이티브 3D 비전 트랜스포머와 언어 모델을 결합하며, 추가적인 공간 토큰 병합 없이 모든 시각 토큰과 그 명시적 3D 좌표를 언어 디코딩에 전달한다. 이를 통해 비전 인코더 내부와 텍스트와의 공동 처리 과정에서 언어 모델의 위치 인코딩(positional encoding)을 거치며 체적(volumetric) 공간 정보를 유지한다. 70,111개의 고유 CT 영상 입력에서 수집한 약 55만 건의 멀티모달 지시(instruction) 예시로 구성된 큐레이션 코퍼스에서 학습하며, 표준화된 리포트, 이상 소견 중심 및 해부학 특화 질문, 다중 턴 상호작용, 녹음·전사된 전문가 CT 판독에서 도출한 방사선 전문의 작성 추론을 결합한다. 전문가 주석은 직접 지도(supervision)를 제공하고 리포트에 근거한 추가 합성 추론을 안내한다. End-to-end 지도 미세조정(SFT) 이후 흉부·복부 이상 소견 집합에 대한 검증 가능한 보상(verifiable rewards)을 사용한 Group Relative Policy Optimization(GRPO)을 수행한다. 모델은 이상 소견 분류, 리포트 생성, 검토 가능한 관찰·감별 진단·불확실성을 포함한 대화형 추론을 지원한다. 평가는 공개 CT 벤치마크와 홀드아웃 NIH 코호트에 걸쳐 이루어진다. CT-RATE에서 NV-Reason-CT는 과제별 분류 헤드 없이 macro-F1 0.614, macro-AUROC 0.871을 달성하며, 생성 리포트는 리포트 기반 macro-F1 0.592를 기록한다. 방사선 전문의를 대상으로 한 예비 연구에서 AI 보조 검토는 긍정적인 신뢰도 평가를 받았고, 평균 보고된 판독·리포팅 시간 50% 감소와 연관되었다. 체적 의료 영상을 위한 설명 가능 AI(explainable AI)의 재현 가능한 연구를 지원하기 위해 모델과 학습 코드를 공개한다."}
Recently, convolutional Neural Networks (CNNs) have exhibited commendable performance for hyperspectral image (HSI) classification. Generally, an important number of samples are needed for each class to properly train CNNs. However, existing HSI datasets suffer from a significant class imbalance problem, where many classes do not have enough samples to characterize the spectral information. The performance of existing CNN models is biased towards the majority classes, which possess more samples for the training. This paper addresses this issue of imbalanced data in HSI classification. In particular, a new 3D-HyperGAMO model is proposed which uses generative adversarial minority oversampling. The proposed 3D-HyperGAMO automatically generates more samples for minority classes at training time, using the existing samples of that class. The samples are generated in the form of a 3D hyperspectral patch. A different classifier from the generator and the discriminator is used in the 3D-HyperGAMO model, which is trained using both original and generated samples to {determine} the classes of newly generated samples to which they actually belong. The generated data are combined class-wise with the original training dataset to learn the network parameters of the class. Finally, the trained 3D classifier network validates the performance of the model using the test set. Four benchmark HSI datasets, namely Indian Pines~(IP), Kennedy Space Center~(KSC), University of Pavia~(UP), and Botswana~(BW), have been considered in our experiments. The proposed model shows outstanding data generation ability during the training, which significantly improves the classification performance over the considered datasets. The source code is available publicly at \url{https://github.com/mhaut/3D-HyperGAMO}.
Memory is identified as a crucial human faculty that allows for the retention of visual and linguistic information within the hippocampus and neurons in the brain, which can subsequently be retrieved to address real-world challenges that arise through a lifetime of learning. The resolution of complex AI tasks through the application of acquired knowledge represents a stride toward the realization of artificial general intelligence. However, despite the prevalence of Large Language Models (LLMs) like GPT-3.5 and GPT-4 \cite{brown2020language, leiter2023chatgpt, zaitsu2023distinguishing, OpenAI2023GPT4TR} , which have displayed remarkable capabilities in language comprehension, generation, interaction, and reasoning, they are inhibited by constraints on context length that preclude the processing of extensive, continually evolving knowledge bases. This paper proposes that LLMs could be augmented through the selective integration of knowledge from external repositories, and in doing so, introduces a novel methodology for External Reasoning, exemplified by ChatPDF. Central to this approach is the establishment of a tiered policy for \textbf{External Reasoning based on Multiple LLM Interchange Assistance} in \cref{fig:overall}, where the level of support rendered is modulated across entry, intermediate, and advanced tiers based on the complexity of the query, with adjustments made in response to human feedback. A comprehensive evaluation of this methodology is conducted using multiple LLMs and the results indicate state-of-the-art performance in \cref{comparison} , surpassing existing solutions including ChatPDF.com. Moreover, the paper emphasizes that this approach is more efficient compared to the direct processing of full text by LLMs. The source code is publicly available at: \url{https://github.com/AkideLiu/ANLP}.
The next generation of machine learning systems must be adept at perceiving and interacting with the physical world through a diverse array of sensory channels. Commonly referred to as the `Internet of Things (IoT)' ecosystem, sensory data from motion, thermal, geolocation, depth, wireless signals, video, and audio are increasingly used to model the states of physical environments and the humans inside them. Despite the potential for understanding human wellbeing, controlling physical devices, and interconnecting smart cities, the community has seen limited benchmarks for building machine learning systems for IoT. Existing efforts are often specialized to a single sensory modality or prediction task, which makes it difficult to study and train large-scale models across many IoT sensors and tasks. To accelerate the development of new machine learning technologies for IoT, this paper proposes MultiIoT, the most expansive and unified IoT benchmark to date, encompassing over 1.15 million samples from 12 modalities and 8 real-world tasks. MultiIoT introduces unique challenges involving (1) generalizable learning from many sensory modalities, (2) multimodal interactions across long temporal ranges, (3) extreme heterogeneity due to unique structure and noise topologies in real-world sensors, and (4) complexity during training and inference. We evaluate a comprehensive set of models on MultiIoT, including modality and task-specific methods, multisensory and multitask supervised models, and large multisensory foundation models. Our results highlight opportunities for ML to make a significant impact in IoT, but many challenges in scalable learning from heterogeneous, long-range, and imperfect sensory modalities still persist. We release all code and data to accelerate future research in machine learning for IoT.