Large language models (LLMs) significantly enhance the performance of various applications, but they are computationally intensive and energy-demanding. This makes it challenging to deploy them on devices with limited resources, such as personal computers and mobile/wearable devices, and results in substantial inference costs in resource-rich environments like cloud servers. To extend the use of LLMs, we introduce a low-rank decomposition approach to effectively compress these models, tailored to the requirements of specific applications. We observe that LLMs pretrained on general datasets contain many redundant components not needed for particular applications. Our method focuses on identifying and removing these redundant parts, retaining only the necessary elements for the target applications. Specifically, we represent the weight matrices of LLMs as a linear combination of base components. We then prune the irrelevant bases and enhance the model with new bases beneficial for specific applications. Deep compression results on the Llama 2-7b and -13B models, conducted on target applications including mathematical reasoning and code generation, show that our method significantly reduces model size while maintaining comparable accuracy to state-of-the-art low-rank compression techniques.
Multimodal conversational agents are highly desirable because they offer natural and human-like interaction. However, there is a lack of comprehensive end-to-end solutions to support collaborative development and benchmarking. While proprietary systems like GPT-4o and Gemini demonstrating impressive integration of audio, video, and text with response times of 200-250ms, challenges remain in balancing latency, accuracy, cost, and data privacy. To better understand and quantify these issues, we developed OpenOmni, an open-source, end-to-end pipeline benchmarking tool that integrates advanced technologies such as Speech-to-Text, Emotion Detection, Retrieval Augmented Generation, Large Language Models, along with the ability to integrate customized models. OpenOmni supports local and cloud deployment, ensuring data privacy and supporting latency and accuracy benchmarking. This flexible framework allows researchers to customize the pipeline, focusing on real bottlenecks and facilitating rapid proof-of-concept development. OpenOmni can significantly enhance applications like indoor assistance for visually impaired individuals, advancing human-computer interaction. Our demonstration video is available https://www.youtube.com/watch?v=zaSiT3clWqY, demo is available via https://openomni.ai4wa.com, code is available via https://github.com/AI4WA/OpenOmniFramework.
Assessing the extent of human edits on texts generated by Large Language Models (LLMs) is crucial to understanding the human-AI interactions and improving the quality of automated text generation systems. Existing edit distance metrics, such as Levenshtein, BLEU, ROUGE, and TER, often fail to accurately measure the effort required for post-editing, especially when edits involve substantial modifications, such as block operations. In this paper, we introduce a novel compression-based edit distance metric grounded in the Lempel-Ziv-77 algorithm, designed to quantify the amount of post-editing applied to LLM-generated texts. Our method leverages the properties of text compression to measure the informational difference between the original and edited texts. Through experiments on real-world human edits datasets, we demonstrate that our proposed metric is highly correlated with actual edit time and effort. We also show that LLMs exhibit an implicit understanding of editing speed, that aligns well with our metric. Furthermore, we compare our metric with existing ones, highlighting its advantages in capturing complex edits with linear computational efficiency. Our code and data are available at: https://github.com/NDV-tiime/CompressionDistance
The generation of incorrect images, such as depictions of people of color in Nazi-era uniforms by Gemini, frustrated users and harmed Google's reputation, motivating us to investigate the relationship between accurately reflecting factuality and promoting diversity and equity. In this study, we focus on 19 real-world statistics collected from authoritative sources. Using these statistics, we develop a checklist comprising objective and subjective queries to analyze behavior of large language models (LLMs) and text-to-image (T2I) models. Objective queries assess the models' ability to provide accurate world knowledge. In contrast, the design of subjective queries follows a key principle: statistical or experiential priors should not be overgeneralized to individuals, ensuring that models uphold diversity. These subjective queries are based on three common human cognitive errors that often result in social biases. We propose metrics to assess factuality and fairness, and formally prove the inherent trade-off between these two aspects. Results show that GPT-4o and DALL-E 3 perform notably well among six LLMs and four T2I models. Our code is publicly available at https://github.com/uclanlp/Fact-or-Fair.
Despite the advancements made in Visual Large Language Models (VLLMs), like text Large Language Models (LLMs), they have limitations in addressing questions that require real-time information or are knowledge-intensive. Indiscriminately adopting Retrieval Augmented Generation (RAG) techniques is an effective yet expensive way to enable models to answer queries beyond their knowledge scopes. To mitigate the dependence on retrieval and simultaneously maintain, or even improve, the performance benefits provided by retrieval, we propose a method to detect the knowledge boundary of VLLMs, allowing for more efficient use of techniques like RAG. Specifically, we propose a method with two variants that fine-tunes a VLLM on an automatically constructed dataset for boundary identification. Experimental results on various types of Visual Question Answering datasets show that our method successfully depicts a VLLM's knowledge boundary based on which we are able to reduce indiscriminate retrieval while maintaining or improving the performance. In addition, we show that the knowledge boundary identified by our method for one VLLM can be used as a surrogate boundary for other VLLMs. Code will be released at https://github.com/Chord-Chen-30/VLLM-KnowledgeBoundary
First-order logic (FOL) reasoning, which involves sequential deduction, is pivotal for intelligent systems and serves as a valuable task for evaluating reasoning capabilities, particularly in chain-of-thought (CoT) contexts. Existing benchmarks often rely on extensive human annotation or handcrafted templates, making it difficult to achieve the necessary complexity, scalability, and diversity for robust evaluation. To address these limitations, we propose a novel framework called ProverGen that synergizes the generative strengths of Large Language Models (LLMs) with the rigor and precision of symbolic provers, enabling the creation of a scalable, diverse, and high-quality FOL reasoning dataset, ProverQA. ProverQA is also distinguished by its inclusion of accessible and logically coherent intermediate reasoning steps for each problem. Our evaluation shows that state-of-the-art LLMs struggle to solve ProverQA problems, even with CoT prompting, highlighting the dataset's challenging nature. We also finetune Llama3.1-8B-Instruct on a separate training set generated by our framework. The finetuned model demonstrates consistent improvements on both in-distribution and out-of-distribution test sets, suggesting the value of our proposed data generation framework. Code available at: https://github.com/opendatalab/ProverGen
Computational Fluid Dynamics (CFD) is an essential simulation tool in various engineering disciplines, but it often requires substantial domain expertise and manual configuration, creating barriers to entry. We present Foam-Agent, a multi-agent framework that automates complex OpenFOAM-based CFD simulation workflows from natural language inputs. Our innovation includes (1) a hierarchical multi-index retrieval system with specialized indices for different simulation aspects, (2) a dependency-aware file generation system that provides consistency management across configuration files, and (3) an iterative error correction mechanism that diagnoses and resolves simulation failures without human intervention. Through comprehensive evaluation on the dataset of 110 simulation tasks, Foam-Agent achieves an 83.6% success rate with Claude 3.5 Sonnet, significantly outperforming existing frameworks (55.5% for MetaOpenFOAM and 37.3% for OpenFOAM-GPT). Ablation studies demonstrate the critical contribution of each system component, with the specialized error correction mechanism providing a 36.4% performance improvement. Foam-Agent substantially lowers the CFD expertise threshold while maintaining modeling accuracy, demonstrating the potential of specialized multi-agent systems to democratize access to complex scientific simulation tools. The code is public at https://github.com/csml-rpi/Foam-Agent
The acquisition of agentic capabilities has transformed LLMs from "knowledge providers" to "action executors", a trend that while expanding LLMs' capability boundaries, significantly increases their susceptibility to malicious use. Previous work has shown that current LLM-based agents execute numerous malicious tasks even without being attacked, indicating a deficiency in agentic use safety alignment during the post-training phase. To address this gap, we propose AgentAlign, a novel framework that leverages abstract behavior chains as a medium for safety alignment data synthesis. By instantiating these behavior chains in simulated environments with diverse tool instances, our framework enables the generation of highly authentic and executable instructions while capturing complex multi-step dynamics. The framework further ensures model utility by proportionally synthesizing benign instructions through non-malicious interpretations of behavior chains, precisely calibrating the boundary between helpfulness and harmlessness. Evaluation results on AgentHarm demonstrate that fine-tuning three families of open-source models using our method substantially improves their safety (35.8% to 79.5% improvement) while minimally impacting or even positively enhancing their helpfulness, outperforming various prompting methods. The dataset and code have both been open-sourced.
Recommendation systems (RS) aim to provide personalized content, but they face a challenge in unbiased learning due to selection bias, where users only interact with items they prefer. This bias leads to a distorted representation of user preferences, which hinders the accuracy and fairness of recommendations. To address the issue, various methods such as error imputation based, inverse propensity scoring, and doubly robust techniques have been developed. Despite the progress, from the structural causal model perspective, previous debiasing methods in RS assume the independence of the exogenous variables. In this paper, we release this assumption and propose a learning algorithm based on likelihood maximization to learn a prediction model. We first discuss the correlation and difference between unmeasured confounding and our scenario, then we propose a unified method that effectively handles latent exogenous variables. Specifically, our method models the data generation process with latent exogenous variables under mild normality assumptions. We then develop a Monte Carlo algorithm to numerically estimate the likelihood function. Extensive experiments on synthetic datasets and three real-world datasets demonstrate the effectiveness of our proposed method. The code is at https://github.com/WallaceSUI/kdd25-background-variable.
Mosquito-borne diseases pose a major global health risk, requiring early detection and proactive control of breeding sites to prevent outbreaks. In this paper, we present VisText-Mosquito, a multimodal dataset that integrates visual and textual data to support automated detection, segmentation, and reasoning for mosquito breeding site analysis. The dataset includes 1,828 annotated images for object detection, 142 images for water surface segmentation, and natural language reasoning texts linked to each image. The YOLOv9s model achieves the highest precision of 0.92926 and mAP@50 of 0.92891 for object detection, while YOLOv11n-Seg reaches a segmentation precision of 0.91587 and mAP@50 of 0.79795. For reasoning generation, our fine-tuned BLIP model achieves a final loss of 0.0028, with a BLEU score of 54.7, BERTScore of 0.91, and ROUGE-L of 0.87. This dataset and model framework emphasize the theme "Prevention is Better than Cure", showcasing how AI-based detection can proactively address mosquito-borne disease risks. The dataset and implementation code are publicly available at GitHub: https://github.com/adnanul-islam-jisun/VisText-Mosquito
The rapid advancement of speech generation models has heightened privacy and security concerns related to voice cloning (VC). Recent studies have investigated disrupting unauthorized voice cloning by introducing adversarial perturbations. However, determined attackers can mitigate these protective perturbations and successfully execute VC. In this study, we conduct the first systematic evaluation of these protective perturbations against VC under realistic threat models that include perturbation purification. Our findings reveal that while existing purification methods can neutralize a considerable portion of the protective perturbations, they still lead to distortions in the feature space of VC models, which degrades the performance of VC. From this perspective, we propose a novel two-stage purification method: (1) Purify the perturbed speech; (2) Refine it using phoneme guidance to align it with the clean speech distribution. Experimental results demonstrate that our method outperforms state-of-the-art purification methods in disrupting VC defenses. Our study reveals the limitations of adversarial perturbation-based VC defenses and underscores the urgent need for more robust solutions to mitigate the security and privacy risks posed by VC. The code and audio samples are available at https://de-antifake.github.io.
Despite rapid progress in end-to-end AI music generation, AI-driven modeling of professional Digital Signal Processing (DSP) workflows remains challenging. In particular, while there is growing interest in neural black-box modeling of audio effect graphs (e.g. reverb, compression, equalization), AI-based approaches struggle to replicate the nuanced signal flow and parameter interactions used in professional workflows. Existing differentiable plugin approaches often diverge from real-world tools, exhibiting inferior performance relative to simplified neural controllers under equivalent computational constraints. We introduce WildFX, a pipeline containerized with Docker for generating multi-track audio mixing datasets with rich effect graphs, powered by a professional Digital Audio Workstation (DAW) backend. WildFX supports seamless integration of cross-platform commercial plugins or any plugins in the wild, in VST/VST3/LV2/CLAP formats, enabling structural complexity (e.g., sidechains, crossovers) and achieving efficient parallelized processing. A minimalist metadata interface simplifies project/plugin configuration. Experiments demonstrate the pipeline's validity through blind estimation of mixing graphs, plugin/gain parameters, and its ability to bridge AI research with practical DSP demands. The code is available on: https://github.com/IsaacYQH/WildFX.
Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient. Code is available at https://github.com/dreamtheater123/TextPro-SLM
고위험 추론 작업은 투명하고 검증 가능한 워크플로를 필요로 하지만, 기존의 단일 모델 대규모 언어 모델(LLM)은 제로샷(zero-shot) 패러다임에서 환각(hallucination)과 낮은 해석 가능성 문제에 자주 어려움을 겪는다. 이러한 일반적인 AI 과제를 해결하기 위해, 협력적이고 다단계인 검증 과정을 시뮬레이션하는 다중 에이전트 감사 프레임워크를 제안한다. 이 아키텍처를 모듈형 LangChain 워크플로를 사용하여 민감한 임상 정신건강 스크리닝 영역에서 실증적으로 검증한다. 우리 프레임워크는 추론 과정을 Perception Agent, 지식 검색-증강 생성(RAG), 사고의 연쇄(Chain-of-Thought, CoT) 임상 추론, 그리고 핵심적인 Audit 검증 단계로 분해한다. 로컬에 배포한 오픈소스 모델을 사용하여 DAIC-WOZ 데이터셋에서 이 프레임워크를 평가했다. 실험 결과, 우리의 다중 에이전트 파이프라인이 단일 에이전트 베이스라인을 유의미하게 능가하며, PHQ-8 우울증 중증도 예측의 평균 절대 오차(MAE)를 5.35에서 5.02로 줄였음을 보여준다. 에이전트 간 검증 추적을 노출함으로써, 프레임워크는 추론 이탈(reasoning drift)을 완화하고 높은 해석 가능성을 갖는 진단 근거를 제공하며, 단독적인 모델 스케일링을 넘어 신뢰할 수 있는 AI 보조 의사결정 지원을 위한 일반화 가능한 패러다임을 제시한다. 재현성을 위해 데이터와 코드를 GitHub에 공개한다.
Can AI systems trained on the existing scientific record forecast the advances that will follow? We introduce PreScience, a dataset and benchmark for scientific forecasting built around 98K recent AI research papers, together with companion papers covering author publication histories and citation links, yielding 502K papers in total. The resulting paper records include titles, abstracts, disambiguated author identities, influential references, topic labels, citation trajectories, and metadata snapshotted to respect temporal cutoffs. We instantiate seven exemplar tasks: five paper-anchored tasks -- contribution generation, collaborator prediction, prior work selection, citation count prediction, and future combination prediction -- and two aggregate topic trend forecasting variants. We develop baselines ranging from simple heuristics and embedding methods to frontier language models and agentic systems, and introduce LACER, an LLM-based metric for evaluating similarity of generated contribution descriptions that agrees better with human judgments than existing metrics. Finally, we compose task models to generate a 12-month synthetic corpus and find that the resulting papers are systematically less diverse and less novel than human-authored research from the same period. We release the PreScience dataset (https://huggingface.co/datasets/allenai/prescience) and code (https://github.com/allenai/prescience).
강력한 추론은 모델의 지식뿐만 아니라 생성 과정에서 인지적 행동이 얼마나 효과적으로 발휘되는지에도 달려 있다. 기존 방법들은 명시적인 행동 수준 제어에 의존하는 경우가 많아, 추론 상태·과제·모델에 따라 실패 양상과 필요한 교정이 달라질 때 충분히 적응하지 못한다. 이를 위해 우리는 잠재 보상 조향(LRS, Latent Reward Steering)을 제안한다. LRS는 인지적 행동을 암묵적으로 담고 있는 희소 오토인코더(SAE, sparse-autoencoder) 잠재 상태를 최적화함으로써 인지적 행동을 촉진하는 적응형 추론 시간 프레임워크이다. 사전에 정의된 인지적 행동이나 그로부터 도출된 조향 방향에 의존하는 대신, LRS는 최종 답안의 정확성을 기준으로 추론 궤적 위에서 잠재 보상 모델을 학습하여 중간 잠재 상태의 품질을 추정한다. 추론 시에는 보상 그래디언트가 취약한 잠재 상태에 대한 상태별 교정 방향을 제공하며, 보상 및 신뢰도 게이트가 보상 신호가 취약하다고 표시한 상태에만 개입을 제한한다. 여러 추론 LLM 백본과 벤치마크에서 수행한 실험은 LRS가 다양한 기준선 대비 일관되게 성능을 향상시킴을 보여주며, 사후 분석을 통해 LRS가 원래의 추론 오류를 수정하는 바람직한 인지적 행동을 암묵적으로 촉진함을 확인하였다. 코드는 https://github.com/jiakanglee/Latent-Reward-Steering 에서 확인할 수 있다.
이미지 목표 기반 시각적 내비게이션(visual navigation)은 체화된 에이전트(embodied agent)의 핵심적인 능력입니다. 기존의 내비게이션 정책들은 경유지 궤적(waypoint trajectory)을 효율적으로 예측하지만 시각적 예측력(visual foresight)이 부족한 반면, 내비게이션 월드 모델은 미래의 관측을 예측할 수 있지만 종종 비용이 많이 드는 계획 롤아웃(planning rollout)을 필요로 합니다. 본 논문에서는 단일 확산(diffusion) 과정을 통해 미래의 시각적 관측과 연속적인 경유지 궤적을 동시에 생성하는 통합 월드-행동 모델인 UniNav를 제시합니다. 과거 프레임과 목표 이미지가 주어지면, UniNav는 단일 트랜스포머(transformer) 내에서 시각적 토큰과 경유지 토큰의 노이즈를 함께 제거(joint denoising)하여 공유된 프레임워크 안에서 미래 예측과 행동 생성을 통합합니다. 공간적 그라운딩(grounding)을 개선하기 위해 기하학 인지 카메라 토큰(geometry-aware camera token)을 통합합니다. 또한 궤적이 라벨링된 내비게이션 데이터와 비디오 전용 데이터를 모두 활용하여 학습함으로써, 모델이 경유지 주석이 없는 다양한 비디오의 이점을 누릴 수 있도록 합니다. 이 통합 프레임워크를 기반으로 두 가지 변형을 제안합니다. UniNav-Full은 해석 가능한 미래 관측과 이에 해당하는 궤적을 함께 예측하며, UniNav-Fast는 효율적인 궤적 예측을 위해 추론 시 미래 이미지 토큰을 제거합니다. 내비게이션 벤치마크 실험에 따르면, UniNav는 모든 데이터셋에 걸쳐 ATE에서 가장 강력한 베이스라인(baseline)을 능가합니다. 1단계 추론(one-step inference)을 통해 UniNav-Fast는 큰 정확도 저하 없이 0.1초의 지연 시간을 달성합니다. 코드는 공개될 예정입니다.
대규모 언어 모델(LLM)은 대화 속에서 생성과 수정을 반복하는 과정을 통해 사용자가 산출물(artifact)을 만들도록 돕는 경우가 많다. 여기서 어려운 점은 사용자가 수정 시 국소적인 변경만 지정하더라도, LLM이 관련된 의존성을 스스로 파악하여 산출물에서 영향을 받는 모든 부분에 수정을 전파해야 한다는 것이다. 본 논문은 산출물의 맥락과 그 의존성이 대화 기록 안에 내재해 있을 수 있는, 대화를 통해 생성된 산출물에 대한 LLM의 이러한 능력을 연구한다. 실용적인 활용을 위해, 이 새로운 설정에서 비용 효율적인 테스트 시점 연산(test-time compute)도 함께 탐구한다. 구체적으로, 이 설정을 위한 새로운 벤치마크를 소개하고, gpt-oss-20b/120b, gpt-5.4-mini, qwen3.5-9b/27b/122b를 사용하여 순차적 성찰(sequential reflection)과 병렬 샘플링(parallel sampling) 변형을 포함한 아홉 가지 수정 방법을 이 벤치마크에서 평가한다. 결과에 따르면 베이스라인은 68.3~93%의 정확도를 달성하며, 가장 비용 효율적인 방법은 LLM 기반 선택 또는 메도이드(medoid) 선택을 사용해 세 개의 병렬 샘플 중에서 선택하는 것으로, 정확도를 2.2~9.7% 향상시킨다. 코드와 데이터셋은 https://github.com/ntt-dkiku/llm-revision-propagation 에서 확인할 수 있다.
오디오 기반 얼굴 애니메이션은 몰입형 디지털 상호작용에 필수적이지만, 기존 프레임워크는 실시간 스트리밍과 고충실도 개인화를 동시에 만족시키기 어렵다. 현재 방법들은 지연을 유발하는 오디오 look-ahead에 의존하거나, 사용자에게 스크립트가 정해진 보정(calibration) 시퀀스를 녹화하여 사전 인코딩된 정적 정체성(identity) 임베딩을 만들도록 요구하는데, 이는 동적인 개인적 특성(idiosyncrasies)을 포착하지 못한다. 우리는 개인화된 오디오 기반 얼굴 모션 생성을 위한 end-to-end 프레임워크를 제안하며, 인과적(causal)이고 look-ahead가 없는(zero-lookahead) 스트리밍을 지원한다. 두 가지 핵심 혁신을 도입한다: (1) 전역 시간적 문맥과 고주파 조음(articulatory) 세부를 모두 포착하는 인과적 다중 해상도 모션 토크나이저, (2) 진행 중인 오디오와 모션을 공동으로 질의하여 비구조화된 참조 라이브러리에서 스타일 사전(stylistic priors)을 추출하는 멀티모달 스타일 리트리버. 큐레이션된, 고정 크기의, 또는 오디오 전용 스타일 뱅크로 제한된 기존 검색 메커니즘과 달리, 우리 설계는 대상 정체성의 임의 영상을 받아들여, 소수의 캐주얼하게 녹화된 클립만으로 개인화를 가능하게 한다. 이 구성 요소들을 통합함으로써, 우리 방법은 립싱크 정확도, 정체성 일관성, 지각적 사실감(perceived realism)에서 최신(state-of-the-art) 접근법을 능가하면서, 실시간 원격 현존(telepresence)의 스트리밍 제약을 유지한다. 코드는 https://github.com/xg-chu/Fallingwater 에서 이용 가능하다.
대규모 언어 모델(LLM)은 복잡한 추론을 위해 긴 사고의 연쇄(Chain-of-Thought, CoT) 궤적에 점점 더 의존하지만, 자기회귀(autoregressive) 생성은 상당한 메모리와 추론(inference) 비용을 유발한다. 잠재 추론 모델(LRM)은 중간 추론을 소수의 연속 잠재 벡터로 압축함으로써 더 효율적인 대안을 제공한다. 그러나 효율성에도 불구하고 LRM의 적대적 견고성(adversarial robustness)은 여전히 거의 탐구되지 않았다. 본 연구에서는 텍스트 및 멀티모달 설정 전반에 걸쳐 잠재 추론의 견고성을 체계적으로 평가하며, 8개 모델과 6개 벤치마크(benchmark)를 다룬다. 평가한 설정 전반에서 LRM은 적대적 교란(adversarial perturbation) 하에서 명시적 CoT 기준선(baseline)보다 일반적으로 덜 견고하며, 특히 화이트박스(white-box) 공격에서는 성능 저하가 현저하다. 추가 분석은 모달리티별로 뚜렷한 실패 양상을 보여준다. 텍스트 잠재 상태는 취약한 동역학과 특정 입력 패턴에 대한 높은 민감도를 보이는 반면, 멀티모달 모델의 잠재 상태는 입력 교란에 대해 대체로 불변(invariant)을 유지할 수 있으며 최종 예측에 미치는 영향이 제한적이다. 이러한 결과는 현재 잠재 추론 접근법의 견고성 한계를 드러내며, 암시적(implicit) 추론 시스템을 설계할 때 효율성과 견고성을 함께 고려해야 할 필요성을 강조한다. 연구 재현을 돕기 위해 코드를 오픈소스로 공개하였다. https://github.com/PKU-ML/latent-reasoning-model-assessment