Personalized tool utilization is essential for aligning large language models (LLMs) with user preference in interaction scenarios with various tools. However, most of the current benchmarks primarily focus on either personalization of text generation or direct tool-utilizing, without considering both. In this work, we introduce a novel benchmark ETAPP for evaluating personalized tool invocation, establishing a sandbox environment, and a comprehensive dataset of 800 testing cases covering diverse user profiles. To improve the accuracy of our evaluation, we propose a key-point-based LLM evaluation method, mitigating biases in the LLM-as-a-judge system by manually annotating key points for each test case and providing them to LLM as the reference. Additionally, we evaluate the excellent LLMs and provide an in-depth analysis. Furthermore, we investigate the impact of different tool-invoking strategies on LLMs' personalization performance and the effects of fine-tuning in our task. The effectiveness of our preference-setting and key-point-based evaluation method is also validated. Our findings offer insights into improving personalized LLM agents. Our Code is available at https://github.com/hypasd-art/ETAPP.
Autoformalization, the automatic translation of mathematical content from natural language into machine-verifiable formal languages, has seen significant progress driven by advances in large language models (LLMs). Nonetheless, a primary barrier to further improvements is the limited availability of parallel corpora that map informal mathematical text to its formal counterpart. To address this limitation, we propose ATLAS (Autoformalizing Theorems through Lifting, Augmentation, and Synthesis of Data), a novel data generation framework designed to produce large-scale, high-quality parallel corpora of theorem statements. Distinct from prior approaches, ATLAS begins with a concept repository, accelerates the improvement of student model through expert iteration combined with knowledge distillation, and introduces two novel augmentation strategies that exploit the structural characteristics of formal languages. With the proposed ATLAS running for 10 iterations, we construct an undergraduate-level dataset comprising 117k theorem statements and develop ATLAS Translator, which demonstrates statistically significant improvements over both the HERALD Translator and the Kimina-Autoformalizer across all benchmarks ($p<0.05$, two-sided t-test), achieving a new state of the art. The datasets, model, and code will be released to the public soon.
Recently, video-based large language models (video-based LLMs) have achieved impressive performance across various video comprehension tasks. However, this rapid advancement raises significant privacy and security concerns, particularly regarding the unauthorized use of personal video data in automated annotation by video-based LLMs. These unauthorized annotated video-text pairs can then be used to improve the performance of downstream tasks, such as text-to-video generation. To safeguard personal videos from unauthorized use, we propose two series of protective video watermarks with imperceptible adversarial perturbations, named Ramblings and Mutes. Concretely, Ramblings aim to mislead video-based LLMs into generating inaccurate captions for the videos, thereby degrading the quality of video annotations through inconsistencies between video content and captions. Mutes, on the other hand, are designed to prompt video-based LLMs to produce exceptionally brief captions, lacking descriptive detail. Extensive experiments demonstrate that our video watermarking methods effectively protect video data by significantly reducing video annotation performance across various video-based LLMs, showcasing both stealthiness and robustness in protecting personal video content. Our code is available at https://github.com/ttthhl/Protecting_Your_Video_Content.
In this paper, we introduce EasyEdit2, a framework designed to enable plug-and-play adjustability for controlling Large Language Model (LLM) behaviors. EasyEdit2 supports a wide range of test-time interventions, including safety, sentiment, personality, reasoning patterns, factuality, and language features. Unlike its predecessor, EasyEdit2 features a new architecture specifically designed for seamless model steering. It comprises key modules such as the steering vector generator and the steering vector applier, which enable automatic generation and application of steering vectors to influence the model's behavior without modifying its parameters. One of the main advantages of EasyEdit2 is its ease of use-users do not need extensive technical knowledge. With just a single example, they can effectively guide and adjust the model's responses, making precise control both accessible and efficient. Empirically, we report model steering performance across different LLMs, demonstrating the effectiveness of these techniques. We have released the source code on GitHub at https://github.com/zjunlp/EasyEdit along with a demonstration notebook. In addition, we provide a demo video at https://zjunlp.github.io/project/EasyEdit2/video for a quick introduction.
Recent advances in test-time scaling have enabled Large Language Models (LLMs) to display sophisticated reasoning abilities via extended Chain-of-Thought (CoT) generation. Despite their potential, these Reasoning LLMs (RLMs) often demonstrate counterintuitive and unstable behaviors, such as performance degradation under few-shot prompting, that challenge our current understanding of RLMs. In this work, we introduce a unified graph-based analytical framework for better modeling the reasoning processes of RLMs. Our method first clusters long, verbose CoT outputs into semantically coherent reasoning steps, then constructs directed reasoning graphs to capture contextual and logical dependencies among these steps. Through comprehensive analysis across models and prompting regimes, we reveal that structural properties, such as exploration density, branching, and convergence ratios, strongly correlate with reasoning accuracy. Our findings demonstrate how prompting strategies substantially reshape the internal reasoning structure of RLMs, directly affecting task outcomes. The proposed framework not only enables quantitative evaluation of reasoning quality beyond conventional metrics but also provides practical insights for prompt engineering and the cognitive analysis of LLMs. Code and resources will be released to facilitate future research in this direction.
We investigate improving the retrieval effectiveness of embedding models through the lens of corpus-specific fine-tuning. Prior work has shown that fine-tuning with queries generated using a dataset's retrieval corpus can boost retrieval effectiveness for the dataset. However, we find that surprisingly, fine-tuning using the conventional InfoNCE contrastive loss often reduces effectiveness in state-of-the-art models. To overcome this, we revisit cross-encoder listwise distillation and demonstrate that, unlike using contrastive learning alone, listwise distillation can help more consistently improve retrieval effectiveness across multiple datasets. Additionally, we show that synthesizing more training data using diverse query types (such as claims, keywords, and questions) yields greater effectiveness than using any single query type alone, regardless of the query type used in evaluation. Our findings further indicate that synthetic queries offer comparable utility to human-written queries for training. We use our approach to train an embedding model that achieves state-of-the-art effectiveness among BERT embedding models. We release our model and both query generation and training code to facilitate further research.
The rapid advancement of GenAI technology over the past few years has significantly contributed towards highly realistic deepfake content generation. Despite ongoing efforts, the research community still lacks a large-scale and reasoning capability driven deepfake benchmark dataset specifically tailored for person-centric object, context and scene manipulations. In this paper, we address this gap by introducing MultiFakeVerse, a large scale person-centric deepfake dataset, comprising 845,286 images generated through manipulation suggestions and image manipulations both derived from vision-language models (VLM). The VLM instructions were specifically targeted towards modifications to individuals or contextual elements of a scene that influence human perception of importance, intent, or narrative. This VLM-driven approach enables semantic, context-aware alterations such as modifying actions, scenes, and human-object interactions rather than synthetic or low-level identity swaps and region-specific edits that are common in existing datasets. Our experiments reveal that current state-of-the-art deepfake detection models and human observers struggle to detect these subtle yet meaningful manipulations. The code and dataset are available on \href{https://github.com/Parul-Gupta/MultiFakeVerse}{GitHub}.
Preference alignment in diffusion models has primarily focused on benign human preferences (e.g., aesthetic). In this paper, we propose a novel perspective: framing unrestricted adversarial example generation as a problem of aligning with adversary preferences. Unlike benign alignment, adversarial alignment involves two inherently conflicting preferences: visual consistency and attack effectiveness, which often lead to unstable optimization and reward hacking (e.g., reducing visual quality to improve attack success). To address this, we propose APA (Adversary Preferences Alignment), a two-stage framework that decouples conflicting preferences and optimizes each with differentiable rewards. In the first stage, APA fine-tunes LoRA to improve visual consistency using rule-based similarity reward. In the second stage, APA updates either the image latent or prompt embedding based on feedback from a substitute classifier, guided by trajectory-level and step-wise rewards. To enhance black-box transferability, we further incorporate a diffusion augmentation strategy. Experiments demonstrate that APA achieves significantly better attack transferability while maintaining high visual consistency, inspiring further research to approach adversarial attacks from an alignment perspective. Code will be available at https://github.com/deep-kaixun/APA.
Generative models, with their success in image and video generation, have recently been explored for synthesizing effective neural network weights. These approaches take trained neural network checkpoints as training data, and aim to generate high-performing neural network weights during inference. In this work, we examine four representative methods on their ability to generate novel model weights, i.e., weights that are different from the checkpoints seen during training. Surprisingly, we find that these methods synthesize weights largely by memorization: they produce either replicas, or at best simple interpolations, of the training checkpoints. Current methods fail to outperform simple baselines, such as adding noise to the weights or taking a simple weight ensemble, in obtaining different and simultaneously high-performing models. We further show that this memorization cannot be effectively mitigated by modifying modeling factors commonly associated with memorization in image diffusion models, or applying data augmentations. Our findings provide a realistic assessment of what types of data current generative models can model, and highlight the need for more careful evaluation of generative models in new domains. Our code is available at https://github.com/boyazeng/weight_memorization.
Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in deployment and inference. While structured pruning of model parameters offers a promising way to reduce computational costs at deployment time, current methods primarily focus on single model pruning. In this work, we develop a novel strategy to compress models by strategically combining or merging layers from finetuned model variants, which preserves the original model's abilities by aggregating capabilities accentuated in different finetunes. We pose the optimal tailoring of these LLMs as a zero-order optimization problem, adopting a search space that supports three different operations: (1) Layer removal, (2) Layer selection from different candidate models, and (3) Layer merging. Our experiments demonstrate that this approach leads to competitive model pruning, for example, for the Llama2-13B model families, our compressed models maintain approximately 97.3\% of the original performance while removing $\sim25\%$ of parameters, significantly outperforming previous state-of-the-art methods. The code is available at https://github.com/Guinan-Su/auto-merge-llm.
Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorization creates a tension between aggregation fidelity and local optimization continuity. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch. We propose FedPA-LoRA, a product-aligned framework that addresses both mismatches and establishes convergence bounds for homogeneous and heterogeneous ranks. Clients preserve local factors across rounds to eliminate initialization mismatch and align their products with rank-specific global references to reduce client drift. The server aggregates heterogeneous-rank updates in product space to mitigate aggregation mismatch and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. By decoupling local and reference ranks, FedPA-LoRA supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show consistent gains under data and resource heterogeneity, including a 6.82 percentage-point improvement in average GLUE accuracy under heterogeneous ranks. The code is available at https://github.com/Juseok-Jeon/FedPA-LoRA.
Interpreting the concepts encoded by individual neurons in deep neural networks is a crucial step towards understanding their complex decision-making processes and ensuring AI safety. Despite recent progress in neuron labeling, existing methods often limit the search space to predefined concept vocabularies or produce overly specific descriptions that fail to capture higher-order, global concepts. We introduce LINE, a novel, training-free iterative approach tailored for open-vocabulary concept labeling in vision models. Operating in a strictly gray-box setting, LINE leverages a large language model and a text-to-image generator to iteratively propose and refine concepts in a closed loop, guided by activation history. We demonstrate that LINE achieves state-of-the-art performance across multiple model architectures, yielding AUC improvements of up to 0.15 on ImageNet and 0.05 on Places365, while discovering, on average, 25% of new concepts missed by predefined vocabularies. Beyond identifying the top concept, LINE provides a complete generation history, enabling polysemanticity evaluation and producing supporting visual explanations that rival gradient-dependent activation maximization methods. The LINE source code is available at https://github.com/CredibleAI/LINE
검색-증강 생성(RAG)은 외부 지식 기반을 활용해 제로샷 시계열(TS) 예측을 향상시키지만, 기존 접근법은 검색된 샘플과 쿼리를 융합할 때 입력 수준의 관련성을 간과한다. 우리는 모든 검색 결과가 동등하게 유용한 것은 아니며, 관련 없는 결과가 성능을 저하시킬 수 있다고 주장한다. 이를 위해 쿼리-검색 크로스-어텐션을 통해 쿼리와 관련된 검색 샘플에 선택적으로 어텐션하는 제로샷 RAG 기반 예측 프레임워크인 Cross-RAG를 제안한다. 쿼리와 검색된 샘플 간의 입력 수준 관련성을 모델링함으로써 Cross-RAG는 1) 쿼리 자체, 2) 검색된 샘플, 3) 이들 간의 관계적 상호작용이라는 세 가지 정보원을 공동으로 통합한다. 특히 이러한 입력 인식형 설계는 검색 샘플 수 $k$가 증가함에 따라 Cross-RAG가 안정적으로 유지되도록 하며, 크로스-어텐션이 없는 기존 방법들은 관련 없는 검색으로 인한 성능 저하를 피하기 위해 $k$를 세심하게 튜닝해야 한다. 광범위한 실험을 통해 Cross-RAG가 다양한 TSFM 백본과 여러 RAG 방법에서 제로샷 예측 성능을 일관되게 개선함을 보였으며, 추가 분석은 다양한 검색 시나리오 전반에서 그 효과를 확인한다. 코드는 https://github.com/seunghan96/cross-rag/ 에서 제공된다.
수치적 점수 예측을 넘어, 최근 자동 에세이 채점(automated essay scoring) 연구는 근거와 실행 가능한 조언을 제공하는 고품질 피드백 생성을 점점 더 강조하고 있습니다. 전문가 주석(annotation)의 높은 비용을 완화하기 위해, 기존 연구는 에세이 평가 모델 학습 시 일반적으로 LLM 생성 피드백에 의존해 왔습니다. 그러나 이러한 피드백은 명시적인 품질 검증(validation) 없이 사용되는 경우가 많아, 다운스트림 애플리케이션에서 노이즈가 전파되는 결과를 초래합니다. 이러한 제한 사항을 해결하기 위해, 우리는 구체성(specificity), 유용성(helpfulness), 타당성(validity)의 세 가지 교육학적 근거에 기반한 차원을 따라 LLM 생성 에세이 피드백을 평가하는 LLM 기반 프레임워크인 FeedEval을 제안합니다. FeedEval은 본 연구에서 큐레이션된 데이터셋(dataset)으로 학습된 차원별 전문 LLM 평가자를 활용하여 여러 피드백 후보를 평가하고, 다운스트림 사용을 위한 고품질 피드백을 선정합니다. ASAP++ 벤치마크(benchmark)에 대한 실험 결과, FeedEval은 인간 전문가의 판단과 밀접하게 일치하며, FeedEval을 통해 필터링된 고품질 피드백으로 학습된 에세이 채점 모델은 우수한 채점 성능을 달성함을 보여줍니다. 더불어 소형 LLM을 활용한 수정 실험을 통해, FeedEval이 식별한 고품질 피드백이 더욱 효과적인 에세이 수정으로 이어진다는 것을 확인했습니다. 우리는 코드와 큐레이션된 데이터셋을 다음 링크에서 공개합니다: https://github.com/BBeeChu/FeedEval.git.
LoRA 파인튜닝된 확산 모델(diffusion model)이 학습 이미지를 암기(memorization)해 저작권 콘텐츠를 재현하는 문제를 해결하기 위해, Base-Anchored Filtering(BAF)은 학습 데이터나 추론 과정 접근 없이 LoRA 업데이트를 SVD로 스펙트럼 채널로 분해한 뒤 사전학습 백본의 주요 부분공간(principal subspace)과의 정렬 정도를 기준으로 암기 관련 채널을 선별적으로 억제한다. 여러 데이터셋과 확산 백본 실험에서 BAF는 생성 품질을 유지하거나 향상시키면서 암기를 일관되게 감소시킴을 입증했다.
언어 모델(language model)은 순차적 처리기이지만, 장기 과제 수행형 에이전시(long-horizon agency)에는 모델 가중치와 활성 컨텍스트를 넘어서는 외부 정보와 연산이 필요합니다. Prime Agent는 장기 평가와 코딩 에이전트(agent) 워크플로를 위한 오픈소스 하네스(harness)입니다. 지속적인 IPython REPL은 프로그래밍 방식의 컨텍스트 처리와 테스트 시점 연산(test-time compute)을 위해 재귀 언어 모델(Recursive Language Model) 추상화를 따르며, Continual Harness는 궤적(trajectory)에 걸쳐 히스토리, 메모리, 스킬, 프롬프트, 서브에이전트 명세를 보존합니다. 재귀적 서브에이전트는 에이전트 간 직접 통신을 통해 협업하고, Agents View를 통해 사람이 데몬 기반 세션을 살펴보고 관리할 수 있습니다. Prime Agent는 실행, 복구, 검증, 자원 회계(resource accounting)를 표준화하면서 전략 수립은 모델에 맡깁니다. 이러한 낮은 마찰의 표현력 높은 막(membrane)은 하네스의 실패가 모델의 실패로 이어지는 것을 막고, 측정 결과가 모델이 지닌 진정한 최대 잠재 역량에 더 가까워지도록 합니다. Prime Agent는 ARC-AGI-3 RHAE Best@1을 30%에서 95.5%로 끌어올렸으며, 장문 컨텍스트 코딩, GPU 커널(kernel) 생성, 에뮬레이터 구축, 자율 nanoGPT 스피드런 전반에서 네이티브 하네스 및 널리 쓰이는 하네스와 동등하거나 더 나은 성능을 보였습니다. Factorio에서는 개선(refinement)을 통해 기술 발전이 지속적으로 이루어질 수 있고, 전담 서브에이전트를 통해 작업을 병렬화할 수 있음을 확인했습니다. 코드는 https://github.com/PrimeIntellect-ai/prime-agent 에서 확인할 수 있습니다.
다양하고 고품질인 여러 개의 솔루션을 생성하는 것은 코드 테스트 생성 및 신약 개발과 같은 많은 응용 분야에서 가치가 있습니다. 그러나 대규모 언어 모델(LLM)은 추론(inference) 중에 신뢰도가 높은 단일 솔루션으로 수렴하는 경향이 있어, 다른 유효한 대안 솔루션 경로에 대한 탐험(exploration)을 제한합니다. 기존의 테스트 타임(test-time) 방법들은 트리 형태의 탐색을 통해 다양성을 촉진하고 응답 수준의 의미론적 임베딩을 사용하여 의미론적으로 유사한 분기를 가지치기(pruning)합니다. 그러나 우리는 이러한 임베딩이 언어적 및 문체적 유사성에 의해 쉽게 교란되어 진정으로 구별되는 솔루션 경로를 식별하기 어렵게 만든다는 점을 발견했습니다. 이를 해결하기 위해 중간 추론 경로로부터 LLM이 도달할 잠재적 정답을 프로빙(probing)하는 Answer Probing을 소개합니다. 우리는 프로빙된 정답의 은닉 상태(hidden state)가 의미론적 임베딩보다 구별되는 솔루션 경로를 더 효과적으로 구분하며, 프로빙된 정답의 펄플렉서티(perplexity)가 추론 정확도를 위한 실용적인 대리 지표로 작용함을 입증합니다. 이러한 발견을 바탕으로 프로빙된 정답의 은닉 상태 유사도와 펄플렉서티를 통해 트리 탐색을 유도하는 Answer Probing-Guided Tree Search (APTS)를 제안합니다. 두 개의 LLM을 대상으로 세 가지 추론 태스크에서 수행한 실험은 APTS가 솔루션 다양성을 일관되게 향상시켜 그 효과성과 견고성을 입증함을 보여줍니다.
대규모 언어 모델 에이전트는 알파를 발견할 수 있지만, 현재 방법에는 세 가지 약점이 있다. 검색은 실행 중에 적응할 수 없고, 자동화는 보통 알파 생성에서 끝나며 라이브러리 선택과 모델 선택은 수동으로 남아 있으며, 알파 발견 과정에서 루프 피드백이나 코드 문제로 테스트 윈도우를 엿볼 수 있다. 우리는 정량 연구를 하나의 예산 제약 검색 문제로 보는 자기 진화형 검색 과정인 AutoScientist-Quant를 제안한다. 단일 컨트롤러는 남은 예산을 바탕으로 모든 결정을 내리며, 각 라운드마다 개선·결합·전환·중단 여부, 확장할 노드, 생성할 알파 수, 공유 메모리에서 과거 궤적을 어떻게 검색할지를 선택한다. 동일한 코어가 이어서 라이브러리에서 선택하고 모델을 튜닝하여 가설에서 배포 가능한 전략까지 루프를 닫는다. 또한 선행 연구에서 재사용한 평가 파이프라인을 검토하고 두 가지 lookahead 문제를 수정하며, 피드백 윈도우와 홀드아웃 테스트 윈도우를 분리하여 모든 비교가 진정한 일반화를 검증하도록 한다. CSI 유니버스에서 본 프레임워크는 거의 모든 설정에서 거의 모든 지표의 최고 값을 달성했으며, 이러한 결론은 여러 백본과 시장에서도 일관되게 성립한다.
자기회귀 대규모 언어 모델(LLM)은 성능 면에서 빠르게 발전해 왔으나, 모델 규모가 커짐에 따라 추론 시 상당한 계산 및 메모리 비용이 발생합니다. 지식 증류(KD)는 이산 확률 분포의 정렬을 통해 거대한 교사 모델의 지식을 더 작은 학생 모델로 전달함으로써 실용적인 해결책을 제공합니다. 그러나 기존의 LLM 대상 지식 증류 방식은 주로 토큰 수준의 의미론적 정보를 명시적으로 활용하지 않고 각 어휘 인덱스의 확률값을 통해 차이를 평가하는 다이버전스에 의존합니다. 본 논문에서는 토큰 임베딩에서 도출된 비용 행렬을 활용하는 바세르슈타인 기반 거리(Wasserstein-based distance)를 통해 토큰 수준의 의미론적 정보를 반영하는 LLM 대상 바세르슈타인 기반 지식 증류(WASD)를 제안합니다. 계산적 다루기 쉬움(tractability)을 보장하기 위해 싱크혼 다이버전스(Sinkhorn divergence)를 도입하고, 추가적인 네트워크 없이도 효율적으로 최적화할 수 있는 그래디언트 동등 목적 함수를 유도합니다. 다양한 LLM 제품군과 규모에 걸친 실험 결과, WASD는 인스트럭션 수행, 수학적 추론, 코드 생성을 포함한 다양한 태스크에서 증류 성능을 일관되게 향상시키는 것으로 나타났습니다. 본 연구 결과는 LLM 증류에서 효과적인 분포 정렬을 위해 토큰 공간에 인코딩된 의미론적 정보의 중요성을 강조합니다. 구현 코드는 https://github.com/aailab-kaist/WASD 에서 공개되어 있습니다.
LiDAR-based 3D single object tracking is a challenging issue in robotics and autonomous driving. Currently, existing approaches usually suffer from the problem that objects at long distance often have very sparse or partially-occluded point clouds, which makes the features extracted by the model ambiguous. Ambiguous features will make it hard to locate the target object and finally lead to bad tracking results. To solve this problem, we utilize the powerful Transformer architecture and propose a Point-Track-Transformer (PTT) module for point cloud-based 3D single object tracking task. Specifically, PTT module generates fine-tuned attention features by computing attention weights, which guides the tracker focusing on the important features of the target and improves the tracking ability in complex scenarios. To evaluate our PTT module, we embed PTT into the dominant method and construct a novel 3D SOT tracker named PTT-Net. In PTT-Net, we embed PTT into the voting stage and proposal generation stage, respectively. PTT module in the voting stage could model the interactions among point patches, which learns context-dependent features. Meanwhile, PTT module in the proposal generation stage could capture the contextual information between object and background. We evaluate our PTT-Net on KITTI and NuScenes datasets. Experimental results demonstrate the effectiveness of PTT module and the superiority of PTT-Net, which surpasses the baseline by a noticeable margin, ~10% in the Car category. Meanwhile, our method also has a significant performance improvement in sparse scenarios. In general, the combination of transformer and tracking pipeline enables our PTT-Net to achieve state-of-the-art performance on both two datasets. Additionally, PTT-Net could run in real-time at 40FPS on NVIDIA 1080Ti GPU. Our code is open-sourced for the research community at https://github.com/shanjiayao/PTT.