대규모 언어 모델(LLM)이 빠르게 확산되면서 모델의 취약점을 밝혀내기 위한 신뢰할 수 있는 안전성 평가의 필요성이 커지고 있다. 이를 위해 수많은 LLM 안전성 평가 벤치마크가 제안되었다. 그러나 기존 벤치마크는 대체로 많은 노동력이 드는 수작업 큐레이션에 의존하며, 이로 인해 시간과 자원이 과도하게 소모된다. 또한 상당한 중복성을 보이고 난이도도 제한적이다. 이러한 문제를 완화하기 위해 본 논문에서는 LLM 안전성 벤치마크 구축을 자동화하도록 설계된 최초의 에이전트 플로우(agent-flow) 시스템인 SafetyFlow를 소개한다. SafetyFlow는 7개의 특화된 에이전트를 조율하여 사람의 개입 없이 단 4일 만에 포괄적인 안전성 벤치마크를 자동으로 구축할 수 있으며, 시간과 자원 비용을 크게 줄인다. 다양한 도구를 갖춘 SafetyFlow의 에이전트들은 자동화된 파이프라인에 인간의 전문성을 통합하면서도 프로세스와 비용의 통제 가능성을 보장한다. 최종적으로 구축된 데이터셋인 SafetyFlowBench는 중복성이 낮고 강한 변별력을 갖춘 23,446개의 쿼리를 포함한다. 본 연구의 기여는 최초의 완전 자동화된 벤치마킹 파이프라인과 포괄적인 안전성 벤치마크를 제시한 것이다. 우리는 구축한 데이터셋에서 49개의 최신 LLM의 안전성을 평가하고, 제안 방법의 효과성과 효율성을 검증하기 위한 폭넓은 실험을 수행한다.
Production large language-model (LLM) agents are increasingly deployed not as lone problem-solvers but as managers: a main model creates specialized subagents, delegates work, and orchestrates their parallel, asynchronous returns through dynamic workflows. Whether one model can actually run such a team is largely unmeasured: existing benchmarks score a policy's own task-solving or a fixed multi-agent system's emergent behavior, but none isolate the management ability of the single LLM acting as leader. We introduce ClawArena-Team, a benchmark of 41 multi-turn, multimodal, multi-directory scenarios spanning 258 evaluation rounds and 72 staged updates that measures this management ability. The main agent is deliberately constrained: it natively perceives only text and directly accesses only part of the workspace. It commands a fixed, locally served subagent pool, so score differences reflect management skill, not raw capability. All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor. Across twelve proprietary, community-hosted, and self-hosted models, experiments show that the management bottleneck is privilege granting rather than perception (no model exceeds 50% workspace-permission precision); that cost and management quality are decoupled (API cost spans over 100 times while the overall score spans under 4 times, with the cheapest open models on the Pareto frontier); and that most leaderboard scores cluster within a 9.9-point band while orchestration behaviors diverge by more than an order of magnitude. Code and data will be released.
LLM agents are increasingly improved without weight updates by evolving a natural-language artifact, such as reflections, workflows, playbooks, cheatsheets, or optimized prompts, that conditions a frozen policy. Such methods are typically reported as wins on the single benchmark where they help. We study them apples-to-apples and surface a sharper picture. We introduce RSEA, a Recursive Self-Evolving Agent that carries a compact three-layer natural-language state: an imperative strategy, reusable skills, and a procedural playbook. Across generations, RSEA rewrites all three layers from its own trajectories and commits a candidate only if it does not regress on a disjoint held-out split, using a strict keep-better gate. Across four diverse benchmarks, ALFWorld, GAIA, (τ)-bench, and WebShop, and six faithful baselines, ReAct, Reflexion, GEPA, AWM, ACE, and Dynamic Cheatsheet, all evaluated on one shared local backbone, we find three main results. First, no artifact universally wins. RSEA is the strongest single-pass method on ALFWorld, reaching 69.3% compared with 64.6% for ReAct (McNemar (p=0.015)), and reaches 79.4% with retry, the best overall result. However, concrete-workflow induction, represented by AWM, is best on the strong-backbone tool-use tasks. Second, unguarded context evolution is high-variance and unsafe. Dynamic Cheatsheet, which curates context online without a held-out gate, is near-best on ALFWorld at 70.7%, yet collapses on WebShop, with a score of 0.14 compared with 0.43 for ReAct. Third, RSEA's strict held-out selection is what makes recursive self-evolution monotone-safe: it never significantly underperforms the base agent on any benchmark and falls back to vanilla ReAct when evolved context would hurt.