CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling
원문 초록
The multi-turn checker of the Berkeley Function Calling Leaderboard (BFCL) checks only the final state of a simulated service, never the calls that produced it. Extra calls cost nothing, no function doc says what a function does, and the scripted user never answers a proposal, so a model that asks before deleting or paying fails the turn; 322 of the 734 reference turns in the base category act destructively without confirmation. We add effect information to BFCL without replacing it. Effect declarations for all 128 functions are inferred by executing each function in BFCL's own simulators and diffing the state; the remaining judgment, which writes are destructive, is made by four annotators, two humans and two language models (Fleiss' kappa = 0.75), and adjudicated. A judge-free replay scorer re-scores any logged BFCL run, reproduces BFCL's results on all 800 reference trajectories, and reports, alongside the official score, unrequested destructive calls, duplicate writes, redundant reads, surplus steps and their token cost. A confirmation-aware protocol adds a deterministic user that answers proposals. On 11 open models the effect-aware score preserves the official ranking (rho = 0.99) but changes the gaps: the leader loses 14.0 points to unrequested destructive calls, and surplus steps account for 2-28% of all input tokens. Effect hints in the function docs produce no measurable change for any of eight models, consistent with their advisory status in the Model Context Protocol. A user that answers recovers up to four points for the models that ask and none for the rest, with every comparison paired on the same conversations. The gains come from clarifying questions rather than requests for permission: of the 712 conversations that require a destructive call, no model completes more than 6 with every such call confirmed, a difference the official score does not record.
제출 이력
확인된 버전만 표시합니다. 전체 제출 이력은 arXiv 원문에서 확인하세요.
- arXiv:2611.02978v2제출 시각 미확인
- arXiv:2611.02978v1
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
공통 태그로 연결된 공개 논문이 아직 없습니다.
토론
이 논문과 연결된 포럼 글이 아직 없습니다.