RAP: Research Attention Prediction in Language Agents

원문 초록

LLM agents increasingly support research planning and execution, where judgments about current and anticipated research activity inform decisions. Yet whether they can infer changes across a whole field from the partial evidence they retrieve through online scholarly search remains unclear. We introduce Research Attention Prediction (RAP), a literature search benchmark for evaluating this ability. We build a local search environment from 356,357 dated arXiv papers and construct two complementary evaluation tasks. The benchmark comprises a Core set of 2,160 questions across 54 AI/ML fields and an Extended set covering 269 fields. Our evaluation and analysis of LLM agents show that search substantially improves stronger agents' recognition of recent changes, while forecasting gains remain limited or negative. Agents update forecasts across dates and adjust their investigations to forecasting goals, but these adaptations do not reliably yield accurate predictions of future shifts. Even for recent changes, final judgments can diverge from what the retrieved papers indicate. These findings show that agents can recognise some recent changes, yet reliably anticipating future research activity remains difficult despite changes in their answers and investigations.