FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization

원문 초록

Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimization problems often require a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines. Existing benchmarks are limited to small or simplified examples far below real-world scale and complexity. We introduce FrontierOR, the first literature-grounded benchmark targeting LLM-generated algorithm efficiency on realistic large-scale optimization problems. FrontierOR includes 180 diverse tasks, each derived from a paper published in a top-tier operations research venue, with standardized instances and a hidden, expert-verified evaluation suite. We evaluate recent LLMs' one-shot algorithm generation and improvements achieved through several self-evolving frameworks. The results show that current models still struggle to move from executable formulations to efficient optimization algorithms. Even the strongest LLM succeeds on only 39% of Full-set task instances, where success is defined as returning a feasible solution with an objective gap of at most 1% relative to Gurobi within its runtime. On the Hard subset, test-time evolution raises this proportion from 51% for the strongest one-shot model to 78%. FrontierOR provides a continuously maintained testbed for evaluating LLM-designed optimization algorithms under a shared protocol and controlled computational settings. Code and data are publicly released at https://github.com/Minw913/FrontierOR.