The Shadow Price of Intelligence: Quality Degradation and Retry Feedback in LLM Inference
Large language model providers are compute constrained, and a common response to congestion is to degrade service: route queries to smaller models, cut reasoning budgets, or shorten context windows. A user who receives an unsatisfactory answer either retries, adding load while the system is busy, or abandons, possibly as churn. We model inference serving as a control problem in which service quality is the provider's decision. A two-regime fluid approximation with retries folded into the service rate gives closed-form trajectories and crossing times, so the dynamic program over quality tiers needs no simulation. We characterize a region in which a weaker tier uses less energy per task but more capacity per task, and show that in the fluid model reactive throttling to such a tier can keep the queue from draining. With heterogeneous customers the problem is a transportation problem with retry-inflated capacity coefficients; optimal routing follows a $c\mu$-type index, and the dual of the capacity constraint gives the marginal cost of one more query by class and time. We illustrate the method on five instances calibrated to public benchmark data.