EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision

원문 초록

Continuous episodic memory is a core capability for autonomous agents operating in dynamic, real-world environments, yet current streaming video benchmarks provide limited tools for diagnosing what models remember and for how long. We introduce EGOSTREAM, a diagnostic benchmark for streaming episodic memory evaluation in egocentric vision. EGOSTREAM organizes 2,250 curated questions along seven cognitive dimensions: detail, spatial, temporal, event, social, causal, and prospective memory. We introduce the Answer Validity Window (AVW), which specifies the temporal span an answer remains valid as the observed scene evolves. This allows us to expand the questions into 8,528 recall-conditioned evaluations, enabling controlled testing from instant to ultra-long-term recall while separating genuine model forgetting from natural world-state changes. We rigorously establish baseline performance through a unified streaming MLLM framework that compares several state-of-the-art memory-management mechanisms, covering sliding windows, attention sinks, KV-cache pruning, merging, adaptive head budgets, and offloading. Experiments on a unified Qwen3-VL backbone, with consistent trends on Molmo2-8B and InternVL3.5-8B, reveal that comparable aggregate accuracies mask starkly different memory profiles. For instance, token pruning preserves fine-grained details and temporal structure significantly better than token merging, adaptive head budgets yield the highest aggregate semantic accuracy, while quantized offloading rescues ultra-long-term recall. Ultimately, all in-cache memory policies operate well below real-time (>1s per frame), and top performing in-cache methods ceil at about 45% accuracy under a 9,804-token budget, exposing critical gaps in current architectures. EGOSTREAM provides the diagnostic testbed needed to close these gaps. Project website: https://coai-iplab.github.io/EgoStream/