PitchBench: Measuring Pitch Hearing in Audio-Language Models

원문 초록

Audio-language models (ALMs) are increasingly used in real-world applications that require music understanding, including tutoring, captioning, recommendation, and production. Yet existing benchmarks rarely evaluate a fundamental component of this understanding: pitch hearing. Pitch underpins the identification of notes, chords, and melodies and the grounding of captions in musical events, but it remains unclear how reliably ALMs perceive fine-grained pitch across sound sources, acoustic conditions, and response formats. We introduce PitchBench, an evaluation suite comprising 28 tasks that assess absolute and relative pitch perception under variations in loudness, note duration, sound source, time stretching, background noise, and other acoustic conditions. Tasks range from identifying isolated pitches to tracking voices in four-part musical textures. Evaluating frontier ALMs, we find that pitch hearing remains highly unreliable. Models perform poorly across settings, with accuracy varying sharply depending on the notation or prompt format, indicating instruction drift. Specialized DSP and neural pitch baselines achieve mean scores of approximately 73.2% and 75.2%, respectively, compared with approximately 53.9% for the strongest ALM. We release PitchBench as an evaluation benchmark, a training dataset, and a Python package with data generation tools to support future research on pitch-aware audio-language modeling.