ROLECUE: Diagnosing Cross-Lingual Role-Binding Failures in Vision-Language Models

원문 초록

Vision-language models (VLMs) can recognize events in images and parse sentences, but it remains unclear when they bind visual Agent-Patient structure to linguistic semantic roles across languages whose role cues depend on morphology, word order or construction. We introduce ROLECUE, a controlled diagnostic that pairs 192 validated two-person action images with German, Turkish, Japanese, English, Korean and Bangla sentences in canonical and patient-first orders. The evaluation separates image-only role recognition, text-only role parsing, direct image-sentence matching, response-compliance behavior and an external modular composition of the model's own component predictions. Across eight recent open-weight VLMs, ROLECUE reveals a structured spectrum: some models fail visual or linguistic component tests, some largely solve the diagnostic, some collapse to constant semantic assent and several show component-joint gaps in which reliable image-only and text-only role predictions do not survive direct joint inference. The largest non-degenerate gaps reach 44.9 percentage points, with final bootstrap intervals confirming that the effect is not a point-estimate artifact. These results reveal multilingual component-joint performance gaps and acceptance biases hidden by aggregate VLM benchmarks and show why component diagnostics are necessary before interpreting joint multimodal errors.