Measuring CLEVRness: Blackbox testing of Visual Reasoning Models
원문 초록
How can we measure the reasoning capabilities of intelligence systems? Visual question answering provides a convenient framework for testing the model's abilities by interrogating the model through questions about the scene. However, despite scores of various visual QA datasets and architectures, which sometimes yield even a super-human performance, the question of whether those architectures can actually reason remains open to debate. To answer this, we extend the visual question answering framework and propose the following behavioral test in the form of a two-player game. We consider black-box neural models of CLEVR. These models are trained on a diagnostic dataset benchmarking reasoning. Next, we train an adversarial player that re-configures the scene to fool the CLEVR model. We show that CLEVR models, which otherwise could perform at a human level, can easily be fooled by our agent. Our results put in doubt whether data-driven approaches can do reasoning without exploiting the numerous biases that are often present in those datasets. Finally, we also propose a controlled experiment measuring the efficiency of such models to learn and perform reasoning.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interaction
Adapting Lightweight Vision Language Models for Radiological Visual Question Answering
OVQA: A Clinically Generated Visual Question Answering Dataset
Describe Anything Model for Visual Question Answering on Text-rich Images
DrishtiKon: Multi-Granular Visual Grounding for Text-Rich Document Images
Evaluating Attribute Confusion in Fashion Text-to-Image Generation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.