A Survey on Evaluation of Multimodal Large Language Models
원문 초록
Multimodal Large Language Models (MLLMs) mimic human perception and reasoning system by integrating powerful Large Language Models (LLMs) with various modality encoders (e.g., vision, audio), positioning LLMs as the "brain" and various modality encoders as sensory organs. This framework endows MLLMs with human-like capabilities, and suggests a potential pathway towards achieving artificial general intelligence (AGI). With the emergence of all-round MLLMs like GPT-4V and Gemini, a multitude of evaluation methods have been developed to assess their capabilities across different dimensions. This paper presents a systematic and comprehensive review of MLLM evaluation methods, covering the following key aspects: (1) the background of MLLMs and their evaluation; (2) "what to evaluate" that reviews and categorizes existing MLLM evaluation tasks based on the capabilities assessed, including general multimodal recognition, perception, reasoning and trustworthiness, and domain-specific applications such as socioeconomic, natural sciences and engineering, medical usage, AI agent, remote sensing, video and audio processing, 3D point cloud analysis, and others; (3) "where to evaluate" that summarizes MLLM evaluation benchmarks into general and specific benchmarks; (4) "how to evaluate" that reviews and illustrates MLLM evaluation steps and metrics; Our overarching goal is to provide valuable insights for researchers in the field of MLLM evaluation, thereby facilitating the development of more capable and reliable MLLMs. We emphasize that evaluation should be regarded as a critical discipline, essential for advancing the field of MLLMs.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
야생에서의 보안 및 프라이버시 프롬프트: 사용자가 LLM에게 묻는 내용과 LLM의 응답 방식
Security and Privacy Prompts in the Wild: What Users Ask LLMs and How LLMs Respond
AI 샌드박스: 위협 모델, 분류 체계 및 측정 프레임워크
AI Sandboxes: A Threat Model, Taxonomy, and Measurement Framework
AGI에서 ASI로
From AGI to ASI
영향력 있는 다원주의적 정렬 연구를 위한 로드맵
A Roadmap to Impactful Pluralistic Alignment Research
Towards Trustworthy Physical Intelligence: From Theory to Practice Across Life Cycle
하나의 해답을 향한 세 가지 경로: 응용 연구자를 위한 AIPW, TMLE 및 이중 머신러닝의 일치화
Three Routes to One Answer: Reconciling AIPW, TMLE, and Double Machine Learning for Applied Researchers
토론
이 논문과 연결된 포럼 글이 아직 없습니다.