Towards Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs
원문 초록
Recent advancements in multimodal large language models (MLLMs) have achieved significant multimodal generation capabilities, akin to GPT-4. These models predominantly map visual information into language representation space, leveraging the vast knowledge and powerful text generation abilities of LLMs to produce multimodal instruction-following responses. We could term this method as LLMs for Vision because of its employing LLMs for visual-language understanding, yet observe that these MLLMs neglect the potential of harnessing visual knowledge to enhance overall capabilities of LLMs, which could be regraded as Vision Enhancing LLMs. In this paper, we propose an approach called MKS2, aimed at enhancing LLMs through empowering Multimodal Knowledge Storage and Sharing in LLMs. Specifically, we introduce the Modular Visual Memory, a component integrated into the internal blocks of LLMs, designed to store open-world visual information efficiently. Additionally, we present a soft Mixtures-of-Multimodal Experts architecture in LLMs to invoke multimodal knowledge collaboration during generation. Our comprehensive experiments demonstrate that MKS2 substantially augments the reasoning capabilities of LLMs in contexts necessitating physical or commonsense knowledge. It also delivers competitive results on multimodal benchmarks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
OmniGen2: Exploration to Advanced Multimodal Generation
Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model
OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks
LIFEBench: Evaluating Length Instruction Following in Large Language Models
CASTILLO: Characterizing Response Length Distributions of Large Language Models
Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding
토론
이 논문과 연결된 포럼 글이 아직 없습니다.