SciDFM: A Large Language Model with Mixture-of-Experts for Science
원문 초록
Recently, there has been a significant upsurge of interest in leveraging large language models (LLMs) to assist scientific discovery. However, most LLMs only focus on general science, while they lack domain-specific knowledge, such as chemical molecules and amino acid sequences. To bridge these gaps, we introduce SciDFM, a mixture-of-experts LLM, which is trained from scratch and is able to conduct college-level scientific reasoning and understand molecules and amino acid sequences. We collect a large-scale training corpus containing numerous scientific papers and books from different disciplines as well as data from domain-specific databases. We further fine-tune the pre-trained model on lots of instruction data to improve performances on downstream benchmarks. From experiment results, we show that SciDFM achieves strong performance on general scientific benchmarks such as SciEval and SciQ, and it reaches a SOTA performance on domain-specific benchmarks among models of similar size. We further analyze the expert layers and show that the results of expert selection vary with data from different disciplines. To benefit the broader research community, we open-source SciDFM at https://huggingface.co/OpenDFM/SciDFM-MoE-A5.6B-v1.0.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery
LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
Mixture of Experts in Large Language Models
Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks
Assay2Mol: large language model-based drug design using BioAssay context
Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.