PK PyTorchKR Papers
Papers with Code · CC BY-SA 4.0

Multiple Choice Question Answering (MCQA)

Question Answering › Multiple Choice Question Answering (MCQA) 179 평가 기록 31 datasets

A multiple-choice question (MCQ) is composed of two parts: a stem that identifies the question or problem, and a set of alternatives or possible answers that contain a key that is the best answer to the question, and a number of distractors that are plausible but incorrect answers to the question.

In a k-way MCQA task, a model is provided with a question q, a set of candidate options O = {O1, . . . , Ok}, and a supporting context for each option C = {C1, . . . , Ck}. The model needs to predict the correct answer option that is best supported by the given contexts.

개요는 각 항목의 일부입니다. 위 탭에서 전체 기록을 페이지별로 탐색할 수 있습니다.

벤치마크

Papers with Code 아카이브의 데이터셋별 기록 예시입니다. 평가 조건과 지표 방향이 검증되지 않아 순위를 의미하지 않습니다.

Dataset기록된 모델 예시MetricPaperCode
Med-PaLM 2 (ER)
Test Set (Acc-%): 0.723
논문 코드
Bloomberg GPT (few-shot, k=3)
Accuracy: 92
논문 코드
PaLM 2 (few-shot, k=3, CoT)
Accuracy: 94.4
논문 코드
PaLM 2 (few-shot, k=3, CoT)
Accuracy: 91.2
논문 코드
PaLM 2 (few-shot, k=3, Direct)
Accuracy: 90
논문 코드
Med-PaLM 2 (ER)
Accuracy: 95.8
논문 코드
Med-PaLM 2 (ER)
Accuracy: 92
논문 코드
Med-PaLM 2 (5-shot)
Accuracy: 95.2
논문 코드
GAL 30B (zero-shot)
Accuracy: 33.3
논문 코드
Chinchilla (few-shot, k=5)
Accuracy: 73.0
논문 코드
Chinchilla (few-shot, k=5)
Accuracy: 51
논문 코드
GAL 120B (zero-shot)
Accuracy: 43
논문 코드

데이터셋

31개
MedMCQABIG-bench (Hyperbaton)BIG-bench (Movie Recommendation)BIG-bench (Navigate)BIG-bench (Ruin Names)MMLU (College Biology)MMLU (Medical Genetics)MMLU (Professional medicine)MMLU (Abstract Algebra)MMLU (Astronomy)MMLU (College Chemistry)MMLU (College Mathematics)

논문

라이브러리

Papers with Code 리더보드 row에 연결된 구현 저장소입니다.

github.com/paperswithcode/galai 1 논문
github.com/m42-health/med42 1 논문
github.com/chrisociepa/allamo 1 논문
github.com/conceptofmind/PaLM-flax 1 논문
github.com/foundation-model-stack/fms-fsdp 1 논문
github.com/google/paxml 1 논문
github.com/lucidrains/CoCa-pytorch 1 논문
github.com/lucidrains/PaLM-jax 1 논문