NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria
원문 초록
Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce NaijaFacts, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, Yor\`ub\'a, Igbo, and Hausa. Answering similar questions for the UK and US yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language). Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49-50% for the US/UK but 20% for Nigeria. The language effect is smaller and tracks web representation, degrading from English to Yor\`ub\'a (9 points on Nigeria, up to 24 on US/UK questions). We find scale does not help: the strongest models reach 66-79% on the UK and the US but none exceed 31% on Nigerian statistics. Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96% on culture and 98% on politics against 26% on statistics, so the gap is not specific to Nigeria.
제출 이력
확인된 버전만 표시합니다. 전체 제출 이력은 arXiv 원문에서 확인하세요.
- arXiv:2611.06504v3제출 시각 미확인
- arXiv:2611.06504v2제출 시각 미확인
- arXiv:2611.06504v1제출 시각 미확인
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
공통 태그로 연결된 공개 논문이 아직 없습니다.
토론
이 논문과 연결된 포럼 글이 아직 없습니다.