Probing Language Models

BioLAMA is a benchmark comprised of 49K biomedical factual knowledge triples for probing biomedical Language Models. It is used to assess the capabilities of Language Models for being valid biomedical knowledge bases.

KAMEL comprises knowledge about 234 relations from Wikidata with a large training, validation, and test dataset. We make sure that all facts are also present in Wikipedia so that they have been seen during the pre-training procedure of the LMs we are probing. Most importantly we overcome the limitations of existing probing datasets by (1) having a larger variety of knowledge graph relations, (2) it contains single- and multi-token entities, (3) we use relations with literals, and (4) have alternative labels for entities. (5) Furthermore, we created an evaluation procedure for higher cardinality relations, which was missing in previous works, and (6) make sure that the dataset can be used for causal LMs.

The $\text{BEAR}$ dataset and its larger version, $\text{BEAR}_{\text{big}}$, are benchmarks for evaluating common factual knowledge contained in language models. This dataset was created as part of the paper "BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models". For more information visit the LM Pub Quiz website. ## Citation When using the dataset or library, please cite the following paper: @misc{wilandBEARUnifiedFramework2024, title = {{{BEAR}}: {{A Unified Framework}} for {{Evaluating Relational Knowledge}} in {{Causal}} and {{Masked Language Models}}}, shorttitle = {{{BEAR}}}, author = {Wiland, Jacek and Ploner, Max and Akbik, Alan}, year = {2024}, number = {arXiv:2404.04113}, eprint = {2404.04113}, publisher = {arXiv}, url = {http://arxiv.org/abs/2404.04113}, }