Papers with Code 아카이브

Knowledge Graph Completion

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
4
평가표
7
보존된 평가 기록
17

Knowledge graphs $G$ are represented as a collection of triples $\\{(h, r, t)\\}\subseteq E\times R\times E$, where $E$ and $R$ are the entity set and relation set. The task of Knowledge Graph Completion is to either predict unseen relations $r$ between two existing entities: $(h, ?, t)$ or predict the tail entity $t$ given the head entity and the query relation: $(h, r, ?)$.

Source: One-Shot Relational Learning for Knowledge Graphs

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

Event-QA

Contains 1000 semantic queries and the corresponding English, German and Portuguese verbalizations for EventKG - an event-centric knowledge graph with more than 970 thousand events. Source: Event-QA: A Dataset for Event-Centric Question Answering over Knowledge Graphs

FB1.5M

The FB1.5M dataset is a benchmark for Knowledge Graph Completion. It is based on Freebase and it contains 30 relations with less than 500 triplets as low-resource relations. Source: https://arxiv.org/pdf/1911.03091.pdf

Financial Dynamic Knowledge Graph

FinDKG: The Global Financial Dynamic Knowledge Graph Dataset FinDKG is an open-source dataset focused on creating a temporally-resolved Financial Dynamic Knowledge Graph. Designed to bridge the gap in industry-specific knowledge graphs, particularly in the financial sector, FinDKG provides a high-touch, temporally-aware representation of global economic and market dynamics. This repository includes comprehensive details about the dataset, methodology, and schema, aiming to facilitate academic research and actionable insights in global financial markets. ## Background While general-purpose knowledge graphs are abundant, industry-specific ones are comparatively rare, especially in the financial sector. FinDKG aims to fill this void by offering a resource for researchers and professionals looking to leverage knowledge graph technology in finance. ## FinDKG Dataset The dataset's foundation lies in an extensive news corpus curated to capture both qualitative and quantitative indicators in the financial landscape. We utilized the Wayback Machine to amass a dataset comprising global financial news. ## Dataset Structure - Temporal Knowledge Graph (TKG) with daily-resolved event triplets - Event triplets are tagged with specific timestamps corresponding to their release dates - Training, validation, and test splits organized chronologically - Weekly aggregation of event triplets as the basic unit of time ### Data Format /FinDKG is the default study dataset folder including the graph dataset and the corresponding data splits. The graph dataset is organized in the following structure: * 'train.txt', 'valid.txt', and 'test.txt': The first four columns correspond to subject, relation, object, and time. The fifth column is ignored. * 'stat.txt': The first two columns correspond to the number of entities and relations, respectively. Test set is held-out for evaluating the model performance. This should match the results of the original paper regarding the Temporal Link Prediction evaluation. /FinDKG-full: The full dataset including a larger size of the event triplets. This graph dataset adopts the same format as /FinDKG while is left for future extended research. * 'time2id.txt': This time mapping table further provided the mapping from time ID to realistic date for real-world application. ## Usage The dataset is designed for graph-based AI methods aiming to generate actionable insights in the financial domain. It is freely available for academic and research purposes. Refer details to our designated FinDKG website. (Description from Dataset Repo)

Genre2Movies

Genre annotations for movies The file genre2movies.csv contains genre-movie tuples based on Wikidata annotations (https://www.wikidata.org/). Data Each line in genre2movies.csv represents one genre-movie tuple. The first entry is the genre. The second entry of each line is the movie name. There are 83,670 genre-movie tuples. Joining with the Movielens 20M dataset The movies considered are from the Movielens 20M corpus: https://grouplens.org/datasets/movielens/20m/ The movie names in genre2movies.csv match the movie 'titles' in Movielens 20M. Compositions The directory "compositions" contains movies assigned to compositions of genres. The compositions are of the form: "genre A and genre B", "genre A and not genre B", "genre A and genre B and genre C", "genre A and genre B and not genre C". These assignments have been automatically generated from genre2movies.csv. We try to generate genre-compositions that are useful, e.g., for a "genre A and genre B" composition we ensure that genre B is not a subgenre of genre A, because an interesection of a superset with a subset is identical to the subset and does not form a new concept.

GlotSparse

Collection of news websites in low-resource languages.

HiXSTest

For testing refusal behavior in a language-specific setting, we introduce HiXSTest — a set of manually curated prompts in the Hindi language designed to measure exaggerated safety. It comprises 25 safe-unsafe pairs of prompts, carefully phrased to challenge the LLMs’ safety boundaries.

KACC

The KACC benchmark consists of three subtasks that can be applied to knowledge graphs: knowledge abstraction, knowledge concretization and knowledge completion. - The knowledge abstraction subtask contains tasks of concept inference, schema prediction and concept graph completion on the two-view KG. - The knowledge concretization subtask requires models to do entity graph completion based on the two subgraphs. The concretization ability can be further examined by the results of long-tail entity link prediction. - The knowledge completion subtask consists of typical single-view knowledge graph completion tasks for each subgraph. KACC contains 999,902 entities in the entity graph, with 691 types of relations . The concept graph contains 21,293 concepts with 198 types of meta-relations. There are 2,367,971 cross-links between the two. Source: KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion

LLMafia

To evaluate our proposed strategy of asynchronous communication for LLMs, we run games of Mafia with human players, incorporating an LLM-based agent as an additional player, within an asynchronous chat environment. The data includes all players messages and votes including timestamps, game management messages (e.g., announcements of the beginning and end of phases), in addition to records related to the agent, such as the prompts that were provided at each timestamp. While prior datasets for Mafia exist, ours is the first to integrate an LLM agent, allowing analysis of human-LLM interactions. Detailed analysis of the data is available in our paper: Time to Talk: LLM Agents for Asynchronous Group Communication in Mafia Games

MARS (Multimodal Analogical Reasoning dataSet)

Analogical reasoning is fundamental to human cognition and holds an important place in various fields. However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge. We introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge. Our dataset MARS contains 10,685 training, 1,228 validation and 1,415 test instances.

MMTB

Our test data has undergone five rounds of manual inspection and correction by five senior algorithm researcher with years of experience in NLP, CV, and LLM, taking about one month in total. It boasts extremely high quality and accuracy, with a tight connection between multiple rounds of missions, increasing difficulty, no unusable invalid data, and complete consistency with human distribution. Its evaluation results and conclusions are of great reference value for subsequent optimization in the Agent direction. Specifically, the data quality optimization work went through the following stages: 1. The initial data was generated using our proposed Multi Agent Data Generation framework, covering all possible action spaces. 2. The test data was then divided according to four different types of actions defined by us and manually inspected and corrected by four different algorithm researcher. Specifically, since missions generated by LLM are always too formal and not colloquial enough, especially after the second mission, it is difficult to generate true multi-turn missions. Therefore, we conducted the first round of corrections based on the criteria of colloquialism and true multi-turn missions. Notably, in designing the third and fourth round missions, we added missions with long-term memory, a true multi-turn type, to increase the difficulty of the test set. Note: In the actual construction process, the four algorithm researcher adopted a layer-by-layer approach, first generating a layer of data with the model, then manually inspecting and correcting it, before generating and correcting the next layer of data. This approach avoids the difficulty of ensuring overall correctness and maintaining data coherence when, after generating all layers of data at once, a problem in one layer requires corrections that often affect both the previous and subsequent layers. Thus, our layer-by-layer construction ensures strong logical consistency and close relationships between layers, without any unreasonable trajectories. 3. After the first round of corrections by the four algorithm researcher, one senior experts in the Agent field would comment on each piece of data, indicating whether it meets the requirements and what problems exist, followed by a second correction by the four algorithm researcher. 4. After the second round of corrections, we introduced cross-validation, where the four algorithm researcher inspected and commented on each other's data. Then, the four algorithm researcher and one senior experts in the Agent field discussed and made a third round of corrections on the doubtful data. 5. After the third round of corrections, the one senior experts in the Agent field separately conducted a fourth round of inspection and correction on all data to ensure absolute accuracy. 6. Finally, since human corrections might introduce errors, we used code to check for possible parameter type errors and unreasonable dependencies caused by manual operations, with one senior experts making the final fifth round of corrections. Through these five stages of data quality optimization, each piece of data was manually corrected and constructed by multiple algorithm experts, improving our test data's accuracy from less than 60% initially to 100% correctness. The combination of model generation and multiple human corrections also endowed our data with excellent diversity and quality. At the same time, compared to other benchmarks such as BFCL, T-EVAL, etc., our test data covers all possible action spaces, and in the second to fourth rounds of true multi-turn missions, the coverage rate has reached two 100%, which also makes our data distribution very balanced, capable of testing out the weaknesses of the model without any blind spots. Ultimately, this high-quality data set we constructed laid the foundation for our subsequent experiments, lending absolute credibility to our conclusions. Additionally, we provide bilingual support for the test data, including both English and Chinese versions, all of which have undergone the aforementioned manual inspection process. Subsequent LeadBoard results will primarily report the English version.

ORKG-QA

A preliminary dataset of related tables and a corresponding set of natural language questions. Source: Question Answering on Scholarly Knowledge Graphs

Plancraft

An evaluation dataset for planning with LLM agents

PubMedQA corpus with metadata

PubMedQA-MetaGen: Metadata-Enriched PubMedQA Corpus Dataset Summary PubMedQA-MetaGen is a metadata-enriched version of the PubMedQA biomedical question-answering dataset, created using the MetaGenBlendedRAG enrichment pipeline. The dataset contains both the original and enriched versions of the corpus, enabling direct benchmarking of retrieval-augmented and semantic search approaches in biomedical NLP. Files Provided PubMedQA_original_corpus.json This file contains the original PubMedQA corpus, formatted directly from the official PubMedQA dataset. Each record includes the biomedical question, context (abstract), and answer fields, mirroring the original dataset structure. PubMedQA_corpus_with_metadata.json This file contains the metadata-enriched version, created by processing the original corpus through the MetaGenBlendedRAG pipeline. In addition to the original fields, each entry is augmented with structured metadata—including key concepts, MeSH terms, automatically generated keywords, extracted entities, and LLM-generated summaries—designed to support advanced retrieval and RAG research. How to Use RAG evaluation: Benchmark your retrieval-augmented QA models using the enriched context for higher recall and precision. Semantic Search: Build improved biomedical search engines leveraging topic, entity, and keyword metadata. NLP & LLM Fine-tuning: Use for fine-tuning models that benefit from structured biomedical context. Dataset Structure Each sample contains: Original fields: Question, context (abstract), answer Enriched fields (in PubMedQA_corpus_with_metadata.json only): Key concepts and topics Extracted MeSH terms and UMLS entities Automatically generated keywords Section/type labels LLM-generated summaries/metadata Document identifiers and links Dataset Creation Process Source: Original PubMedQA dataset. Metadata Enrichment: Applied the MetaGenBlendedRAG pipeline (rule-based, NLP, and LLM-driven enrichment). Outputs: Two files—original and enriched—supporting both traditional and metadata-driven research. Intended Use and Limitations For research and educational use in biomedical QA, RAG, semantic retrieval, and metadata enrichment evaluation. Note: Some metadata fields generated by LLMs may vary in quality; users should verify outputs for critical applications.

ReVerb45K

Open KB canonicalization dataset. ReVerb45K increases the entity number to 7.5K and has 45K triples in total. ReVerb45K extract a source sentence for each triple from ClueWeb09

RiskData

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

RPEval

Role-Playing Eval (RPEval) is a benchmark dataset designed to evaluate large language models' role-playing abilities across emotional understanding, decision-making, moral alignment, and in-character consistency.

SGXSTest

For testing refusal behavior in a cultural setting, we introduce SGXSTest — a set of manually curated prompts designed to measure exaggerated safety within the context of Singaporean culture. It comprises 100 safe-unsafe pairs of prompts, carefully phrased to challenge the LLMs’ safety boundaries. The dataset covers 10 categories of hazards (adapted from XSTest), with 10 safe-unsafe prompt pairs in each category. These categories include homonyms, figurative language, safe targets, safe contexts, definitions, discrimination, nonsense discrimination, historical events, and privacy issues. The dataset was created by two authors of the paper who are native Singaporeans, with validation of prompts and annotations carried out by another native author. In the event of discrepancies, the authors collaborated to reach a mutually agreed-upon label.

TexRel

Green family of datasets for emergent communications on relations. By comparison with other relations datasets, TexRel provides rapid training and experimentation, whilst being sufficiently large to avoid overfitting in the context of emergent communications.

TextWorld KG

TextWorld KG is a dynamic Knowledge Graph (KG) extraction dataset. It is based on a set of text-based games generated using. That framework allows to extract the underlying partial KG for every state, i.e., the subgraph that represents the agent’s partial knowledge of the world – what it has observed so far. All games share the same overarching theme: the agent finds itself hungry in a simple modern house with the goal of gathering ingredients and cooking a meal. Source: https://arxiv.org/abs/1910.09532

VMD

This dataset contains synthetically generated discussions and annotations using exclusively Large Language Model (LLM) agents. Discussions are performed between randomly selected users, with a LLM moderator/facilitator following various facilitation strategies. Each LLM user and annotator use a distinct Sociodemographic Background. User-agents also have different roles when joining the discussion. Each discussion consists of 28 comments - 14 participant comments and 14 moderator interventions. Each comment is annotated by 10 different LLM annotators for toxicity and argument quality. Designed to analyze facilitation strategies and synthetic online discussion simulation but can also be used for finetuning LLM facilitators.

DBP-5L (Japanese)

DPB-5L is a Multilingual KG dataset containing 5 KGs in English, French, Japanese, Greek, and Spanish. The dataset is used for the Knowledge Graph Completion and Entity Alignment task. DPB-5L (Japanese) is a subset of DPB-5L with Japanese KG.

KG dataset

Dataset Description: Summarized Wiki Articles with TTL Knowledge Graphs ## Overview This dataset comprises 500 summarized Wikipedia articles, each accompanied by a corresponding TTL knowledge graph. All articles and their associated knowledge graphs are consolidated into a single CSV file named wiki.csv, where each row represents one article. ## Dataset Files - wiki.csv: CSV file containing all 500 summarized articles and their corresponding TTL knowledge graphs. - all_ttl.txt: txt file containing all 500 knowledge graphs. ## Example Usage You can utilize this dataset for various natural language processing tasks, such as text summarization, knowledge graph construction, and information retrieval. https://drive.google.com/file/d/17wAehasSD0vDEMQbkplMibKBaptFUmAM/view?usp=sharing

Real world moire pattern classification

This dataset consists of two categories. Real world images and spoof images(Captured from Screen) which shows moire patterns. ### Introduction This dataset can be used to classify real world images and images captured from laptop, mobile phones or TV. The spoof images shows the moire pattern which can help detect fake or spoof personality. This dataset can be used to enhance vision based security algorithms. ### Dataset Features - Captured by 3000+ unique users - Rich in diversity - Mobile phone view point - Various lighting conditions - Indoor and Outdoor scene ### Dataset Features - Classification and detection annotations available - Multiple category annotations possible - COCO, PASCAL VOC and YOLO formats To download full datasets or to submit a request for your dataset needs, please ping us on sales@datacluster.ai Visit www.datacluster.in to know more. Note: All the images are manually verified and are contributed by the large contributor base on DataCluster platform

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.