Papers with Code 아카이브

Text Generation

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
25
평가표
155
보존된 평가 기록
54

Text Generation is the task of generating text with the goal of appearing indistinguishable to human-written text. This task is more formally known as "natural language generation" in the literature.

Text generation can be addressed with Markov processes or deep generative models like LSTMs. Recently, some of the most advanced methods for text generation include BART, GPT and other GAN-based approaches. Text generation systems are evaluated either through human ratings or automatic evaluation metrics like METEOR, ROUGE, and BLEU.

Further readings:

<span style="color:grey; opacity: 0.6">( Image credit: Adversarial Ranking for Language Generation )</span>

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

DTGB

We introduce Dynamic Text-attributed Graph Benchmark (DTGB), a collection of large-scale, time-evolving graphs from diverse domains, with nodes and edges enriched by dynamically changing text attributes and categories. To facilitate the use of DTGB, we design standardized evaluation procedures based on four real-world use cases: future link prediction, destination node retrieval, edge classification, and textual relation generation. These tasks require models to understand both dynamic graph structures and natural language, highlighting the unique challenges posed by DyTAGs.

Elsevier OA CC-BY

An open corpus of Scientific Research papers which has a representative sample from across scientific disciplines. This corpus not only includes the full text of the article, but also the metadata of the documents, along with the bibliographic information for each reference. Source: Elsevier OA CC-By Corpus

Kaleidoscope

The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks. Summary: 18 languages (Bengali → Spanish), 14 subjects (STEM to Humanities), 20,911 questions (55% requiring image understanding

Live Comment Dataset

The Live Comment Dataset is a large-scale dataset with 2,361 videos and 895,929 live comments that were written while the videos were streamed. Source: LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts

Negotiation Dialogues Dataset

This dataset consists of 5808 dialogues, based on 2236 unique scenarios. Each dialogue is converted into two training examples in the dataset, showing the complete conversation from the perspective of each agent. The perspectives differ on their input goals, output choice, and in special tokens marking whether a statement was read or written. Source: Negotiation Dialogues Dataset

OpenAsp

OpenAsp Dataset OpenAsp is an Open Aspect-based Multi-Document Summarization dataset derived from DUC and MultiNews summarization datasets. Antarctica ## Dataset Access To generate OpenAsp, you require access to the DUC dataset which OpenAsp is derived from. ### Steps: 1. Grant access to DUC dataset by following NIST instructions here. * you should receive two user-password pairs (for DUC01-02 and DUC06-07) * you should receive a file named fwdrequestingducdata.zip 2. Clone this repository by running the following command: git clone https://github.com/liatschiff/OpenAsp.git 3. Optionally create a conda or virtualenv environment: bash conda create -n openasp 'python>3.10,<3.11' conda activate openasp 4. Install python requirements, currently requires python3.8-3.10 (later python versions have issues with spacy) bash pip install -r requirements.txt 5. copy fwdrequestingducdata.zip into the OpenAsp repo directory 6. run the prepare script command: bash python prepare_openasp_dataset.py --nist-duc2001-user '<2001-user>' --nist-duc2001-password '<2001-pwd>' --nist-duc2006-user '<2006-user>' --nist-duc2006-password '<2006-pwd>' 7. load the dataset using huggingface datasets python from glob import glob import os import gzip import shutil from datasets import load_dataset openasp_files = os.path.join('openasp-v1', '*.jsonl.gz') data_files = { os.path.basename(fname).split('.')[0]: fname for fname in glob(openasp_files) } for ftype, fname in data_files.copy().items(): with gzip.open(fname, 'rb') as gz_file: with open(fname[:-3], 'wb') as output_file: shutil.copyfileobj(gz_file, output_file) data_files[ftype] = fname[:-3] # load OpenAsp as huggingface's dataset openasp = load_dataset('json', data_files=data_files) # print first sample from every split for split in ['train', 'valid', 'test']: sample = openasp[split][0] # print title, aspect_label, summary and documents for the sample title = sample['title'] aspect_label = sample['aspect_label'] summary = '\n'.join(sample['summary_text']) input_docs_text = ['\n'.join(d['text']) for d in sample['documents']] print('* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *') print(f'Sample from {split}\nSplit title={title}\nAspect label={aspect_label}') print(f'\naspect-based summary:\n {summary}') print('\ninput documents:\n') for i, doc_txt in enumerate(input_docs_text): print(f'---- doc #{i} ----') print(doc_txt[:256] + '...') print('* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\n\n\n') ### Troubleshooting 1. Dataset failed loading with load_dataset() - you may want to delete huggingface datasets cache folder 2. 401 Client Error: Unauthorized - you're DUC credentials are incorrect, please verify them (case sensitive, no extra spaces etc) 3. Dataset created but prints a warning about content verification - you may be using different version of NLTK or spacy model which affects the sentence tokenization process. You must use exact versions as pinned on requirements.txt. 4. IndexError: list index out of range - similar to (3), try to reinstall the requirements with exact package versions. ### Under The Hood The prepare_openasp_dataset.py script downloads DUC and Multi-News source files, uses sacrerouge package to prepare the datasets and uses the openasp_v1_dataset_metadata.json file to extract the relevant aspect summaries and compile the final OpenAsp dataset. ## License This repository, including the openasp_v1_dataset_metadata.json and prepare_openasp_dataset.py, are released under APACHE license. OpenAsp dataset summary and source document for each sample, which are generated by running the script, are licensed under the respective generic summarization dataset - Multi-News license and DUC license.

OpenViDial 2.0

OpenViDial 2.0 is a larger-scale open-domain multi-modal dialogue dataset compared to the previous version OpenViDial 1.0. OpenViDial 2.0 contains a total number of 5.6 million dialogue turns extracted from either movies or TV series from different resources, and each dialogue turn is paired with its corresponding visual context. Image source: https://github.com/ShannonAI/OpenViDial

QTuna

The QTUNA dataset is the result of a series of elicitation experiments in which human speakers were asked to perform a linguistic task that invites the use of quantified expressions in order to inform possible Natural Language Generation algorithms that mimic humans' use of quantified expressions. Source: https://github.com/a-quei/qtuna

Reddit Ideological and Extreme Bias Dataset

Articles originating from subreddits with explicitly stated ideologies are categorized into three groups: 72,488 articles in the Liberal class, 79,573 articles in the Conservative class, and 225,083 articles in the Restricted class. Conversely, articles from subreddits lacking a clearly defined ideology, including those with implicit or explicit ideologies, are merged to form a holdout dataset comprising 922,522 articles. Part 1 https://data.mendeley.com/datasets/2tdr9sjd83/3 Part 2 https://data.mendeley.com/datasets/dxpp5983yb/1 Part 3 https://data.mendeley.com/datasets/f7knr8r94w/1

SciGen

SciGen is a challenge dataset for the task of reasoning-aware data-to-text generation consisting of tables from scientific articles and their corresponding descriptions. The unique properties of SciGen are that (1) tables mostly contain numerical values, and (2) the corresponding descriptions require arithmetic reasoning. SciGen is therefore the first dataset that assesses the arithmetic reasoning capabilities of generation models on complex input structures, i.e., tables from scientific articles. SciGen opens new avenues for future research in reasoning-aware text generation and evaluation. The dataset consists of 1.3K pairs of tables with their descriptions, with an average of 53 cells in each table.

SportSett

This resource is designed to allow for research into Natural Language Generation. In particular, with neural data-to-text approaches although it is not limited to these.

StyleKQC

StyleKQC is a style-variant paraphrase corpus for korean questions and commands. It was built with a corpus construction scheme that simultaneously considers the core content and style of directives, namely intent and formality, for the Korean language. Utilizing manually generated natural language queries on six daily topics, the corpus was expanded to formal and informal sentences by human rewriting and transferring.

ThreatGram 101 - Extreme Telegram Data

Data 1: Raw and Unlabeled; 2 million unlabeled replies from 17 Telegram channels. Data 2: Raw and Labeled; 15,076 replies from 17 Telegram channels categorized as no threat, judicial threat, and non-judicial threat.

VIST-Edit

The dataset, VIST-Edit, includes 14,905 human-edited versions of 2,981 machine-generated visual stories. The stories were generated by two state-of-the-art visual storytelling models, each aligned to 5 human-edited versions. Source: Visual Story Post-Editing

XAlign

It consists of an extensive collection of a high quality cross-lingual fact-to-text dataset in 11 languages: Assamese (as), Bengali (bn), Gujarati (gu), Hindi (hi), Kannada (kn), Malayalam (ml), Marathi (mr), Oriya (or), Punjabi (pa), Tamil (ta), Telugu (te), and monolingual dataset in English (en). This is the Wikipedia text <--> Wikidata KG aligned corpus used to train the data-to-text generation model. The Train & validation splits are created using distant supervision methods and Test data is generated through human annotations. ## Data Format Dataset is publicly available here. Each directory contains language specific dataset (refered through language ISO code) and contains of three files: - train.jsonl - test.jsonl - val.jsonl Data stored in the above files are of JSON Line (jsonl) format. ### Record structure (JSON structure) Each record consist of the following entries: - sentence (string) : Native language wikipedia sentence. (non-native language strings were removed.) - facts (List[Dict]) : List of facts associated with the sentence where each fact is stored as dictionary. - language (string) : Language identifier. The facts key contains list of facts where each facts is stored as dictionary. A single record within fact list contains following entries: - subject (string) : central entity. - object (string) : entity or a piece of information about the subject. - predicate (string) : relationship that connects the subject and the object. - qualifiers (List[Dict]) : It provide additional information about the fact, is stored as list of qualifier where each record is a dictionary. The dictionary contains two keys: qualifier_predicate to represent property of qualifer and qualifier_object to store value for the qualifier's predicate. ### Examples Example from English dataset { "sentence": "Mark Paul Briers (born 21 April 1968) is a former English cricketer.", "facts": [ { "subject": "Mark Briers", "predicate": "date of birth", "object": "21 April 1968", "qualifiers": [] }, { "subject": "Mark Briers", "predicate": "occupation", "object": "cricketer", "qualifiers": [] }, { "subject": "Mark Briers", "predicate": "country of citizenship", "object": "United Kingdom", "qualifiers": [] } ], "language": "en" } Example from one of the low-resource languages (i.e. Hindi) { "sentence": "बोरिस पास्तेरनाक १९५८ में साहित्य के क्षेत्र में नोबेल पुरस्कार विजेता रहे हैं।", "facts": [ { "subject": "Boris Pasternak", "predicate": "nominated for", "object": "Nobel Prize in Literature", "qualifiers": [ { "qualifier_predicate": "point in time", "qualifier_subject": "1958" } ] } ], "language": "hi" }

YTSeg

We present YTSeg, a topically and structurally diverse benchmark for the text segmentation task based on YouTube transcriptions. The dataset comprises 19,299 videos from 393 channels, amounting to 6,533 content hours. The topics are wide-ranging, covering domains such as science, lifestyle, politics, health, economy, and technology. The videos are from various types of content formats, such as podcasts, lectures, news, corporate events & promotional content, and, more broadly, videos from individual content creators. We refer to the paper for further information.

Alpaca Data Galician

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

AMR3.0

Abstract Meaning Representation (AMR) Annotation Release 3.0 was developed by the Linguistic Data Consortium (LDC), SDL/Language Weaver, Inc., the University of Colorado's Computational Language and Educational Research group and the Information Sciences Institute at the University of Southern California. It contains a sembank (semantic treebank) of over 59,255 English natural language sentences from broadcast conversations, newswire, weblogs, web discussion forums, fiction and web text.

AP

This is a paraphrasing dataset created using the adversarial paradigm. A task was designed called the Adversarial Paraphrasing Task (APT) whose objective was to write sentences that mean the same as a given sentence but have as different syntactical and lexical properties as possible. As shown in the paper, this dataset can be used to measure the performance of paraphrase identifier models and train them. This dataset and the task associated with it (APT) can also be used to challenge neural networks to generate better adversarial paraphrases (the work has done this for T5-base), which will in turn help create better paraphrase identifiers.

AppealCase

The AppealCase dataset is the first large-scale resource specifically designed to support LegalAI research in appellate judgment scenarios. While prior work in LegalAI has focused heavily on one-shot trials, the appellate procedure—critical to ensuring fairness and correcting judicial errors—remains largely underexplored. To bridge this gap, we construct 10,000 pairs of matched first-instance and second-instance civil cases from China Judgments Online, covering 91 causes of action. We also design a structured annotation schema and define new tasks tailored to the appellate context.

Arabic-ToD

The Arabic-TOD dataset is based on the BiToD dataset. Of the 3,689 BiToD-English dialogues, 1,500 dialogues (30,000 utterances) were translated into Arabic. We translated the task-related keywords such as cuisine, dietary restrictions, and price-level for the restaurant domain, price-level for the hotel domain, type, and price-level for the attraction domain, day, weather, and city for the weather domain. We keep the rest of values without translation, like hotels’ and restaurants’ names, locations, and addresses. These values are real entities in Hong Kong city (literals), and most of them contain Chinese words written in English, therefore they have not been translated. According to the slot-values in the Arabic-TOD dataset, we used the slots names as they are in English and translated their corresponding values, except the entities in Hong Kong city since the Arabic-TOD dataset supports codeswitching. We did not translate the 'UserTask' for all dialogues, since it is not important in developing the system. It is just as a summarization of the dialogue contents.

AskParents

AskParents is a dataset for advice classification extracted from Reddit. In this dataset, posts are annotated for whether they contain advice or not. It contains 8,701 samples for training, 802 for validation and 1,091 for testing. Source: https://github.com/venkatasg/Advice-EMNLP2020

Avicenna: Deductive Commonsense Reasoning

A syllogism is a common form of deductive reasoning that requires precisely two premises and one conclusion. The Avicenna corpus is a benchmark for syllogistic NLI and syllogistic NLG: - syllogistic NLI: Identifying the possibility of inferring between pairs of inputted sentences. - syllogistic NLG: Generating a conclusion sentence for two sentences with a syllogistic relation.

BrWac2Wiki

This is a dataset for multi-document summarization in Portuguese, what means that it has examples of multiple documents (input) related to human-written summaries (output). In particular, it has entries of multiple related texts from Brazilian websites about a subject, and the summary is the Portuguese Wikipedia lead section on the same subject (lead: the first section, i.e., summary, of any Wipedia article). Input texts were extracted from BrWac corpus, and the output from Brazilian Wikipedia dumps page. BrWac2Wiki contains 114.652 examples of (documents, wikipedia) pairs! So it is suitable for training and validating AI models for multi-document summarization in Portuguese. More information on the paper "PLSUM: Generating PT-BR Wikipedia by Summarizing Websites", by André Seidel Oliveira¹ and Anna Helena Reali Costa¹, that is going to be presented at ENIAC 2021. Our work is inspired by WikiSum, a similar dataset for the English language.

COLLIE-v1

COLLIE-v1 is a dataset with 2080 instances comprising 13 constraint structures designed for text generation under constraints. It is a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage).

Colors

A large dataset of color names and their respective RGB values stores in CSV.

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.