Task-Oriented Dialogue Systems

The Multi-domain Wizard-of-Oz (MultiWOZ) dataset is a large-scale human-human conversational corpus spanning over seven domains, containing 8438 multi-turn dialogues, with each dialogue averaging 14 turns. Different from existing standard datasets like WOZ and DSTC2, which contain less than 10 slots and only a few hundred values, MultiWOZ has 30 (domain, slot) pairs and over 4,500 possible values. The dialogues span seven domains: restaurant, hotel, attraction, taxi, train, hospital and police. Source: Contents Image Source: Zhang et al

The Schema-Guided Dialogue (SGD) dataset consists of over 20k annotated multi-domain, task-oriented conversations between a human and a virtual assistant. These conversations involve interactions with services and APIs spanning 20 domains, ranging from banks and events to media, calendar, travel, and weather. For most of these domains, the dataset contains multiple different APIs, many of which have overlapping functionalities but different interfaces, which reflects common real-world scenarios. The wide range of available annotations can be used for intent prediction, slot filling, dialogue state tracking, policy imitation learning, language generation, user simulation learning, among other tasks in large-scale virtual assistants. Besides these, the dataset has unseen domains and services in the evaluation set to quantify the performance in zero-shot or few shot settings. Source: The Schema-Guided Dialogue Dataset

MuTual is a retrieval-based dataset for multi-turn dialogue reasoning, which is modified from Chinese high school English listening comprehension test data. It tests dialogue reasoning via next utterance prediction. Source: https://github.com/Nealcly/MuTual Image Source: https://github.com/Nealcly/MuTual

CrossWOZ is the first large-scale Chinese Cross-Domain Wizard-of-Oz task-oriented dataset. It contains 6K dialogue sessions and 102K utterances for 5 domains, including hotel, restaurant, attraction, metro, and taxi. Moreover, the corpus contains rich annotation of dialogue states and dialogue acts at both user and system sides. Source: CrossWOZ

Task Oriented Parsing v2 (TOPv2) representations for intent-slot based dialog systems. Provided under the CC-BY-SA license. Please cite the accompanying paper when using this dataset - @inproceedings{chen-etal-2020-low-resource, title={Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing}, author={Xilun Chen and Asish Ghoshal and Yashar Mehdad and Luke Zettlemoyer and Sonal Gupta}, booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, year={2020}, publisher = "Association for Computational Linguistics" } CHANGELOG:<br> 03/10/2021 (V1.1): Added the low-resource splits used in the paper.<br> 09/18/2020 (V1.0): Initial release. TOPv2 is a multi-domain task-oriented semantic parsing dataset. It is an extension to the TOP dataset (http://fb.me/semanticparsingdialog) with 6 additional domains and 137k new samples. In total, TOPv2 has 8 domains (alarm, event, messaging, music, navigation, reminder, timer, weather) and 180k samples randomly split into train, eval, and test sets for each domain. Please refer to the paper for more data statistics. Note: As TOPv2 data is provided on a per-domain basis, the UNSUPPORTED utterances in the original TOP dataset were removed as they could not be mapped to any domain. The training, evaluation and test sets for each domain are provided as tab-separated value (TSV) files with file names of "domain_split.tsv". The first row of each file contains the column headers, while each following row is of the format: domain <tab> utterance <tab> semantic_parse where the semantic_parse follows the same format as the original TOP dataset. e.g. event <tab> Art fairs this weekend in Detroit <tab> [IN:GET_EVENT [SL:CATEGORY_EVENT Art fairs ] [SL:DATE_TIME this weekend ] in [SL:LOCATION Detroit ] ] The low-resource splits used in our experiments are provided in the low_resource_splits subdirectory, including training and validation sets from the reminder and weather domains under 10, 25, 50, 100, 250, 500 and 1000 SPIS.

Action-Based Conversations Dataset (ABCD) is a goal-oriented dialogue fully-labeled dataset with over 10K human-to-human dialogues containing 55 distinct user intents requiring unique sequences of actions constrained by policies to achieve task success. The dataset is proposed to study customer service dialogue systems in more realistic settings.

BiToD is a bilingual multi-domain dataset for end-to-end task-oriented dialogue modeling. BiToD contains over 7k multi-domain dialogues (144k utterances) with a large and realistic bilingual knowledge base. It serves as an effective benchmark for evaluating bilingual ToD systems and cross-lingual transfer learning approaches.

A Game Of Sorts is a collaborative image ranking task. Players are asked to rank a set of images based on a given sorting criterion. The game provides a framework for the evaluation of visually grounded language understanding and generation of referring expressions in multimodal dialogue settings.

The ODSQA dataset is a spoken dataset for question answering in Chinese. It contains more than three thousand questions from 20 different speakers. Source: https://github.com/chiahsuan156/ODSQA

A novel dataset of document-grounded task-based dialogues, where an Information Giver (IG) provides instructions (by consulting a document) to an Information Follower (IF), so that the latter can successfully complete the task. In this unique setting, the IF can ask clarification questions which may not be grounded in the underlying document and require commonsense knowledge to be answered.

SSD (Sub-slot Dialog) dataset: This is the dataset for the ACL 2022 paper "A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots". arxiv

SSD (Sub-slot Dialog) dataset: This is the dataset for the ACL 2022 paper "A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots".

The Arabic-TOD dataset is based on the BiToD dataset. Of the 3,689 BiToD-English dialogues, 1,500 dialogues (30,000 utterances) were translated into Arabic. We translated the task-related keywords such as cuisine, dietary restrictions, and price-level for the restaurant domain, price-level for the hotel domain, type, and price-level for the attraction domain, day, weather, and city for the weather domain. We keep the rest of values without translation, like hotels’ and restaurants’ names, locations, and addresses. These values are real entities in Hong Kong city (literals), and most of them contain Chinese words written in English, therefore they have not been translated. According to the slot-values in the Arabic-TOD dataset, we used the slots names as they are in English and translated their corresponding values, except the entities in Hong Kong city since the Arabic-TOD dataset supports codeswitching. We did not translate the 'UserTask' for all dialogues, since it is not important in developing the system. It is just as a summarization of the dialogue contents.

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset An agent benchmark for adaptive decision-making in dialog measuring agent accuracy and dialog turn efficiency in helping users determine eligibility for public, real-world opportunities. This dataset containss the following: * Natural language descriptions of 82 NYC public benefits programs (e.g., tax credits, childcare, subsidized AC) Two simulated user datasets containing all features relevant to that household's eligibility: * Representative: 25 households with features drawn from real NYC demographic data * Diverse: 56 households which, collectively, satisfy nearly every possible acceptance/rejection criteria in all 82 public benefits opportunities

Dataset for our paper Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky which includes 5000 enterprise tools and the corresponding dialogues generated using DiaFORGE UTC data engine. The dataset is generated with the data generation engine. The engine simulates a user agent and an assistant agent in a dialogue, where the user agent has a persona and the assistant agent has access to a set of tools. Detailed information about the data generation process can be found in the paper. Each entry in the dataset contains: - seed: Gold tool for the generated dialogue. - user_persona: Persona of the simulated user agent. - messages: List of messages in the dialogue, where each message is a dictionary containing: - distractor_tools: List of distractor tools that are not the gold tool but are relevant to the dialogue. These tools are used by user agent to generate utterances that are hard to disambiguate from the gold tool. - retrieved_tools: List of tools retrieved by the assistant agent and used by the assistant agent in order to ask clarifying questions to the user agent.

HR-Multiwoz is a fully-labeled dataset of 550 conversations spanning 10 HR domains to evaluate LLM Agent. It is the first labeled open-sourced conversation dataset in the HR domain for NLP research. Please refer to HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agent for details about the dataset construction. Dataset Sources Repository: xwjzds/extractive_qa_question_answering_hr Paper: HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agent Leaderboard: github repo

Simulates unanticipated user needs in the deployment stage. Source: Incremental Learning from Scratch for Task-Oriented Dialogue Systems

To evaluate our proposed strategy of asynchronous communication for LLMs, we run games of Mafia with human players, incorporating an LLM-based agent as an additional player, within an asynchronous chat environment. The data includes all players messages and votes including timestamps, game management messages (e.g., announcements of the beginning and end of phases), in addition to records related to the agent, such as the prompts that were provided at each timestamp. While prior datasets for Mafia exist, ours is the first to integrate an LLM agent, allowing analysis of human-LLM interactions. Detailed analysis of the data is available in our paper: Time to Talk: LLM Agents for Asynchronous Group Communication in Mafia Games

MultiWOZ-coref, (or MultiWOZ 2.3) is an extension of the MultiWOZ dataset that adds co-reference annotations in addition to corrections of dialogue acts and dialogue states. Source: https://github.com/lexmen318/MultiWOZ_2.3

SSD (Sub-slot Dialog) dataset: This is the dataset for the ACL 2022 paper "A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots".

SSD (Sub-slot Dialog) dataset: This is the dataset for the ACL 2022 paper "A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots".

SSD (Sub-slot Dialog) dataset: This is the dataset for the ACL 2022 paper "A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots".