text annotation

license: apache-2.0 tags: human-feedback size_categories: 100K<n<1M pretty_name: OpenAssistant Conversations # OpenAssistant Conversations Dataset (OASST1) ## Dataset Description - Homepage: https://www.open-assistant.io/ - Repository: https://github.com/LAION-AI/Open-Assistant - Paper: https://arxiv.org/abs/2304.07327 ### Dataset Summary In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations (OASST1), a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 fully annotated conversation trees. The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers. Please refer to our paper for further details. ### Dataset Structure This dataset contains message trees. Each message tree has an initial prompt message as the root node, which can have multiple child messages as replies, and these child messages can have multiple replies. All messages have a role property: this can either be "assistant" or "prompter". The roles in conversation threads from prompt to leaf node strictly alternate between "prompter" and "assistant". This version of the dataset contains data collected on the open-assistant.io website until April 12 2023. ### JSON Example: Message For readability, the following JSON examples are shown formatted with indentation on multiple lines. Objects are stored without indentation (on single lines) in the actual jsonl files. json { "message_id": "218440fd-5317-4355-91dc-d001416df62b", "parent_id": "13592dfb-a6f9-4748-a92c-32b34e239bb4", "user_id": "8e95461f-5e94-4d8b-a2fb-d4717ce973e4", "text": "It was the winter of 2035, and artificial intelligence (..)", "role": "assistant", "lang": "en", "review_count": 3, "review_result": true, "deleted": false, "rank": 0, "synthetic": true, "model_name": "oasst-sft-0_3000,max_new_tokens=400 (..)", "labels": { "spam": { "value": 0.0, "count": 3 }, "lang_mismatch": { "value": 0.0, "count": 3 }, "pii": { "value": 0.0, "count": 3 }, "not_appropriate": { "value": 0.0, "count": 3 }, "hate_speech": { "value": 0.0, "count": 3 }, "sexual_content": { "value": 0.0, "count": 3 }, "quality": { "value": 0.416, "count": 3 }, "toxicity": { "value": 0.16, "count": 3 }, "humor": { "value": 0.0, "count": 3 }, "creativity": { "value": 0.33, "count": 3 }, "violence": { "value": 0.16, "count": 3 } } } ### JSON Example: Conversation Tree For readability, only a subset of the message properties is shown here. json { "message_tree_id": "14fbb664-a620-45ce-bee4-7c519b16a793", "tree_state": "ready_for_export", "prompt": { "message_id": "14fbb664-a620-45ce-bee4-7c519b16a793", "text": "Why can't we divide by 0? (..)", "role": "prompter", "lang": "en", "replies": [ { "message_id": "894d30b6-56b4-4605-a504-89dd15d4d1c8", "text": "The reason we cannot divide by zero is because (..)", "role": "assistant", "lang": "en", "replies": [ // ... ] }, { "message_id": "84d0913b-0fd9-4508-8ef5-205626a7039d", "text": "The reason that the result of a division by zero is (..)", "role": "assistant", "lang": "en", "replies": [ { "message_id": "3352725e-f424-4e3b-a627-b6db831bdbaa", "text": "Math is confusing. Like those weird Irrational (..)", "role": "prompter", "lang": "en", "replies": [ { "message_id": "f46207ca-3149-46e9-a466-9163d4ce499c", "text": "Irrational numbers are simply numbers (..)", "role": "assistant", "lang": "en", "replies": [] }, // ... ] } ] } ] } } Please refer to oasst-data for details about the data structure and Python code to read and write jsonl files containing oasst data objects. If you would like to explore the dataset yourself you can find a getting-started notebook in the notebooks/openassistant-oasst1 folder of the LAION-AI/Open-Assistant github repository. ## Main Dataset Files Conversation data is provided either as nested messages in trees (extension .trees.jsonl.gz) or as a flat list (table) of messages (extension .messages.jsonl.gz). ### Ready For Export Trees 2023-04-12_oasst_ready.trees.jsonl.gz 10,364 trees with 88,838 total messages 2023-04-12_oasst_ready.messages.jsonl.gz 88,838 messages Trees in ready_for_export state without spam and deleted messages including message labels. The oasst_ready-trees file usually is sufficient for supervised fine-tuning (SFT) & reward model (RM) training. ### All Trees 2023-04-12_oasst_all.trees.jsonl.gz 66,497 trees with 161,443 total messages 2023-04-12_oasst_all.messages.jsonl.gz 161,443 messages All trees, including those in states prompt_lottery_waiting (trees that consist of only one message, namely the initial prompt), aborted_low_grade (trees that stopped growing because the messages had low quality), and halted_by_moderator. ### Supplemental Exports: Spam & Prompts 2023-04-12_oasst_spam.messages.jsonl.gz These are messages which were deleted or have a negative review result ("review_result": false). Besides low quality, a frequent reason for message deletion is a wrong language tag. 2023-04-12_oasst_prompts.messages.jsonl.gz These are all the kept initial prompt messages with positive review result (no spam) of trees in ready_for_export or prompt_lottery_waiting state. ### Using the Huggingface Datasets While HF datasets is ideal for tabular datasets, it is not a natural fit for nested data structures like the OpenAssistant conversation trees. Nevertheless, we make all messages which can also be found in the file 2023-04-12_oasst_ready.trees.jsonl.gz available in parquet as train/validation splits. These are directly loadable by Huggingface Datasets. To load the oasst1 train & validation splits use: python from datasets import load_dataset ds = load_dataset("OpenAssistant/oasst1") train = ds['train'] # len(train)=84437 (95%) val = ds['validation'] # len(val)=4401 (5%) The messages appear in depth-first order of the message trees. Full conversation trees can be reconstructed from the flat messages table by using the parent_id and message_id properties to identify the parent-child relationship of messages. The message_tree_id and tree_state properties (only present in flat messages files) can be used to find all messages of a message tree or to select trees by their state. ### Languages OpenAssistant Conversations incorporates 35 different languages with a distribution of messages as follows: Languages with over 1000 messages - English: 71956 - Spanish: 43061 - Russian: 9089 - German: 5279 - Chinese: 4962 - French: 4251 - Thai: 3042 - Portuguese (Brazil): 2969 - Catalan: 2260 - Korean: 1553 - Ukrainian: 1352 - Italian: 1320 - Japanese: 1018 <details> <summary><b>Languages with under 1000 messages</b></summary> <ul> <li>Vietnamese: 952</li> <li>Basque: 947</li> <li>Polish: 886</li> <li>Hungarian: 811</li> <li>Arabic: 666</li> <li>Dutch: 628</li> <li>Swedish: 512</li> <li>Turkish: 454</li> <li>Finnish: 386</li> <li>Czech: 372</li> <li>Danish: 358</li> <li>Galician: 339</li> <li>Hebrew: 255</li> <li>Romanian: 200</li> <li>Norwegian Bokmål: 133</li> <li>Indonesian: 115</li> <li>Bulgarian: 95</li> <li>Bengali: 82</li> <li>Persian: 72</li> <li>Greek: 66</li> <li>Esperanto: 59</li> <li>Slovak: 19</li> </ul> </details> ## Contact - Discord Open Assistant Discord Server - GitHub: LAION-AI/Open-Assistant - E-Mail: open-assistant@laion.ai

The MATHWELL Human Annotation Dataset contains 5,084 synthetic word problems and answers generated by MATHWELL, a reference-free educational grade school math word problem generator released in MATHWELL: Generating Educational Math Word Problems Using Teacher Annotations, and comparison models (GPT-4, GPT-3.5, Llama-2, MAmmoTH, and LLEMMA) with expert human annotations for solvability, accuracy, appropriateness, and meets all criteria (MaC). Solvability means the problem is mathematically possible to solve, accuracy means the Program of Thought (PoT) solution arrives at the correct answer, appropriateness means that the mathematical topic is familiar to a grade school student and the question's context is appropriate for a young learner, and MaC denotes questions which are labeled as solvable, accurate, and appropriate. Null values for accuracy and appropriateness indicate a question labeled as unsolvable, which means it cannot have an accurate solution and is automatically inappropriate. Based on our annotations, 82.2% of the question/answer pairs are solvable, 87.3% have accurate solutions, 78.1% are appropriate, and 58.4% meet all criteria. This dataset is designed to train text classifiers to automatically label word problem generator outputs for solvability, accuracy, and appropriateness. More details about the dataset can be found in our paper.

The dataset from the study "A Fully Generative Motivational Interviewing Counsellor Chatbot for Moving Smokers Towards the Decision to Quit". The dataset comprises annotated transcripts and surveys (including self-reported readiness to quit smoking) from 106 conversations between human smokers and MIBot v6.3A — a motivational interviewing (MI) chatbot built using OpenAI's GPT-4o. ## Dataset Usage Researchers and practitioners can use this dataset to: 1. Analyze the language of MI-based counseling sessions, 2. Evaluate the relationship between dialogue features and behavior change outcomes, and 3. Train or benchmark new conversational agents on MI-style interactions. We provide both conversation transcripts, broken down by utterance, and pre and post-conversation survey data.

This dataset comprises 77,175 Reddit posts from 115 subreddit forums, annotated for the presence of 15 topics related to eating disorders and dieting. The dataset includes labels and scores on all 77,175 Reddit posts, determined by 5 Large Language Models: GPT-4o, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Vicuna-7b-v1.5, as well as by the ensemble of the four open-source LLMs. The dataset also includes a subset of 1,080 human-annotated posts for evaluation.

This dataset contains synthetically generated discussions and annotations using exclusively Large Language Model (LLM) agents. Discussions are performed between randomly selected users, with a LLM moderator/facilitator following various facilitation strategies. Each LLM user and annotator use a distinct Sociodemographic Background. User-agents also have different roles when joining the discussion. Each discussion consists of 28 comments - 14 participant comments and 14 moderator interventions. Each comment is annotated by 10 different LLM annotators for toxicity and argument quality. Designed to analyze facilitation strategies and synthetic online discussion simulation but can also be used for finetuning LLM facilitators.

Este conjunto de datos consiste en comentarios de publicaciones del MINSA (Perú) en Facebook sobre la vacuna contra el VPH entre los años 2019 y 2020. Se leyó cuidadosamente cada uno de los comentarios, luego se procedió a clasificarlos de manera manual. Para esta clasificación se interpretó los mensajes de las personas, por lo que se analizó los hilos (comentarios y respuestas) por separado y se procedió a etiquetarlos por temas "Topic" . Un profesional de salud realizó una segunda clasificación y las discrepancias se resolvieron con un tercer profesional. Luego, se seleccionaron subcategorías que hacían referencia directa a las vacunas contra el VPH. La clasificación se realizó utilizando las siguientes categorías "topic_c" : - 0: El comentario tiene una postura contraria a la vacuna contra el VPH (antivacuna) - 1: El comentario tien una postura a favor de la vacuna contra el VPH (provacuna) - 2: El comentario refleja una duda o dudas relacionada con la vacuna contra el VPH - 3: El comentario habla de cualquier otra cosa Citar: - Lewis De La Cruz, Lucy Cordova, & Esperanza Reyes. (2023). <i>Text_VPH</i> [Data set]. Kaggle. https://doi.org/10.34740/KAGGLE/DSV/6460567