Automatic Speech Recognition (ASR)

We introduce FLEURS, the Few-shot Learning Evaluation of Universal Representations of Speech benchmark. FLEURS is an n-way parallel speech dataset in 102 languages built on top of the machine translation FLoRes-101 benchmark, with approximately 12 hours of speech supervision per language. FLEURS can be used for a variety of speech tasks, including Automatic Speech Recognition (ASR), Speech Language Identification (Speech LangID), Translation and Retrieval. In this paper, we provide baselines for the tasks based on multilingual pre-trained models like mSLAM. The goal of FLEURS is to enable speech technology in more languages and catalyze research in low-resource speech understanding.

The Oxford-BBC Lip Reading Sentences 2 (LRS2) dataset is one of the largest publicly available datasets for lip reading sentences in-the-wild. The database consists of mainly news and talk shows from BBC programs. Each sentence is up to 100 characters in length. The training, validation and test sets are divided according to broadcast date. It is a challenging set since it contains thousands of speakers without speaker labels and large variation in head pose. The pre-training set contains 96,318 utterances, the training set contains 45,839 utterances, the validation set contains 1,082 utterances and the test set contains 1,242 utterances. Source: Audio-visual Recognition of Overlapped speech for the LRS2 dataset Image Source: https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs2.html

VoxPopuli is a large-scale multilingual corpus providing 100K hours of unlabelled speech data in 23 languages. It is the largest open data to date for unsupervised representation learning as well as semi-supervised learning. VoxPopuli also contains 1.8K hours of transcribed speeches in 16 languages and their aligned oral interpretations into 5 other languages totaling 5.1K hours. Source: VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation

LRS3-TED is a multi-modal dataset for visual and audio-visual speech recognition. It includes face tracks from over 400 hours of TED and TEDx videos, along with the corresponding subtitles and word alignment boundaries. The new dataset is substantially larger in scale compared to other public datasets that are available for general research. Source: LRS3-TED: a large-scale dataset for visual speech recognition

The REVERB (REverberant Voice Enhancement and Recognition Benchmark) challenge is a benchmark for evaluation of automatic speech recognition techniques. The challenge assumes the scenario of capturing utterances spoken by a single stationary distant-talking speaker with 1-channe, 2-channel or 8-channel microphone-arrays in reverberant meeting rooms. It features both real recordings and simulated data. The challenge constis of speech enhancement and automatic speech recognition tasks in reverberant environments. The speech enhancement challenge task consists of enhancing noisy reverberant speech with single-/multi-channel speech enhancement techniques, and evaluating the enhanced data in terms of objective and subjective evaluation metrics. The automatic speech recognition challenge task consists of improving the recognition accuracy of the same reverberant speech. The background noise is mostly stationary and the signal-to-noise ratio is modest. Source: https://reverb2014.audiolabs-erlangen.de

The MagicData-RAMC corpus contains 180 hours of conversational speech data recorded from native speakers of Mandarin Chinese over mobile phones with a sampling rate of 16 kHz. The dialogs in the dialogs are classified into 15 diversified domains and tagged with topic labels, ranging from science and technology to ordinary life. Accurate transcription and precise speaker voice activity timestamps are manually labeled for each sample. Speakers' detailed information is also provided.

The Audio Signal and Information Processing Lab at Westlake University, in collaboration with AISHELL, has released the Real-recorded and annotated Microphone Array speech&Noise (RealMAN) dataset, which provides annotated multi-channel speech and noise recordings for dynamic speech enhancement and localization: - Microphone array: A 32-channel microphone array with high-fidelity microphones is used for recording - Speech source: A loudspeaker is used for playing source speech signals (about 35 hours of Mandarin speech) - Recording duration and scene: A total of 83.7 hours of speech signals (about 48.3 hours for static speaker and 35.4 hours for moving speaker) are recorded in 32 different scenes, and 144.5 hours of background noise are recorded in 31 different scenes. Both speech and noise recording scenes cover various common indoor, outdoor, semi-outdoor and transportation environments, which enables the training of general-purpose speech enhancement and source localization networks. - Annotation: To obtain the task-specific annotations, speaker location is annotated with an omni-directional fisheye camera by automatically detecting the loudspeaker. The direct-path signal is set as the target clean speech for speech enhancement, which is obtained by filtering the source speech signal with an estimated direct-path propagation filter.

The Edinburgh International Accents of English Corpus (EdAcc) is a new automatic speech recognition (ASR) dataset composed of 40 hours of English dyadic conversations between speakers with a diverse set of accents. EdAcc includes a wide range of first and second-language varieties of English and a linguistic background profile of each speaker. Source: The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR Image Source: The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR

ITALIC: An ITALian Intent Classification Dataset ITALIC is an intent classification dataset for the Italian language, which is the first of its kind. It includes spoken and written utterances and is annotated with 60 intents. The dataset is available on Zenodo and connectors ara available for the HuggingFace Hub. ### Data collection The data collection follows the MASSIVE NLU dataset which contains an annotated textual dataset for 60 intents. The data collection process is described in the paper Massive Natural Language Understanding. Following the MASSIVE NLU dataset, a pool of 70+ volunteers has been recruited to annotate the dataset. The volunteers were asked to record their voice while reading the utterances (the original text is available on MASSIVE dataset). Together with the audio, the volunteers were asked to provide a self-annotated description of the recording conditions (e.g., background noise, recording device). The audio recordings have also been validated and, in case of errors, re-recorded by the volunteers. All the audio recordings included in the dataset have received a validation from at least two volunteers. All the audio recordings have been validated by native italian speakers (self-annotated).

SpeechInstruct is a large-scale cross-modal speech instruction dataset. It contains 37,969 quadruplets composed of speech instructions, text instructions, text responses, and speech responses. Source: SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities Image Source: SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

Corpus of Egyptian Arabic-English Code-switching (ArzEn) is a spontaneous conversational speech corpus, obtained through informal interviews held at the German University in Cairo. The participants discussed broad topics, including education, hobbies, work, and life experiences. The corpus currently contains 12 hours of speech, having 6,216 utterances. The recordings were transcribed and translated into monolingual Egyptian Arabic and monolingual English. Source: https://arxiv.org/pdf/2211.16319v1.pdf

The data set contains several speakers. The 5 largest are listed individually, the rest are summarized as other. All audio files have a sampling rate of 44.1kHz. For each speaker, there is a clean variant in addition to the full data set, where the quality is even higher. Furthermore, there are various statistics. The dataset can also be used for automatic speech recognition (ASR) if audio files are converted to 16 kHz.

NusaCrowd is a collaborative initiative to collect and unite existing resources for Indonesian languages, including opening access to previously non-public resources. Through this initiative, the authors have has brought together 137 datasets and 117 standardized data loaders. The quality of the datasets has been assessed manually and automatically, and their effectiveness has been demonstrated in multiple experiments. Source: https://arxiv.org/pdf/2212.09648v2.pdf Image Source: NusaCrowd: Open Source Initiative for Indonesian NLP Resources

ESB is a benchmark for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets. It comprises eight English speech recognition datasets, capturing a broad range of domains, acoustic conditions, speaker styles, and transcription requirements. Source: ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition Image Source: https://arxiv.org/pdf/2210.13352v1.pdf

JamALT is a revision of the JamendoLyrics dataset (80 songs in 4 languages), adapted for use as an automatic lyrics transcription (ALT) benchmark. The lyrics have been revised according to the newly compiled annotation guide, which include rules about spelling, punctuation, and formatting. The audio is identical to the JamendoLyrics dataset. However, only 79 songs are included, as one of the 20 French songs has been removed due to concerns about potentially harmful content.

The Norwegian Parliamentary Speech Corpus (NPSC) is a speech corpus made by the Norwegian Language Bank at the National Library of Norway in 2019-2021. The NPSC consists of recordings of speech from Stortinget, the Norwegian parliament, and corresponding orthographic transcriptions to Norwegian Bokmål and Norwegian Nynorsk. All transcriptions are done manually by trained linguists or philologists, and the manual transcriptions are subsequently proofread to ensure consistency and accuracy. Entire days of Parliamentary meetings are transcribed in the dataset.

OpenSLR is a repository of open speech and language resources, including large-scale transcribed audio corpora and related software. It serves as a central platform for researchers and practitioners to access and share datasets used in speech recognition (ASR), text-to-speech (TTS), and linguistic research. The OpenSLR collection includes over 30 diverse datasets spanning more than 25 languages, such as Javanese, Nepali, Malayalam, Yoruba, and various English and Spanish dialects. These datasets are contributed by institutions including Google, North West University, and others. Most contain audio recordings along with transcriptions, covering both crowd-sourced and professionally recorded material. Many of the datasets are high-quality multi-speaker corpora intended for use in building ASR and TTS models, particularly for under-resourced languages. Use cases include multilingual speech recognition, dialect modeling, language technology research, and building open-source voice applications. The OpenSLR site also acts as a mirror for widely-used tools and models to ensure continued availability. Website: OpenSLR website Source: openslr/openslr

A Brazilian Portuguese TTS dataset featuring a female voice recorded with high quality in a controlled environment, with neutral emotion and more than 20 hours of recordings. with neutral emotion and more than 20 hours of recordings. Our dataset aims to facilitate transfer learning for researchers and developers working on TTS applications: a highly professional neutral female voice can serve as a good warm-up stage for learning language-specific structures, pronunciation and other non-individual characteristics of speech, leaving to further training procedures only to learn the specific adaptations needed (e.g. timbre, emotion and prosody). This can surely help enabling the accommodation of a more diverse range of female voices in Brazilian Portuguese. By doing so, we also hope to contribute to the development of accessible and high-quality TTS systems for several use cases such as virtual assistants, audiobooks, language learning tools and accessibility solutions. Possible use cases: TTS; Voice Conversion; ASR; Speech Enhancement

A database containing high sampling rate recordings of a single speaker reading sentences in Brazilian Portuguese with neutral voice, along with the corresponding text corpus. Intended for speech synthesis and automatic speech recognition applications, the dataset contains text extracted from a popular Brazilian news TV program, totalling roughly 20 h of audio spoken by a trained individual in a controlled environment. The text was normalized in the recording process and special textual occurrences (e.g. acronyms, numbers, foreign names etc.) were replaced by their phonetic translation to a readable text in Portuguese. There are no noticeable accidental sounds and background noise has been kept to a minimum in all audio samples. Intended for: TTS ASR Speech Enhancement Voice Conversion

IMaSC is a Malayalam text and speech corpus made available by ICFOSS for the purpose of developing speech technology for Malayalam, particularly text-to-speech. The corpus contains 34,473 text-audio pairs of Malayalam sentences spoken by 8 speakers, totalling in approximately 50 hours of audio.

The M-AILABS Speech Dataset is the first large dataset that we are providing free-of-charge, freely usable as training data for speech recognition and speech synthesis. Most of the data is based on LibriVox and Project Gutenberg. The training data consist of nearly thousand hours of audio and the text-files in prepared format. A transcription is provided for each clip. Clips vary in length from 1 to 20 seconds and have a total length of approximately shown in the list (and in the respective info.txt-files) below. The texts were published between 1884 and 1964, and are in the public domain. The audio was recorded by the LibriVox project and is also in the public domain

OpenSpeaks Voice: Odia is a large speech dataset in the Odia language of India that is stewarded by Subhashish Panigrahi and is hosted at the O Foundation. It currently hosts over 70,000 audio files under a Universal Public Domain (CC0 1.0) Release. Of these, 66,000, hosted on Wikimedia Commons, include pronunciation of words and phrases, and the remaining 4,400 include pronunciation of sentences and are hosted on Mozilla Common Voice. The files on Wikimedia Commons were also released n 2023 as four physical media in the form of DVD-ROMs titled OpenSpeaks Voice: Odia Volume I, OpenSpeaks Voice: Odia Volume II, OpenSpeaks Voice: Balesoria-Odia Volume I, and OpenSpeaks Voice: Balesoria-Odia Volume II. The dataset uses Free/Libre and Open Source Software, primarily using web-based platforms such as Lingua Libre and Common Voice. Other tools used for this project include Kathabhidhana, developed by Panigrahi by forking the Voice Recorder for Tamil Wiktionary by Shrinivasan T, and Spell4wiki, Audacity among others. Over 64,000 files in this dataset are in the standard spoken variant of Odia (Central Odia), and the remaining 6,300 files are in Balesoria (Baleswari), the northern dialect of Odia. OpenSpeaks Voice: Balesoria-Odia Volume II was created by extracting words and phrases from the Nani Ma, a Balesoria-Odia documentary short directed by Panigrahi. The files within this dataset include transcription in Odia, making them accessible for automatic speech recognition (ASR). All the files are publicly available for ASR research and application building. Source: OpenSpeaks before Image Source: OpenSpeaks before

Speech Recognition Dataset for Oromo Language. 📊 Key features of Sagalee: *100 hours of read speech. *283 gender balanced speakers * Covers different dialects in Oromo language * Open source for research

arXiv GitHub ## Dataset Summary Speech Brown is a comprehensive, synthetic, and diverse paired speech-text dataset in 15 categories, covering a wide range of topics from fiction to religion. This dataset consists of over 55,000 sentence-level samples. To train the CLASP model, we created this dataset based on the Brown Corpus. The synthetic speech was generated using the NVIDIA Tacotron 2 text-to-speech model. For more information about our proposed model, please refer to this paper. The dataset generation pipeline, along with code and usage instructions, is available on this GitHub page. ## Dataset Statistics 1. Total size: Approximately 30 GB. 2. Number of samples: 55,173 pairs of speech and text. 3. Average tokens per sample: 19.00. 4. Maximum tokens in a sample: 48. 5. Average characters per sample: 96.72. 6. Number of unique tokens: 50,667 7. Categories: 15 categories consist of adventure, belles_lettres, editorial, fiction, government, hobbies, humor, learned, lore, mystery, news, religion, reviews, romance, science_fiction. ## Dataset Structure To ensure ease of use, the dataset is partitioned into 10 parts. Each part can be used independently if it meets the requirements of your task and model. ### Metadata Files 1. global_metadata: A JSON file containing metadata for all 55,173 samples. 2. localized_metadata: A JSON file containing metadata for all samples, categorized into the 10 dataset partitions. ### Metadata Fields 1. id: The unique identifier for the sample. 2. audio_file_path: The file path for the audio in the dataset. 3. category: The category of the sample's text. 4. text: The corresponding text of the audio file. ## Usage Instructions To use this dataset, download the parts and metadata files as follows: #### Option 1: Manual Download Visit the dataset repository and download all dataset_partX.zip files and the global_metadata.json file. #### Option 2: Programmatic Download Use the huggingface_hub library to download the files programmatically: python from huggingface_hub import hf_hub_download from zipfile import ZipFile import os import json # Download dataset parts zip_file_path1 = hf_hub_download(repo_id="llm-lab/SpeechBrown", filename="dataset_part1.zip", repo_type="dataset") zip_file_path2 = hf_hub_download(repo_id="llm-lab/SpeechBrown", filename="dataset_part2.zip", repo_type="dataset") # Download other parts... # Download metadata metadata_file_path = hf_hub_download(repo_id="llm-lab/SpeechBrown", filename="global_metadata.json", repo_type="dataset") for i in range(1, 11): with ZipFile(f'dataset_part{i}.zip', 'r') as zip_ref: zip_ref.extractall(f'dataset_part{i}') os.remove(f'dataset_part{i}.zip') with open('global_metadata.json', 'r') as f: metadata = json.load(f) metadata.keys() ## Citations If you find our paper, code, data, or models useful, please cite the paper: @misc{abootorabi2024claspcontrastivelanguagespeechpretraining, title={CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval}, author={Mohammad Mahdi Abootorabi and Ehsaneddin Asgari}, year={2024}, eprint={2412.13071}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2412.13071}, } ## Contact If you have questions, please email mahdi.abootorabi2@gmail.com or asgari@berkeley.edu.

The SWC is a corpus of aligned Spoken Wikipedia articles from the English, German, and Dutch Wikipedia. This corpus has several outstanding characteristics: - hundreds of hours of aligned audio - from a diverse set of readers - about a diverse set of topics - in a well-researched textual genre - licensed under a free license (CC BY-SA 4.0) - Annotations can be mapped back to the original html - phoneme-level alignments

This is the forehead accelerometer variant of the VibraVox dataset. VibraVox aims at serving as a valuable resource for advancing the field of body-conducted speech analysis and facilitating the development of robust communication systems for real-world applications. This dataset, available on HuggingFace can be used for various audio machine learning tasks : - Automatic Speech Recognition (ASR) (Speech-to-Text , Speech-to-Phoneme) - Audio Bandwidth Extension (BWE) - Speaker Verification (SPKV) / identification - Voice cloning - etc ... The VibraVox dataset speech corpus has been released in July 2024. It includes french speech recorded simultaneously using multiple audio and vibration sensors : a forehead miniature vibration sensor, an in-ear comply foam-embedded microphone, an in-ear rigid earpiece-embedded microphone, a temple vibration pickup, a headset microphone located near the mouth, and a laryngophone. VibraVox has been recorded with 200 participants under various acoustic conditions imposed by a 5th order ambisonics spatialization sphere.

This is the reference headset microphone variant of the VibraVox dataset. VibraVox aims at serving as a valuable resource for advancing the field of body-conducted speech analysis and facilitating the development of robust communication systems for real-world applications. This dataset, available on HuggingFace can be used for various audio machine learning tasks : - Automatic Speech Recognition (ASR) (Speech-to-Text , Speech-to-Phoneme) - Audio Bandwidth Extension (BWE) - Speaker Verification (SPKV) / identification - Voice cloning - etc ... The VibraVox dataset speech corpus has been released in July 2024. It includes french speech recorded simultaneously using multiple audio and vibration sensors : a forehead miniature vibration sensor, an in-ear comply foam-embedded microphone, an in-ear rigid earpiece-embedded microphone, a temple vibration pickup, a headset microphone located near the mouth, and a laryngophone. VibraVox has been recorded with 200 participants under various acoustic conditions imposed by a 5th order ambisonics spatialization sphere.

This is the in-ear rigid earpiece-embedded microphone variant of the VibraVox dataset. VibraVox aims at serving as a valuable resource for advancing the field of body-conducted speech analysis and facilitating the development of robust communication systems for real-world applications. This dataset, available on HuggingFace can be used for various audio machine learning tasks : - Automatic Speech Recognition (ASR) (Speech-to-Text , Speech-to-Phoneme) - Audio Bandwidth Extension (BWE) - Speaker Verification (SPKV) / identification - Voice cloning - etc ... The VibraVox dataset speech corpus has been released in July 2024. It includes french speech recorded simultaneously using multiple audio and vibration sensors : a forehead miniature vibration sensor, an in-ear comply foam-embedded microphone, an in-ear rigid earpiece-embedded microphone, a temple vibration pickup, a headset microphone located near the mouth, and a laryngophone. VibraVox has been recorded with 200 participants under various acoustic conditions imposed by a 5th order ambisonics spatialization sphere.

This is the in-ear comply foam-embedded microphone variant of the VibraVox dataset. VibraVox aims at serving as a valuable resource for advancing the field of body-conducted speech analysis and facilitating the development of robust communication systems for real-world applications. This dataset, available on HuggingFace can be used for various audio machine learning tasks : - Automatic Speech Recognition (ASR) (Speech-to-Text , Speech-to-Phoneme) - Audio Bandwidth Extension (BWE) - Speaker Verification (SPKV) / identification - Voice cloning - etc ... The VibraVox dataset speech corpus has been released in July 2024. It includes french speech recorded simultaneously using multiple audio and vibration sensors : a forehead miniature vibration sensor, an in-ear comply foam-embedded microphone, an in-ear rigid earpiece-embedded microphone, a temple vibration pickup, a headset microphone located near the mouth, and a laryngophone. VibraVox has been recorded with 200 participants under various acoustic conditions imposed by a 5th order ambisonics spatialization sphere.

This is the temple vibration pickup variant of the VibraVox dataset. VibraVox aims at serving as a valuable resource for advancing the field of body-conducted speech analysis and facilitating the development of robust communication systems for real-world applications. This dataset, available on HuggingFace can be used for various audio machine learning tasks : - Automatic Speech Recognition (ASR) (Speech-to-Text , Speech-to-Phoneme) - Audio Bandwidth Extension (BWE) - Speaker Verification (SPKV) / identification - Voice cloning - etc ... The VibraVox dataset speech corpus has been released in July 2024. It includes french speech recorded simultaneously using multiple audio and vibration sensors : a forehead miniature vibration sensor, an in-ear comply foam-embedded microphone, an in-ear rigid earpiece-embedded microphone, a temple vibration pickup, a headset microphone located near the mouth, and a laryngophone. VibraVox has been recorded with 200 participants under various acoustic conditions imposed by a 5th order ambisonics spatialization sphere.

This is the throat microphone (laryngophone) variant of the VibraVox dataset. VibraVox aims at serving as a valuable resource for advancing the field of body-conducted speech analysis and facilitating the development of robust communication systems for real-world applications. This dataset, available on HuggingFace can be used for various audio machine learning tasks : - Automatic Speech Recognition (ASR) (Speech-to-Text , Speech-to-Phoneme) - Audio Bandwidth Extension (BWE) - Speaker Verification (SPKV) / identification - Voice cloning - etc ... The VibraVox dataset speech corpus has been released in July 2024. It includes french speech recorded simultaneously using multiple audio and vibration sensors : a forehead miniature vibration sensor, an in-ear comply foam-embedded microphone, an in-ear rigid earpiece-embedded microphone, a temple vibration pickup, a headset microphone located near the mouth, and a laryngophone. VibraVox has been recorded with 200 participants under various acoustic conditions imposed by a 5th order ambisonics spatialization sphere.

VoxForge is an open speech dataset that was set up to collect transcribed speech for use with Free and Open Source Speech Recognition Engines (on Linux, Windows and Mac). We will make available all submitted audio files under the GPL license, and then 'compile' them into acoustic models for use with Open Source speech recognition engines such as CMU Sphinx, ISIP, Julius (github) and HTK (note: HTK has distribution restrictions).

This open-source dataset consists of 5.04 hours of transcribed English conversational speech beyond telephony, where 13 conversations were contained.

This dataset was created for Fongbe automatic speech recognition task and contains about 3979 recordings of 13 participants reading a text written in Fongbe, one sentence at a time. Fongbe is a vernacular language spoken mainly in Benin, by more than 50% of the population, and a littke in Togo and in Nigeria. It’s an under-resourced because it lacks linguistics resources (speech corpus and text data) and very few websites provide textual data. In this dataset, each example contains the audio files and the associated text. The audio is high-quality (16-bit, 16kHz) recorded using an adroid app that we built for the need. The dataset is multi-speaker, containing recordings from 13 volunteers (male and female).