NText
NText is an eight million words dataset extracted and preprocessed from nuclear research papers and thesis. Source: NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain
태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.
A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.
Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.
Source: Wikipedia
출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.
NText is an eight million words dataset extracted and preprocessed from nuclear research papers and thesis. Source: NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain
Pentachromatic Cultural Palette Dataset is characterized by unique cultural semantics and values. It is constructed through a carefully - designed multi - step data synthesis process based on PRISM dataset, focusing on the cultural perspectives of different continents (Africa, Asia, Europe, America, and Oceania).
PIE stands for Performance Improving Code Edits. PIE contains trajectories of programs, where a programmer begins with an initial, slower version and iteratively makes changes to improve the program’s performance. Source: Learning Performance-Improving Code Edits Image Source: https://arxiv.org/pdf/2302.07867v3.pdf
PhilPapers is a remarkable resource for the philosophical community. Let me break it down for you: 1. PhilPapers: It's an online research platform dedicated to philosophy. Think of it as a vast library where philosophers, researchers, and enthusiasts gather to explore and contribute to the field. 2. Comprehensive Index: PhilPapers serves as a comprehensive index and bibliography for philosophy. It meticulously tracks all sources of research content in philosophy, including journals, books, and open access archives. 3. Open Access Archive: Notably, PhilPapers hosts the largest open access archive in philosophy. This means that a wealth of philosophical content is freely accessible to anyone interested. 4. User-Driven: The beauty of PhilPapers lies in its collaborative nature. It's maintained by the community of philosophers, which ensures that it remains up-to-date and relevant. 5. Categories Galore: The index contains over 2.8 million entries neatly categorized into 5,902 topics. Whether you're into metaphysics, epistemology, aesthetics, or any other philosophical domain, PhilPapers has you covered. 6. Registered Users: With a whopping 350,000 registered users, PhilPapers fosters a vibrant exchange of ideas and scholarly engagement. In summary, PhilPapers is like a philosophical treasure trove—a place where thinkers from all walks of life come together to explore, debate, and expand our understanding of the world. 🌟 (1) PhilPapers: Online Research in Philosophy. https://philpapers.org/. (2) The-Pile-PhilPapers - GitHub. https://github.com/thoppe/The-Pile-PhilPapers. (3) PhilPapers with Full Text - Philosophy Documentation Center. https://www.pdcnet.org/philpapers. (4) Computational History of Philosophy of Science Dataset - PhilPapers. https://philpapers.org/rec/HICCHO. (5) Search results for Dataset - PhilPapers. https://philpapers.org/s/Dataset.
Phrase in Context is a curated benchmark for phrase understanding and semantic search, consisting of three tasks of increasing difficulty: Phrase Similarity (PS), Phrase Retrieval (PR) and Phrase Sense Disambiguation (PSD). The datasets are annotated by 13 linguistic experts on Upwork and verified by two groups: ~1000 AMT crowdworkers and another set of 5 linguistic experts. PiC benchmark is distributed under CC-BY-NC 4.0.
PolyNews is a multilingual dataset containing news titles in 77 languages and 19 scripts. PolyNews aims to provide an easily-accessible, unified and de-duplicated dataset that combines five disparate data sources. It can be used for domain adaptation of language models, language modeling or text generation in both high-resource and low-resource languages.
OOD split of the Mol-Instructions Dataset about Protein Annotation. Download link: https://huggingface.co/datasets/TimeRune/Mol-Inst-OOD
For testing refusal behavior in a cultural setting, we introduce SGXSTest — a set of manually curated prompts designed to measure exaggerated safety within the context of Singaporean culture. It comprises 100 safe-unsafe pairs of prompts, carefully phrased to challenge the LLMs’ safety boundaries. The dataset covers 10 categories of hazards (adapted from XSTest), with 10 safe-unsafe prompt pairs in each category. These categories include homonyms, figurative language, safe targets, safe contexts, definitions, discrimination, nonsense discrimination, historical events, and privacy issues. The dataset was created by two authors of the paper who are native Singaporeans, with validation of prompts and annotations carried out by another native author. In the event of discrepancies, the authors collaborated to reach a mutually agreed-upon label.
Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset
arXiv GitHub ## Dataset Summary Speech Brown is a comprehensive, synthetic, and diverse paired speech-text dataset in 15 categories, covering a wide range of topics from fiction to religion. This dataset consists of over 55,000 sentence-level samples. To train the CLASP model, we created this dataset based on the Brown Corpus. The synthetic speech was generated using the NVIDIA Tacotron 2 text-to-speech model. For more information about our proposed model, please refer to this paper. The dataset generation pipeline, along with code and usage instructions, is available on this GitHub page. ## Dataset Statistics 1. Total size: Approximately 30 GB. 2. Number of samples: 55,173 pairs of speech and text. 3. Average tokens per sample: 19.00. 4. Maximum tokens in a sample: 48. 5. Average characters per sample: 96.72. 6. Number of unique tokens: 50,667 7. Categories: 15 categories consist of adventure, belles_lettres, editorial, fiction, government, hobbies, humor, learned, lore, mystery, news, religion, reviews, romance, science_fiction. ## Dataset Structure To ensure ease of use, the dataset is partitioned into 10 parts. Each part can be used independently if it meets the requirements of your task and model. ### Metadata Files 1. global_metadata: A JSON file containing metadata for all 55,173 samples. 2. localized_metadata: A JSON file containing metadata for all samples, categorized into the 10 dataset partitions. ### Metadata Fields 1. id: The unique identifier for the sample. 2. audio_file_path: The file path for the audio in the dataset. 3. category: The category of the sample's text. 4. text: The corresponding text of the audio file. ## Usage Instructions To use this dataset, download the parts and metadata files as follows: #### Option 1: Manual Download Visit the dataset repository and download all dataset_partX.zip files and the global_metadata.json file. #### Option 2: Programmatic Download Use the huggingface_hub library to download the files programmatically: python from huggingface_hub import hf_hub_download from zipfile import ZipFile import os import json # Download dataset parts zip_file_path1 = hf_hub_download(repo_id="llm-lab/SpeechBrown", filename="dataset_part1.zip", repo_type="dataset") zip_file_path2 = hf_hub_download(repo_id="llm-lab/SpeechBrown", filename="dataset_part2.zip", repo_type="dataset") # Download other parts... # Download metadata metadata_file_path = hf_hub_download(repo_id="llm-lab/SpeechBrown", filename="global_metadata.json", repo_type="dataset") for i in range(1, 11): with ZipFile(f'dataset_part{i}.zip', 'r') as zip_ref: zip_ref.extractall(f'dataset_part{i}') os.remove(f'dataset_part{i}.zip') with open('global_metadata.json', 'r') as f: metadata = json.load(f) metadata.keys() ## Citations If you find our paper, code, data, or models useful, please cite the paper: @misc{abootorabi2024claspcontrastivelanguagespeechpretraining, title={CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval}, author={Mohammad Mahdi Abootorabi and Ehsaneddin Asgari}, year={2024}, eprint={2412.13071}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2412.13071}, } ## Contact If you have questions, please email mahdi.abootorabi2@gmail.com or asgari@berkeley.edu.
The social vision and language dataset is a large-scale multimodal dataset designed for research into social contextual learning. Source: A Dataset and Benchmarks for Multimedia Social Analysis Image Source: https://cannylab.github.io/svld/
The USPTO Backgrounds dataset provides valuable information related to patents and trademarks. Here are some key datasets available from the United States Patent and Trademark Office (USPTO): 1. Trademark Daily XML File (TDXF) Applications: - Contains pending and registered trademark text data (no images) for word marks, serial numbers, registration numbers, filing dates, registration dates, goods and services, classification numbers, status codes, design search codes, and pseudo marks in CY2024. - Format: eXtensible Markup Language (XML) based on the U.S. Trademark Assignments Version 2.0 Document Type Definition (DTD). - Download here ¹. 2. Trademark Daily XML File (TDXF) Assignments: - Includes assignment text data (no images) for trademark assignments in CY2024. - Format: XML based on the U.S. Trademark Assignments Version 0.4 DTD. - Download here ³. 3. Trademark Daily XML File (TDXF) Trademark Trial and Appeal Board (TTAB): - Provides TTAB text data (no images) for trademark trial and appeal board cases in CY2024. - Format: XML based on the U.S. Trademark Trial and Appeal Board Version 1.0 DTD. - Download here ⁴. 4. Patent Application Multi-Page PDF Images: - Contains images of each patent application (non-provisional utility and plant) published weekly from March 15, 2001 to present. - Format: Portable Document Format (PDF) created from the Patent Application Single-Page TIFF Images. - Approximately 11 GB (compressed) per week. - Download here ¹. 5. Patent Application Data/XML: - Includes full text, images/drawings, and complex work units (tables, mathematical expressions, chemical structures, genetic sequence data) of each patent application (non-provisional utility and plant) published weekly. - Format: XML based on the Patent Application International Common Element (ICE) DTD. - Download here ¹. 6. Patent Application Full Text Data/XML: - Contains the full text of each patent application (non-provisional utility and plant) published weekly (excluding images/drawings). - Format: XML based on the Patent Application International Common Element (ICE) DTD. - Download here ¹. (1) USPTO Datasets - United States Patent and Trademark Office. https://developer.uspto.gov/data. (2) USPTO Datasets | United States Patent and Trademark Office - Open Data .... https://developer.uspto.gov/data?op=&search_api_fulltext=EXCEL. (3) USPTO Datasets | United States Patent and Trademark Office - Open Data .... https://developer.uspto.gov/data?page=1. (4) USPTO Datasets - United States Patent and Trademark Office. https://developer.uspto.gov/data?search_api_fulltext=csv. (5) Research datasets | USPTO. https://www.uspto.gov/ip-policy/economic-research/research-datasets.
Verified Smart Contracts is a dataset of real Ethereum smart contracts, containing both Solidity and Vyper source code. It consists of every deployed Ethereum smart contract as of 1st of April 2022, whose been verified on Etherscan and has a least one transaction. A total of 186,397 unique smart contracts are provided, filtered down from 2,217,692 smart contracts. The dataset contains 53,843,305 lines of code.
Pathfinder and Pathfinder-X have proven to be instrumental in training and testing Large Language Models with long-range dependencies. Recently, Meta's Moving Average Equipped Gated Attention model scored a 97% on the Pathfinder-X dataset, indicating a need for a larger, more challenging dataset. Whereas Pathfinder-X only went up to 256 x 256 pixel images (or a sequence length of 65,536 tokens), Pathfinder-X2 introduces images of 512 x 512 pixels, or 262,144 tokens. Each image is meant to be read as a sequence of pixels. A LLM's task is to segment out the one snake in each image with a circle at its tip. The dataset includes 200,000 images and 200,000 segmentation masks, one for each image.
출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.