Code Classification

xCodeEval is one of the largest executable multilingual multitask benchmarks consisting of 17 programming languages with execution-level parallelism. It features a total of seven tasks involving code understanding, generation, translation, and retrieval, and it employs an execution-based evaluation instead of traditional lexical approaches. It also provides a test-case-based multilingual code execution engine, ExecEval that supports all the programming languages in xCodeEval.

PyTorrent contains 218,814 Python package libraries from PyPI and Anaconda environment. This is because earlier studies have shown that much of the code is redundant and Python packages from these environments are better in quality and are well-documented. PyTorrent enables users (such as data scientists, students, etc.) to build off the shelf machine learning models directly without spending months of effort on large infrastructure.

• Description A dataset of assembly functions that are vulnerable to Spectre-V1 attack. * Motivation Several techniques have been proposed to detect vulnerable Spectre gadgets in widely deployed commercial software. Unfortunately, detection techniques proposed so far rely on hand-written rules which fall short in covering subtle variations of known Spectre gadgets as well as demand a huge amount of time to analyze each conditional branch in software. Moreover, detection tool evaluations are based only on a handful of these gadgets, as it requires arduous effort to craft new gadgets manually. * Potential Use Cases Generating assembly code. Evaluating new detection tools.

This dataset contains 304 manual evaluations of class-level software maintainability, drawn from 5 open-source projects: ArgoUML, Art of Illusion, Diary Management, JUnit 4, JSweet. Each Java class is labelled along 5 axis: readability, understandability, complexity, modularity and overall maintainability. Each Java class was assessed by several experts independently of its relation to other classes. It can be used to develop and evaluate automated quality prediction tools.

We introduce FixEval , a dataset for competitive programming bug fixing along with a comprehensive test suite and show the necessity of execution based evaluation compared to suboptimal match based evaluation metrics like BLEU, CodeBLEU, Syntax Match, Exact Match etc.

This repository holds two datasets: one with both the original binaries and the code sections extracted from them (“full dataset”), and one with only the code sections (“only code sections”). The code sections were extracted by carving out sections of the binary that were marked as executable. The binaries were scraped from Debian repositories. There are also two CSV files available, one with full binaries and one with only code sections, which include the 293 features extracted from about 3000 binaries per architecture. These features can be used to train classifiers. The dataset consists of thousands of binaries for the following 23 architectures: alpha, amd64, arm64, armel, armhf, hppa, i386, ia64, m68k, mips, mips64el, mipsel, powerpc, powerpcspe, powerpc64, powerpc64el, riscv, s390, s390x, sh4, sparc, sparc64 and x32. There are 98 500 binary files, about 27 gigabytes (uncompressed) of binary files and about 15 gigabytes (uncompressed) of only code sections from those binary files. Both datasets hold the binaries in directories named by the architecture. The files inside the folders are named as MD5 hashes of the original binary files, and a hash file ending with “.code” contains only the concatenation of all code sections of the original binary file. Each architecture folder also holds a JSON file named after the architecture, e.g. amd64 holds amd64.json. The structure of the JSON file is as follows (described in a JSON Schema-like notation) This work is based on work by John Clemens, 2015, “Automatic classification of object code using machine learning” and De Nicolao, Pietro et al., 2018, “ELISA: ELiciting ISA of Raw Binaries for Fine-Grained Code and Data Separation” This dataset is released as part of the following papers: Sami Kairajärvi, Andrei Costin, and Timo Hämäläinen. 2020. ISAdetect: Usable automated detection of ISA (CPU architecture and endianness) for executable binary files and object code. In Tenth ACM Conference on Data and Application Security and Privacy (CODASPY’20), March 16–18, 2020, New Orleans, LA, USA. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3374664.3375742 Kairajärvi, Sami, Andrei Costin, and Timo Hämäläinen. "Towards usable automated detection of CPU architecture and endianness for arbitrary binary files and object code sequences." arXiv preprint arXiv:1908.05459 (2019). Kairajärvi, Sami. "Automatic identification of architecture and endianness using binary file contents." (2019). The code associated with this dataset can be found at https://github.com/kairis/isadetect Changelog: version 6 - 29.3.2020 Add Weka models version 5 - 17.1.2020 Clean up dataset version 4 - 13.1.2020 Initial release

TACO (Topics in Algorithmic Code generation dataset) is a dataset focused on algorithmic code generation, designed to provide a more challenging training dataset and evaluation benchmark for the code generation model field. The dataset consists of programming competition problems that are more difficult and closer to real programming scenarios. It emphasizes improving or evaluating the model's understanding and reasoning abilities in practical application scenarios, rather than just implementing predefined function functionalities.

CodeSCAN is the first large-scale and diverse dataset of coding screenshots with pixel-perfect annotations. It features: - 24 popular programming languages (according to Github) - 100 random repositories per language (with MIT, BSD-3 or WTFPL License), i.e. 2.400 repositories in total - Per repository we use 5 files, i.e. 12.000 files in total - ~100 different themes and 25 different fonts - Diverse layouts changes, such as menu bar visibility, sidebar position, output window content, etc. - Numerous realistic interactions such as searching, typing and selecting within a file, etc. Check our project page (https://a-nau.github.io/codescan/) for details.