• Description A dataset of assembly functions that are vulnerable to Spectre-V1 attack. * Motivation Several techniques have been proposed to detect vulnerable Spectre gadgets in widely deployed commercial software. Unfortunately, detection techniques proposed so far rely on hand-written rules which fall short in covering subtle variations of known Spectre gadgets as well as demand a huge amount of time to analyze each conditional branch in software. Moreover, detection tool evaluations are based only on a handful of these gadgets, as it requires arduous effort to craft new gadgets manually. * Potential Use Cases Generating assembly code. Evaluating new detection tools.

Test-driven benchmark to challenge LLMs to write JavaScript React application GitHub Script

The Java dataset introduced in DeepCom (Deep Code Comment Generation), commonly used to evaluate automated code summarization.

DISL The full dataset report is available at: https://arxiv.org/abs/2403.16861 The DISL dataset features a collection of 514, 506 unique Solidity files that have been deployed to Ethereum mainnet. It caters to the need for a large and diverse dataset of real-world smart contracts. DISL serves as a resource for developing machine learning systems and for benchmarking software engineering tools designed for smart contracts. - Curated by: Gabriele Morello - License: [MIT] ## Instructions to explore the dataset python from datasets import load_dataset # Load the raw dataset dataset = load_dataset("ASSERT-KTH/DISL", "raw") # OR # Load the decomposed dataset dataset = load_dataset("ASSERT-KTH/DISL", "decomposed") # number of rows and columns num_rows = len(dataset["train"]) num_columns = len(dataset["train"].column_names) # random row import random random_row = random.choice(dataset["train"]) # random source code random_sc = random.choice(dataset["train"])['source_code'] print(random_sc)

In this paper, we introduce a novel benchmarking framework designed specifically for evaluations of data science agents. Our contributions are three-fold. First, we propose DSEval, an evaluation paradigm that enlarges the evaluation scope to the full lifecycle of LLM-based data science agents. We also cover aspects including but not limited to the quality of the derived analytical solutions or machine learning models, as well as potential side effects such as unintentional changes to the original data. Second, we incorporate a novel bootstrapped annotation process letting LLM themselves generate and annotate the benchmarks with ``human in the loop''. A novel language (i.e., DSEAL) has been proposed and the derived four benchmarks have significantly improved the benchmark scalability and coverage, with largely reduced human labor. Third, based on DSEval and the four benchmarks, we conduct a comprehensive evaluation of various data science agents from different aspects. Our findings reveal the common challenges and limitations of the current works, providing useful insights and shedding light on future research on LLM-based data science agents. This is one of DSEval benchmarks.

In this paper, we introduce a novel benchmarking framework designed specifically for evaluations of data science agents. Our contributions are three-fold. First, we propose DSEval, an evaluation paradigm that enlarges the evaluation scope to the full lifecycle of LLM-based data science agents. We also cover aspects including but not limited to the quality of the derived analytical solutions or machine learning models, as well as potential side effects such as unintentional changes to the original data. Second, we incorporate a novel bootstrapped annotation process letting LLM themselves generate and annotate the benchmarks with ``human in the loop''. A novel language (i.e., DSEAL) has been proposed and the derived four benchmarks have significantly improved the benchmark scalability and coverage, with largely reduced human labor. Third, based on DSEval and the four benchmarks, we conduct a comprehensive evaluation of various data science agents from different aspects. Our findings reveal the common challenges and limitations of the current works, providing useful insights and shedding light on future research on LLM-based data science agents. This is one of DSEval benchmarks.

In this paper, we introduce a novel benchmarking framework designed specifically for evaluations of data science agents. Our contributions are three-fold. First, we propose DSEval, an evaluation paradigm that enlarges the evaluation scope to the full lifecycle of LLM-based data science agents. We also cover aspects including but not limited to the quality of the derived analytical solutions or machine learning models, as well as potential side effects such as unintentional changes to the original data. Second, we incorporate a novel bootstrapped annotation process letting LLM themselves generate and annotate the benchmarks with ``human in the loop''. A novel language (i.e., DSEAL) has been proposed and the derived four benchmarks have significantly improved the benchmark scalability and coverage, with largely reduced human labor. Third, based on DSEval and the four benchmarks, we conduct a comprehensive evaluation of various data science agents from different aspects. Our findings reveal the common challenges and limitations of the current works, providing useful insights and shedding light on future research on LLM-based data science agents. This is one of DSEval benchmarks.

To automatically generate Python and assembly programs used for security exploits, we curated a large dataset for feeding NMT techniques. A sample in the dataset consists of a snippet of code from these exploits and their corresponding description in the English language. We collected exploits from publicly available databases (exploitdb, shellstorm), public repositories (e.g., GitHub), and programming guidelines. In particular, we focused on exploits targeting Linux, the most common OS for security-critical network services, running on IA-32 (i.e., the 32-bit version of the x86 Intel Architecture). The dataset is stored in the folder EVIL/datasets and consists of two parts: i) Encoders: a Python dataset, which contains Python code used by exploits to encode the shellcode; ii) Decoders: an assembly dataset, which includes shellcode and decoders to revert the encoding.

145k natural language and PDDL problem pairs from the Blocks World, Gripper, and Floor Tile domains.

The first benchmark comprising 473 prompts designed to assess the ability of LLMs to resist malicious code generation.

The dataset consists of source code and LLVM IR pairs generated from accepted and de-duped programming contest solutions. The dataset is divided into language configs and mode splits. The language can be one of C, C++, D, Fortran, Go, Haskell, Nim, Objective-C, Python, Rust and Swift, indicating the source files' languages. The mode split indicates the compilation mode, which can be wither Size_Optimized or Perf_Optimized.

VerilogEval Dataset The VerilogEval Dataset is a benchmark specifically designed to assess the ability of large language models (LLMs) to generate syntactically correct and functionally accurate Verilog code. Introduced in the paper VerilogEval: Evaluating Large Language Models for Verilog Code Generation, it has become a cornerstone for research in hardware code generation. --- ## Dataset Characteristics - Diverse Problem Set: The dataset comprises 156 distinct problems sourced from HDLBits, covering a wide range of digital design tasks—from simple combinational circuits to more complex sequential and state-based designs. - Dual Descriptions: It includes two types of problem statements: - VerilogEval-human: Handcrafted descriptions by experts, closely reflecting real-world design challenges. - VerilogEval-machine: Machine-generated descriptions, typically more verbose, to simulate automated problem formulation. - Automated Evaluation Framework: Each problem comes with a canonical solution and an evaluation harness that uses simulation (e.g., via Icarus Verilog) to verify the functional correctness of generated code. Evaluation metrics such as pass@k account for the non-deterministic outputs of LLMs. - Rich Evaluation Metrics: The dataset supports detailed error analysis by classifying failures (e.g., syntax errors, simulation mismatches) and quantifying performance improvements through metrics like pass@1 and pass@5. --- ## Motivations and Content Summary The primary motivations behind the VerilogEval Dataset are: - Standardized Benchmarking: To provide a reproducible framework for evaluating and comparing the performance of various LLMs on Verilog code generation tasks. - Advancing Hardware Design Automation: By focusing on Verilog—a key hardware description language—the dataset encourages research that bridges AI and digital hardware design, ultimately helping to automate and accelerate chip design processes. - Facilitating Model Improvement: The dataset’s detailed error classifications and pass rate metrics help pinpoint specific weaknesses in LLM-generated code, guiding future research in prompt engineering, in-context learning, and fine-tuning methods for improved performance. --- ## Potential Use Cases - Benchmarking and Comparison: Researchers can use the dataset to measure and compare the performance of different LLMs (e.g., GPT-4, CodeGen, etc.) in generating correct Verilog code. - Prompt Engineering Research: The dataset allows exploration of the effects of prompt tuning and in-context learning on the quality of generated hardware description language code. - Fine-Tuning and Domain Adaptation: It serves as an excellent resource for supervised fine-tuning, enabling models to adapt better to the nuances of Verilog and hardware design tasks. - Educational Resource: Educators and students can leverage the dataset to practice Verilog coding, test design understanding, and learn automated testing techniques in digital design courses. - EDA Tool Integration: The dataset can be integrated into Electronic Design Automation (EDA) workflows for automatic code verification, debugging, and performance analysis. --- ## Example Evaluation Metric An example of a metric used in the evaluation is the pass rate, defined as: \[ \text{Pass Rate} = \frac{\text{Number of samples that pass functional tests}}{n} \] This metric quantifies the reliability of generated code by considering the non-deterministic nature of LLM outputs, where multiple samples are generated per problem.

The dataset is specifically constructed for the library-oriented code generation task, which are constructed in the paper “CodeGen4Libs: A Two-Stage Approach for Library-Oriented Code Generation”.

COFFE COFFE is a Python benchmark for evaluating the time efficiency of LLM-generated code. It is released by the FSE'25 paper "COFFE: A Code Efficiency Benchmark for Code Generation". You can also refer to the project webpage for more details. ## Data COFFE is designed for evaluating both function-level code and file-level code. It contains selected instances from HumanEval, MBPP, APPS and Code Contests. COFFE keeps the original test cases in these benchmarks as correctness test cases and adds new test cases designed for time efficiency evaluation as stressful test cases. Statistics: |Category|#Instance|#Solution/Instance|#Correctness/Instance | #Stressful/Instance| |----|----|----|----|----| |Function-Level|398|1.00|5.72|4.99| |File-Level|358|66.93|43.68|4.95| Data Files: All instances in COFFE are in Coffe/datasets, where Coffe/datasets/function contains all function-level instances and Coffe/datasets/file contains all file-level instances. In each repo: - best_solutions.json contains the best ground truth solution COFFE uses to calculate efficient@1 and speedup. - stressful_testcases.json contains all stressful test cases COFFE adds. - solutions.json contains all ground truth solutions in original benchmarks. - testcases.json contains all correctness test cases in original benchmarks. ## Installation Requirements: - Linux Machine - Docker - Python>=3.10 We suggest you create a virtural environment before installing COFFE. 1. To use COFFE, please clone this repo in your current workspace workspace/ and execute: bash cd Coffe && pip install . 2. COFFE comes with a docker image, to install it: bash docker build . -t coffe Note that if your network requires proxy, please modify the Dockerfile in Coffe/ to indicate it, otherwise the docker image building process could fail. 3. Go back to your workspace and initialize COFFE: bash cd .. && coffe init If your installation succeeds, you could see the statistics of COFFE. ## Usage ### Pipeline When you prepare the predictions from LLMs, COFFE provides a pipeline to calculate the efficient@1 and speedup defined in the paper: bash coffe pipe <benchmark_name> <output_repo> -p <pred_file_1>,...,<pred_file_n> -f <efficient_at_1/speedup> -n <number_of_workers> This command has four phases: 1. santize the predictions. 2. select the correct predictions based on correctness test cases. 3. evaluate the GPU instruction count based on stressful test cases. 4. calculate the final metrics. For example: bash coffe pipe function Coffe/examples/function -p Coffe/examples/function/GPT-4o.json -f efficient_at_1 -n 8 This command evaluates the predictions from GPT-4o on the function-level instances of COFFE. If you want to evaluate other LLMs, please prepare a JSON file with the same format as Coffe/examples/function/GPT-4o.json. Prediction File Format: In the JSON file, the key is the prompt used to query the LLM for the results, you could get the prompts in datasets/function/prompts.json and datasets/file/prompts.json. The value contains two objects, the first is a list contains the raw outputs from LLMs and the second is an indicator for the whether the raw output is valid. Note: In default, COFFE will run all predictions in docker. However, if you could not successfully install the docker or want to run the predictions on the host machine, you can add the -x option. ### Single Evaluation The pipe command provides an entire pipeline for calculating the final metrics. This pipeline could also be completed by executing the following four single evaluation commands. 1. Sanitize the predictions bash coffe eval <benchmark_name> <output_repo> -p <pred_file_1>,...,<pred_file_n> -m compilable_rate This commands output a file ending with SOLUTIONS.json that contains the predictions without syntax errors. 2. Select correct predictions bash coffe eval <benchmark_name> <output_repo> -p <pred_file_1>,...,<pred_file_n> -m correctness -n <number_of_workers> This commands accept prediction files ending with SOLUTIONS.json and output a file ending with PASSED_SOLUTIONS.json that contains the predictions pass all correctness solutions. Note: This command will combine all correct solutions and ground truth solutions together into files <dataset_name>_all_indexes.json (used in step 4) and <dataset_name>_all_PASSED_SOLUTIONS.json for the next step. 3. Evaluate the GPU instruction count bash coffe eval <benchmark_name> <output_repo> -p <pred_file> -m instr_count -n <number_of_workers> This command will evaluate the GPU instruction count each prediction consumes and output a file ending with STRESSFUL_INSTRUCTION.json. Note: This command could only accept one single prediction file ending with PASSED_SOLUTIONS.json. 4. Calculating the efficient@1/speedup bash coffe eval <benchmark_name> <output_repo> -p <index_file>,<pred_file> -m instr_count -f <efficient_at_1/speedup> This command calculate the efficient@1 or speedup. Note: This command requires the index file and the instruction file as COFFE compares the performance of predictions with grouth truth solutions to calculate the metrics. ## STGen For details about the stressful test case generation approach STGen, please see stgen/. ## Cite If you use COFFE, please cite us: @misc{peng2025coffe, title={COFFE: A Code Efficiency Benchmark for Code Generation}, author={Yun Peng and Jun Wan and Yichen Li and Xiaoxue Ren}, year={2025}, eprint={2502.02827}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2502.02827}, }

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

Educational Grade School Math (EGSM) contains 2,093 question/answer pairs generated by MATHWELL, a reference-free educational grade school math word problem generator that outputs a word problem and Program of Thought (PoT) solution based solely on an optional student interest, as introduced in MATHWELL: Generating Educational Math Word Problems Using Teacher Annotations. The question/answer pairs are verified by human experts. EGSM is the first teacher-annotated math word problem training dataset for LLMs. EGSM is designed to train reference-free educational grade school math word problem generators, but can also be used to train math QA models. Please refer to our paper for more information on the dataset.

This is an assembly dataset built on top of Shellcode_IA32, a dataset for automatically generating assembly from natural language descriptions that consists of 3,200 assembly instructions, commented in the English language, which were collected from shellcodes for IA-32 and written for the Netwide Assembler (NASM) for Linux. In order to make the data more representative of the code that we aim to generate (i.e., complete exploits, inclusive of decoders to be delivered in the shellcode), we enriched the dataset with further samples of assembly code, drawn from the exploits that we collected from public databases. Different from the previous dataset, the new one includes assembly code from real decoders used in actual exploits. The final dataset contains 3,715 unique pairs of assembly code snippets/English intents. To better support developers in the automatic generation of the assembly programs, we looked beyond a one-to-one mapping between natural language intents and their corresponding code. Therefore, the dataset includes 783 lines (~21% of the dataset) of multi-line snippets, i.e., intents that generate multiple lines of assembly code, separated by the newline character (\n). These multi-line snippets contain a number of different assembly instructions that can range between 2 and 5.

This dataset contains samples to generate Python code for security exploits. In order to make the dataset representative of real exploits, it includes code snippets drawn from exploits from public databases. Differing from general-purpose Python code found in previous datasets, the Python code of real exploits entails low-level operations on byte data for obfuscation purposes (i.e., to encode shellcodes). Therefore, real exploits make extensive use of Python instructions for converting data between different encoders, for performing low-level arithmetic and logical operations, and for bit-level slicing, which cannot be found in the previous general-purpose Python datasets. In total, we built a dataset that consists of 1,114 original samples of exploit-tailored Python snippets and their corresponding intent in the English language. These samples include complex and nested instructions, as typical of Python programming. In order to perform more realistic training and for a fair evaluation, we left untouched the developers' original code snippets and did not decompose them. We provided English intents to describe nested instructions altogether. In order to bootstrap the training process for the NMT model, we include in our dataset both the original, exploit-oriented snippets and snippets from a previous general-purpose Python dataset. This enables the NMT model to generate code that can mix general-purpose and exploit-oriented instructions. Among the several datasets for Python code generation, we choose the Django dataset due to its large size. This corpus contains 14,426 unique pairs of Python statements from the Django Web application framework and their corresponding description in English. Therefore, our final dataset contains 15,540 unique pairs of Python code snippets alongside their intents in natural language.

We introduce FixEval , a dataset for competitive programming bug fixing along with a comprehensive test suite and show the necessity of execution based evaluation compared to suboptimal match based evaluation metrics like BLEU, CodeBLEU, Syntax Match, Exact Match etc.

This dataset contains: (1) Slforge Generated Simulink Models : Synthetic Simulink Models (2) Source of Real World Simulink Models The .txt file is a combined text file that contains all the real world Simulink models based on SLGPT's experimental setup.

the FloCo dataset that contains 11,884 flowchart images and their corresponding Python codes.

GenoTEX (Genomics Data Automatic Exploration Benchmark) is a benchmark dataset for the automated analysis of gene expression data to identify disease-associated genes while considering the influence of other biological factors. It provides analysis code and results for solving a wide range of gene-trait association (GTA) analysis problems, encompassing dataset selection, preprocessing, and statistical analysis, in a pipeline that follows computational genomics standards. The benchmark includes expert-curated annotations from bioinformaticians to ensure accuracy and reliability.

The Java dataset introduced in Hybrid-DeepCom (Deep code comment generation with hybrid lexical and syntactical information), commonly used to evaluate automated code summarization. It is basically a further version of DeepCom-Java.

🤗 MigrationBench is a large-scale code migration benchmark dataset at the repository level, across multiple programming languages. - Current and initial release includes java 8 repositories with the maven build system, as of May 2025. It has 3 datasets: 1. 🤗 migration-bench-java-full has 5,102 repos, and each of them has a test directory or at least one test case. 2. 🤗 migration-bench-java-selected is a subset of migration-bench-java-full, with 300 repos. 3. 🤗 migration-bench-java-utg contains 4,184 repos, complementary to migration-bench-java-full.

🤗 MigrationBench is a large-scale code migration benchmark dataset at the repository level, across multiple programming languages. - Current and initial release includes java 8 repositories with the maven build system, as of May 2025. It has 3 datasets: 1. 🤗 migration-bench-java-full has 5,102 repos, and each of them has a test directory or at least one test case. 2. 🤗 migration-bench-java-selected is a subset of migration-bench-java-full, with 300 repos. 3. 🤗 migration-bench-java-utg contains 4,184 repos, complementary to migration-bench-java-full.

🤗 MigrationBench is a large-scale code migration benchmark dataset at the repository level, across multiple programming languages. - Current and initial release includes java 8 repositories with the maven build system, as of May 2025. It has 3 datasets: 1. 🤗 migration-bench-java-full has 5,102 repos, and each of them has a test directory or at least one test case. 2. 🤗 migration-bench-java-selected is a subset of migration-bench-java-full, with 300 repos. 3. 🤗 migration-bench-java-utg contains 4,184 repos, complementary to migration-bench-java-full.

Inspired by Wang et al. 2021, we decided to utilize the top-voted and well-documented Kaggle notebooks to construct the notebookCDGdataset We collected the top 10% highly-voted notebooks from the top 20 popular competitions on Kaggle (e.g. Titanic). We checked the data policy of each of the 20 competitions, none of them has copyright issues. We also contacted the Kaggle administrators to make sure our data collection complies with the platform’s policy. In total, we collected 3,944 notebooks as raw data. After data preprocessing, the final dataset contains 2,476 notebooks out of the 3,944 notebooks from the raw data. It has 28,625 code–documentation pairs. The overall code-to-markdown ratio is 2.2195 Download notebookCDG dataset

Source code to obfuscated code dataset in C, C++, Go, Java, Python, Rust and TypeScript.

A human-refined dataset of OpenAPI definitions based on the APIs.guru OpenAPI directory. The dataset was collected from the APIs.guru OpenAPI definitions directory. The directory contains more than 4,000 definitions in yaml format. Analysis of the repository revealed that about 75% of the definitions in the directory are produced by a handful of major companies like Amazon, Google, and Microsoft. To avoid the dataset bias towards a specific producer, the maximum number of definitions from a single producer was limited to 20. Multiple versions of the same API were also excluded from the dataset as they are likely to contain very similar definitions.

Recent advancements in large language models (LLMs) have showcased their exceptional abilities across various tasks, such as code generation, problem-solving and reasoning. Existing benchmarks evaluate tasks in isolation, yet the extent to which LLMs can understand prose-style tasks, identify the underlying problems, and then generate appropriate code solutions is still unexplored. Addressing this gap, we introduce PECC, a novel benchmark derived from Advent Of Code (AoC) challenges and Project Euler, including 2396 problems. Unlike conventional benchmarks, PECC requires LLMs to interpret narrative-embedded problems, extract requirements, and generate executable code. A key feature of our dataset is the complexity added by natural language prompting in chat-based evaluations, mirroring real-world instruction ambiguities. Results show varying model performance between narrative and neutral problems, with specific challenges in the Euler math-based subset with GPT-3.5-Turbo passing 50% of the AoC challenges and only 8% on the Euler problems. By probing the limits of LLMs' capabilities, our benchmark provides a framework to monitor and assess the subsequent progress of LLMs as a universal problem solver.

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

RES-Q is a natural language instruction-based benchmark for evaluating $\textbf{R}$epository $\textbf{E}$diting $\textbf{S}$ystems, which consists of 100 handcrafted repository editing tasks derived from real GitHub commits. Given an edit instruction and a code repository, RES-Q evaluates an LLM system’s ability to interpret edit instructions, gather information, and construct appropriate edits to the repository.

SOEVAL is created by us by mining questions from StackOverflow. Our goal was to create a prompt dataset that reflects the real-life needs of software developers. To build this dataset, we first collected 500 popular and recent questions with Python and Java tags for each. From these 1,000 questions, we applied a set of inclusion and exclusion criteria. The inclusion criteria were: the question has to (1) explicitly ask “how to do X” in Python or Java; (2) include code in its body; (3) have an accepted answer that includes code. We excluded questions that were (1) open-ended and asking for best practices/guidelines for a specific problem in Python/Java; (2) related to finding a specific API/module for a given task; (3) related to errors due to environment configuration (e.g., missing dependency library); (4) related to configuring libraries/API; (5) syntax specific types of questions. By applying the criteria above to these 1K questions, we obtained 28 and 42 prompts for Java and Python, respectively.

TACO (Topics in Algorithmic Code generation dataset) is a dataset focused on algorithmic code generation, designed to provide a more challenging training dataset and evaluation benchmark for the code generation model field. The dataset consists of programming competition problems that are more difficult and closer to real programming scenarios. It emphasizes improving or evaluating the model's understanding and reasoning abilities in practical application scenarios, rather than just implementing predefined function functionalities.

The dataset contains more than 100k code patch pairs extracted from open source projects on GitHub. Each pair comes with the erroneous and the fixed version of the corresponding code snippet. Instead of the whole file, the code snippets are extracted to focus on the problematic region (error line + other lines around it). For each sample, the repository name, the commit id, and the file names are provided so that one can access the complete files in case of interest. The dataset only has JavaScript programs and the error are detected by the popular static code analyzer ESLint. The dataset can be used in the fields of: program repair, code generation, bug finding, transfer learning and many more fields related to machine learning for code

$\textbf{Turbulence}$ is a new benchmark for systematically evaluating the correctness and robustness of instruction-tuned large language models (LLMs) for code generation. Turbulence consists of a large set of natural language question templates, each of which is a programming problem, parameterised so that it can be asked in many different forms. Each question template has an associated test oracle that judges whether a code solution returned by an LLM is correct. Thus, from a single question template, it is possible to ask an LLM a $\textit{neighbourhood}$ of very similar programming questions, and assess the correctness of the result returned for each question. This new benchmark systematically and automatically identifies cases where LLMs are able to solve some problems in a neighbourhood but do not manage to generalise to solve the whole neighbourhood. Therefore, this method is effective at highlighting robustness issues.