CodeXGLUE is a benchmark dataset and open challenge for code intelligence. It includes a collection of code intelligence tasks and a platform for model evaluation and comparison. CodeXGLUE stands for General Language Understanding Evaluation benchmark for CODE. It includes 14 datasets for 10 diversified code intelligence tasks covering the following scenarios: - code-code (clone detection, defect detection, cloze test, code completion, code repair, and code-to-code translation) - text-code (natural language code search, text-to-code generation) - code-text (code summarization) - text-text (documentation translation) A brief summary of CodeXGLUE is provided in the figure, including tasks, datasets, language, sizes in various states, baseline systems, providers, and short definitions of each task. Datasets highlighted in BLUE are newly introduced. Image source: https://github.com/microsoft/CodeXGLUE
CodeContests is a competitive programming dataset for machine-learning. This dataset was used when training AlphaCode. It consists of programming problems, from a variety of sources. Problems include test cases in the form of paired inputs and outputs, as well as both correct and incorrect human solutions in a variety of languages.
PyTorrent contains 218,814 Python package libraries from PyPI and Anaconda environment. This is because earlier studies have shown that much of the code is redundant and Python packages from these environments are better in quality and are well-documented. PyTorrent enables users (such as data scientists, students, etc.) to build off the shelf machine learning models directly without spending months of effort on large infrastructure.
Syntax-Aware Fill-in-the-Middle (SAFIM) is a benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task. SAFIM has three subtasks: Algorithmic Block Completion, Control-Flow Expression Completion, and API Function Call Completion. SAFIM is sourced from code submitted from April 2022 to January 2023 to minimize the impact of data contamination on evaluation results. - Authors: Linyuan Gong, Sida Wang, Mostafa Elhoushi, Alvin Cheung - Paper: https://arxiv.org/abs/2403.04814 - Huggingface Dataset: https://huggingface.co/datasets/gonglinyuan/safim - Leaderboard: https://safimbenchmark.com - Code & Submission Instructions: https://github.com/gonglinyuan/safim The SAFIM benchmark is partially derived from problem descriptions and code solutions from https://codeforces.com. According to the license of CodeForces, you may publish the texts of Codeforces problems in any open sources, but you must preserve a direct link to the site.
📚 BlendNet The dataset contains $12k$ samples. To balance cost savings with data quality and scale, we manually annotated $2k$ samples and used GPT-4o to annotate the remaining $10k$ samples. For more details, please visit our GitHub repository
📚 CADBench CADBench is a comprehensive benchmark to evaluate the ability of LLMs to generate CAD scripts. It contains 500 simulated data samples and 200 data samples collected from online forums. For more details, please visit our GitHub repository
To assess a model’s ability to create microcontroller-driven electronic devices, we developed a benchmark, MICRO25, that includes 25 tasks intended for the common ARDUINO microcontroller ecosystem.. These tasks, shown in Table 2, span 5 core categories including: input, interface protocols, output, sensors, and logic. Each task is either tailored to test a specific fundamental competency required to build basic microcontroller-driven electronic devices, or the integration of several competencies into larger design flows.
RES-Q is a natural language instruction-based benchmark for evaluating $\textbf{R}$epository $\textbf{E}$diting $\textbf{S}$ystems, which consists of 100 handcrafted repository editing tasks derived from real GitHub commits. Given an edit instruction and a code repository, RES-Q evaluates an LLM system’s ability to interpret edit instructions, gather information, and construct appropriate edits to the repository.
Verireason-RTL-Coder_7b_reasoning_tb For implementation details, visit our GitHub repository: VeriReason Check out our paper: VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation ## Update Log 2025.05.17: Initial release of Nellyw888/Verireason-RTL-Coder_7b_reasoning_tb ## Project Description This study introduces VeriReason, a novel approach utilizing reinforcement learning with testbench feedback to enhance the performance of pre-trained models for Verilog RTL code generation. VeriReason combines supervised fine-tuning with Guided Reward Proximal Optimization (GRPO) reinforcement learning, specifically tailored for RTL code generation. Using our curated high-quality training examples alongside a feedback-driven reward model, VeriReason achieves 83.1% functional correctness on the VerilogEval Machine benchmark, substantially outperforming both comparable-sized models and much larger commercial systems like GPT-4 Turbo. The model integrates explicit reasoning capabilities with reinforcement learning for Verilog generation, establishing a new state-of-the-art for automated RTL synthesis. Our 7B parameter model based on Code Llama demonstrates up to a 2.8× increase in first-attempt functional correctness compared to baseline methods and exhibits robust generalization to unseen designs. ## Installation To install this project, follow these steps: 1. Clone the repository: git clone https://github.com/NellyW8/VeriReason.git 2. Navigate to the project directory: cd VeriReason 3. Install the dependencies as specified in the repository ## Dataset Summary VeriReason-RTL-Coder_7b_reasoning_tb is a high-quality dataset for training and evaluating models for Verilog Register-Transfer Level (RTL) code generation with explicit reasoning. The dataset is a curated and enhanced version of the original RTLCoder dataset, specifically designed to improve model reasoning capabilities and functional correctness in RTL code generation. ## Dataset Creation ### Source Data - Base Dataset: Built and selected upon the RTLCoder dataset - Enhancements: - Added explicit reasoning steps using GPT-4.1 - Improved code quality to better follow instructions - Included testbenches generated with GPT-4.1 - Incorporated simulation results from running testbenches with the generated code ## Dataset Structure ### Data Instances Each instance in the dataset contains: - id: Unique identifier for the example - instruction: Problem specification for RTL design - output: Generated Verilog code solution - tb: Testbench code for validation - tb_result: Results from running the testbench on the solution ### Data Fields - instruction: String containing the RTL design problem statement (average length: ~3,973 characters) - output: String containing the Verilog code solution (average length: ~2,024 characters) - tb: String containing the testbench code (average length: ~1,551-12,700 characters) - tb_result: String containing simulation results (average length: ~5,800 characters) ## Usage Example To use this dataset for model training: python from datasets import load_dataset # Load the dataset dataset = load_dataset("Nellyw888/Verireason-RTL-Coder_7b_reasoning_tb") # Example of accessing an entry example = dataset['train'][0] instruction = example['instruction'] reasoning_and_solution = example['output'] testbench = example['tb'] test_results = example['tb_result'] ## Models Trained Using This Dataset - Nellyw888/VeriReason-Qwen2.5-7b-SFT-Reasoning - A 7B parameter model based on Qwen2.5-Coder-7B-Instruct, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. - ## Citation Please cite our paper if you use our model or dataset: bibtex @misc{wang2025verireason, title={VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation}, author={Yiting Wang and Guoheng Sun and Wanghao Ye and Gang Qu and Ang Li}, year={2025}, eprint={2505.11849}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2505.11849}, }
Verireason-RTL-Coder_7b_reasoning_tb_simple For implementation details, visit our GitHub repository: VeriReason Check out our paper: VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation ## Update Log 2025.05.17: Initial release of Nellyw888/Verireason-RTL-Coder_7b_reasoning_tb_simple ## Project Description This study introduces VeriReason, a novel approach utilizing reinforcement learning with testbench feedback to enhance the performance of pre-trained models for Verilog RTL code generation. VeriReason combines supervised fine-tuning with Guided Reward Proximal Optimization (GRPO) reinforcement learning, specifically tailored for RTL code generation. Using our curated high-quality training examples alongside a feedback-driven reward model, VeriReason achieves 83.1% functional correctness on the VerilogEval Machine benchmark, substantially outperforming both comparable-sized models and much larger commercial systems like GPT-4 Turbo. The model integrates explicit reasoning capabilities with reinforcement learning for Verilog generation, establishing a new state-of-the-art for automated RTL synthesis. Our 7B parameter model based on Code Llama demonstrates up to a 2.8× increase in first-attempt functional correctness compared to baseline methods and exhibits robust generalization to unseen designs. ## Installation To install this project, follow these steps: 1. Clone the repository: git clone https://github.com/NellyW8/VeriReason.git 2. Navigate to the project directory: cd VeriReason 3. Install the dependencies as specified in the repository ## Dataset Summary VeriReason-RTL-Coder_7b_reasoning_tb is a high-quality dataset for training and evaluating models for Verilog Register-Transfer Level (RTL) code generation with explicit reasoning. The dataset is a curated and enhanced version of the original RTLCoder dataset, specifically designed to improve model reasoning capabilities and functional correctness in RTL code generation. ## Dataset Creation ### Source Data - Base Dataset: Built and selected upon the RTLCoder dataset - Enhancements: - Added explicit reasoning steps using GPT-4.1 - Improved code quality to better follow instructions - Included testbenches generated with GPT-4.1 - Incorporated simulation results from running testbenches with the generated code ## Dataset Structure ### Data Instances Each instance in the dataset contains: - id: Unique identifier for the example - instruction: Problem specification for RTL design - output: Generated Verilog code solution - tb: Testbench code for validation - tb_result: Results from running the testbench on the solution ### Data Fields - instruction: String containing the RTL design problem statement (average length: ~3,973 characters) - output: String containing the Verilog code solution (average length: ~2,024 characters) - tb: String containing the testbench code (average length: ~1,551-12,700 characters) - tb_result: String containing simulation results (average length: ~5,800 characters) ## Usage Example To use this dataset for model training: python from datasets import load_dataset # Load the dataset dataset = load_dataset("Nellyw888/Verireason-RTL-Coder_7b_reasoning_tb_simple") # Example of accessing an entry example = dataset['train'][0] instruction = example['instruction'] reasoning_and_solution = example['output'] testbench = example['tb'] test_results = example['tb_result'] ## Models Trained Using This Dataset - Nellyw888/VeriReason-Qwen2.5-7b-SFT-Reasoning - A 7B parameter model based on Qwen2.5-Coder-7B-Instruct, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. - Nellyw888/VeriReason-Qwen2.5-3B-Verilog-RTL-GRPO-reasoning-tb - A 3B parameter model based on Qwen2.5-Coder-3B-Instruct, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. - Nellyw888/VeriReason-Qwen2.5-1.5B-grpo-small - A 1.5B parameter model based on Qwen2.5-Coder-1.5B-Instruct, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. - Nellyw888/VeriReason-Llama-7b-RTLCoder-GRPO-reasoning-tb - A 7B parameter model based on codellama/CodeLlama-7b-Instruct-hf, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. ## Citation Please cite our paper if you use our model: bibtex @misc{wang2025verireason, title={VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation}, author={Yiting Wang and Guoheng Sun and Wanghao Ye and Gang Qu and Ang Li}, year={2025}, eprint={2505.11849}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2505.11849}, }