Verified Smart Contracts Code Comments is a dataset of real Ethereum smart contract functions, containing "code, comment" pairs of both Solidity and Vyper source code. The dataset is based on every deployed Ethereum smart contract as of 1st of April 2022, whose been verified on Etherscan and has a least one transaction. A total of 1,541,370 smart contract functions are provided, parsed from 186,397 unique smart contracts, filtered down from 2,217,692 smart contracts.

Verified Smart Contracts is a dataset of real Ethereum smart contracts, containing both Solidity and Vyper source code. It consists of every deployed Ethereum smart contract as of 1st of April 2022, whose been verified on Etherscan and has a least one transaction. A total of 186,397 unique smart contracts are provided, filtered down from 2,217,692 smart contracts. The dataset contains 53,843,305 lines of code.

Verireason-RTL-Coder_7b_reasoning_tb For implementation details, visit our GitHub repository: VeriReason Check out our paper: VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation ## Update Log 2025.05.17: Initial release of Nellyw888/Verireason-RTL-Coder_7b_reasoning_tb ## Project Description This study introduces VeriReason, a novel approach utilizing reinforcement learning with testbench feedback to enhance the performance of pre-trained models for Verilog RTL code generation. VeriReason combines supervised fine-tuning with Guided Reward Proximal Optimization (GRPO) reinforcement learning, specifically tailored for RTL code generation. Using our curated high-quality training examples alongside a feedback-driven reward model, VeriReason achieves 83.1% functional correctness on the VerilogEval Machine benchmark, substantially outperforming both comparable-sized models and much larger commercial systems like GPT-4 Turbo. The model integrates explicit reasoning capabilities with reinforcement learning for Verilog generation, establishing a new state-of-the-art for automated RTL synthesis. Our 7B parameter model based on Code Llama demonstrates up to a 2.8× increase in first-attempt functional correctness compared to baseline methods and exhibits robust generalization to unseen designs. ## Installation To install this project, follow these steps: 1. Clone the repository: git clone https://github.com/NellyW8/VeriReason.git 2. Navigate to the project directory: cd VeriReason 3. Install the dependencies as specified in the repository ## Dataset Summary VeriReason-RTL-Coder_7b_reasoning_tb is a high-quality dataset for training and evaluating models for Verilog Register-Transfer Level (RTL) code generation with explicit reasoning. The dataset is a curated and enhanced version of the original RTLCoder dataset, specifically designed to improve model reasoning capabilities and functional correctness in RTL code generation. ## Dataset Creation ### Source Data - Base Dataset: Built and selected upon the RTLCoder dataset - Enhancements: - Added explicit reasoning steps using GPT-4.1 - Improved code quality to better follow instructions - Included testbenches generated with GPT-4.1 - Incorporated simulation results from running testbenches with the generated code ## Dataset Structure ### Data Instances Each instance in the dataset contains: - id: Unique identifier for the example - instruction: Problem specification for RTL design - output: Generated Verilog code solution - tb: Testbench code for validation - tb_result: Results from running the testbench on the solution ### Data Fields - instruction: String containing the RTL design problem statement (average length: ~3,973 characters) - output: String containing the Verilog code solution (average length: ~2,024 characters) - tb: String containing the testbench code (average length: ~1,551-12,700 characters) - tb_result: String containing simulation results (average length: ~5,800 characters) ## Usage Example To use this dataset for model training: python from datasets import load_dataset # Load the dataset dataset = load_dataset("Nellyw888/Verireason-RTL-Coder_7b_reasoning_tb") # Example of accessing an entry example = dataset['train'][0] instruction = example['instruction'] reasoning_and_solution = example['output'] testbench = example['tb'] test_results = example['tb_result'] ## Models Trained Using This Dataset - Nellyw888/VeriReason-Qwen2.5-7b-SFT-Reasoning - A 7B parameter model based on Qwen2.5-Coder-7B-Instruct, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. - ## Citation Please cite our paper if you use our model or dataset: bibtex @misc{wang2025verireason, title={VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation}, author={Yiting Wang and Guoheng Sun and Wanghao Ye and Gang Qu and Ang Li}, year={2025}, eprint={2505.11849}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2505.11849}, }

Verireason-RTL-Coder_7b_reasoning_tb_simple For implementation details, visit our GitHub repository: VeriReason Check out our paper: VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation ## Update Log 2025.05.17: Initial release of Nellyw888/Verireason-RTL-Coder_7b_reasoning_tb_simple ## Project Description This study introduces VeriReason, a novel approach utilizing reinforcement learning with testbench feedback to enhance the performance of pre-trained models for Verilog RTL code generation. VeriReason combines supervised fine-tuning with Guided Reward Proximal Optimization (GRPO) reinforcement learning, specifically tailored for RTL code generation. Using our curated high-quality training examples alongside a feedback-driven reward model, VeriReason achieves 83.1% functional correctness on the VerilogEval Machine benchmark, substantially outperforming both comparable-sized models and much larger commercial systems like GPT-4 Turbo. The model integrates explicit reasoning capabilities with reinforcement learning for Verilog generation, establishing a new state-of-the-art for automated RTL synthesis. Our 7B parameter model based on Code Llama demonstrates up to a 2.8× increase in first-attempt functional correctness compared to baseline methods and exhibits robust generalization to unseen designs. ## Installation To install this project, follow these steps: 1. Clone the repository: git clone https://github.com/NellyW8/VeriReason.git 2. Navigate to the project directory: cd VeriReason 3. Install the dependencies as specified in the repository ## Dataset Summary VeriReason-RTL-Coder_7b_reasoning_tb is a high-quality dataset for training and evaluating models for Verilog Register-Transfer Level (RTL) code generation with explicit reasoning. The dataset is a curated and enhanced version of the original RTLCoder dataset, specifically designed to improve model reasoning capabilities and functional correctness in RTL code generation. ## Dataset Creation ### Source Data - Base Dataset: Built and selected upon the RTLCoder dataset - Enhancements: - Added explicit reasoning steps using GPT-4.1 - Improved code quality to better follow instructions - Included testbenches generated with GPT-4.1 - Incorporated simulation results from running testbenches with the generated code ## Dataset Structure ### Data Instances Each instance in the dataset contains: - id: Unique identifier for the example - instruction: Problem specification for RTL design - output: Generated Verilog code solution - tb: Testbench code for validation - tb_result: Results from running the testbench on the solution ### Data Fields - instruction: String containing the RTL design problem statement (average length: ~3,973 characters) - output: String containing the Verilog code solution (average length: ~2,024 characters) - tb: String containing the testbench code (average length: ~1,551-12,700 characters) - tb_result: String containing simulation results (average length: ~5,800 characters) ## Usage Example To use this dataset for model training: python from datasets import load_dataset # Load the dataset dataset = load_dataset("Nellyw888/Verireason-RTL-Coder_7b_reasoning_tb_simple") # Example of accessing an entry example = dataset['train'][0] instruction = example['instruction'] reasoning_and_solution = example['output'] testbench = example['tb'] test_results = example['tb_result'] ## Models Trained Using This Dataset - Nellyw888/VeriReason-Qwen2.5-7b-SFT-Reasoning - A 7B parameter model based on Qwen2.5-Coder-7B-Instruct, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. - Nellyw888/VeriReason-Qwen2.5-3B-Verilog-RTL-GRPO-reasoning-tb - A 3B parameter model based on Qwen2.5-Coder-3B-Instruct, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. - Nellyw888/VeriReason-Qwen2.5-1.5B-grpo-small - A 1.5B parameter model based on Qwen2.5-Coder-1.5B-Instruct, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. - Nellyw888/VeriReason-Llama-7b-RTLCoder-GRPO-reasoning-tb - A 7B parameter model based on codellama/CodeLlama-7b-Instruct-hf, fine-tuned using this dataset with both supervised learning and reinforcement learning techniques. ## Citation Please cite our paper if you use our model: bibtex @misc{wang2025verireason, title={VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation}, author={Yiting Wang and Guoheng Sun and Wanghao Ye and Gang Qu and Ang Li}, year={2025}, eprint={2505.11849}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2505.11849}, }

Test-driven benchmark to challenge LLMs to write long JavaScript React application GitHub Script

WebGen-Bench WebGen-Bench is created to benchmark LLM-based agent's ability to generate websites from scratch. The dataset is introduced in WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch. It contains 101 instructions and 647 test cases. It also has a training set of 6667 instructions, named WebGen-Instruct. The code for evaluation as well as the training code and data are released at WebGen-Bench (Github) ## Citation If you find our project useful, please cite: @misc{lu2025webgenbenchevaluatingllmsgenerating, title={WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch}, author={Zimu Lu and Yunqiao Yang and Houxing Ren and Haotian Hou and Han Xiao and Ke Wang and Weikang Shi and Aojun Zhou and Mingjie Zhan and Hongsheng Li}, year={2025}, eprint={2505.03733}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2505.03733}, }