CodeXGLUE is a benchmark dataset and open challenge for code intelligence. It includes a collection of code intelligence tasks and a platform for model evaluation and comparison. CodeXGLUE stands for General Language Understanding Evaluation benchmark for CODE. It includes 14 datasets for 10 diversified code intelligence tasks covering the following scenarios: - code-code (clone detection, defect detection, cloze test, code completion, code repair, and code-to-code translation) - text-code (natural language code search, text-to-code generation) - code-text (code summarization) - text-text (documentation translation) A brief summary of CodeXGLUE is provided in the figure, including tasks, datasets, language, sizes in various states, baseline systems, providers, and short definitions of each task. Datasets highlighted in BLUE are newly introduced. Image source: https://github.com/microsoft/CodeXGLUE

CriticBench is a comprehensive benchmark designed to assess the abilities of Large Language Models (LLMs) to critique and rectify their reasoning across various tasks. It encompasses five reasoning domains: 1. Mathematical 2. Commonsense 3. Symbolic 4. Coding 5. Algorithmic CriticBench compiles 15 datasets and incorporates responses from three LLM families. By utilizing CriticBench, researchers evaluate and dissect the performance of 17 LLMs in generation, critique, and correction reasoning (referred to as GQC reasoning). Notable findings include: 1. A linear relationship in GQC capabilities, with critique-focused training significantly enhancing performance. 2. Task-dependent variation in correction effectiveness, with logic-oriented tasks being more amenable to correction. 3. GQC knowledge inconsistencies that decrease as model size increases. 4. An intriguing inter-model critiquing dynamic, where stronger models excel at critiquing weaker ones, while weaker models surprisingly surpass stronger ones in self-critique. (1) CriticBench: Benchmarking LLMs for Critique-Correct Reasoning. https://arxiv.org/abs/2402.14809. (2) CriticBench: Benchmarking LLMs for Critique-Correct Reasoning. http://export.arxiv.org/abs/2402.14809. (3) CriticBench: Benchmarking LLMs for Critique-Correct Reasoning. https://openreview.net/forum?id=sc5i7q6DQO. (4) CriticBench: Benchmarking LLMs for Critique-Correct Reasoning - arXiv.org. https://arxiv.org/html/2402.14809v2. (5) undefined. https://doi.org/10.48550/arXiv.2402.14809.

PyTorrent contains 218,814 Python package libraries from PyPI and Anaconda environment. This is because earlier studies have shown that much of the code is redundant and Python packages from these environments are better in quality and are well-documented. PyTorrent enables users (such as data scientists, students, etc.) to build off the shelf machine learning models directly without spending months of effort on large infrastructure.

DISL The full dataset report is available at: https://arxiv.org/abs/2403.16861 The DISL dataset features a collection of 514, 506 unique Solidity files that have been deployed to Ethereum mainnet. It caters to the need for a large and diverse dataset of real-world smart contracts. DISL serves as a resource for developing machine learning systems and for benchmarking software engineering tools designed for smart contracts. - Curated by: Gabriele Morello - License: [MIT] ## Instructions to explore the dataset python from datasets import load_dataset # Load the raw dataset dataset = load_dataset("ASSERT-KTH/DISL", "raw") # OR # Load the decomposed dataset dataset = load_dataset("ASSERT-KTH/DISL", "decomposed") # number of rows and columns num_rows = len(dataset["train"]) num_columns = len(dataset["train"].column_names) # random row import random random_row = random.choice(dataset["train"]) # random source code random_sc = random.choice(dataset["train"])['source_code'] print(random_sc)

We introduce FixEval , a dataset for competitive programming bug fixing along with a comprehensive test suite and show the necessity of execution based evaluation compared to suboptimal match based evaluation metrics like BLEU, CodeBLEU, Syntax Match, Exact Match etc.

🤗 MigrationBench is a large-scale code migration benchmark dataset at the repository level, across multiple programming languages. - Current and initial release includes java 8 repositories with the maven build system, as of May 2025. It has 3 datasets: 1. 🤗 migration-bench-java-full has 5,102 repos, and each of them has a test directory or at least one test case. 2. 🤗 migration-bench-java-selected is a subset of migration-bench-java-full, with 300 repos. 3. 🤗 migration-bench-java-utg contains 4,184 repos, complementary to migration-bench-java-full.

🤗 MigrationBench is a large-scale code migration benchmark dataset at the repository level, across multiple programming languages. - Current and initial release includes java 8 repositories with the maven build system, as of May 2025. It has 3 datasets: 1. 🤗 migration-bench-java-full has 5,102 repos, and each of them has a test directory or at least one test case. 2. 🤗 migration-bench-java-selected is a subset of migration-bench-java-full, with 300 repos. 3. 🤗 migration-bench-java-utg contains 4,184 repos, complementary to migration-bench-java-full.

🤗 MigrationBench is a large-scale code migration benchmark dataset at the repository level, across multiple programming languages. - Current and initial release includes java 8 repositories with the maven build system, as of May 2025. It has 3 datasets: 1. 🤗 migration-bench-java-full has 5,102 repos, and each of them has a test directory or at least one test case. 2. 🤗 migration-bench-java-selected is a subset of migration-bench-java-full, with 300 repos. 3. 🤗 migration-bench-java-utg contains 4,184 repos, complementary to migration-bench-java-full.

PIE stands for Performance Improving Code Edits. PIE contains trajectories of programs, where a programmer begins with an initial, slower version and iteratively makes changes to improve the program’s performance. Source: Learning Performance-Improving Code Edits Image Source: https://arxiv.org/pdf/2302.07867v3.pdf