Introductory Pass@1: 20Interview Pass@1: 13.5Competition Pass@1: 33.3
Papers with Code · CC BY-SA 4.0
Code Generation
Code Generation is an important field to predict explicit code or program structure from multimodal data sources such as incomplete code, programs in another programming language, natural language descriptions or execution examples. Code Generation tools can assist the development of automatic programming tools to improve programming productivity.
<span class="description-source">Source: Deep Learning for Source Code Modeling and Generation </span>
Image source: Measuring Coding Challenge Competence With APPS
논문
Introductory Pass@1: 68.44Interview Pass@1: 44.49Competition Pass@1: 27.84
Introductory Pass@1: 29.3%Interview Pass@1: 6.4%Competition Pass@1: 2.5%Competition Pass@any: 14.5%
Introductory Pass@1: 26.04Interview Pass@1: 4.21Competition Pass@1: 0.81
Introductory Pass@1: 47.3%Interview Pass@1: 14.3%Competition Pass@1: 6.2%
Competition Pass@any: 22.0
Introductory Pass@1: 1.30%Interview Pass@1: 0.70%Competition Pass@1: 0.00%Introductory Pass@1000: 25.00%
Introductory Pass@1: 87.2Interview Pass@1: 65.2Competition Pass@1: 34.8Pass@1: 62.6
Introductory Pass@1: 33.80Interview Pass@1: 19.70Competition Pass@1: 11.09
Introductory Pass@1: 5.60%Interview Pass@1: 1.00%Competition Pass@1: 0.50%Introductory Pass@1000: 35.20%
Test Set pass@1: 58.18
Java/EM: -Python/EM: -Java/BLEU: -Python/BLEU: 22.3
Introductory Pass@1: 3.90%Interview Pass@1: 0.57%Competition Pass@1: 0.00%Introductory Pass@1000: 27.90%
Java/EM: 10.21Python/EM: 9.61Java/BLEU: 28.98Python/BLEU: 24.43
easy pass@1: 9.32%
BLEU: 67.4CodeBLEU: 75.7
Test Set pass@1: 1.11Test Set pass@5: 3.18Val Set pass@1: 1.98Val Set pass@5: 3.27