CodeXGLUE is a benchmark dataset and open challenge for code intelligence. It includes a collection of code intelligence tasks and a platform for model evaluation and comparison. CodeXGLUE stands for General Language Understanding Evaluation benchmark for CODE. It includes 14 datasets for 10 diversified code intelligence tasks covering the following scenarios: - code-code (clone detection, defect detection, cloze test, code completion, code repair, and code-to-code translation) - text-code (natural language code search, text-to-code generation) - code-text (code summarization) - text-text (documentation translation) A brief summary of CodeXGLUE is provided in the figure, including tasks, datasets, language, sizes in various states, baseline systems, providers, and short definitions of each task. Datasets highlighted in BLUE are newly introduced. Image source: https://github.com/microsoft/CodeXGLUE
In everyday language processing, sentence context affects how readers and listeners process upcoming words. In experimental situations, it can be useful to identify words that are predicted to greater or lesser degrees by the preceding context. Here we report completion norms for 3085 English sentences, collected online using a written cloze procedure in which participants were asked to provide their best guess for the word completing a sentence. Sentences varied between 8–10 words in length. At least 100 unique participants contributed to each sentence. All responses were reviewed by human raters to mitigate the influence of mis-spellings and typographical errors. The responses provide a range of predictability values for 13,438 unique target words, 6,790 of which appear in more than one sentence context. We also provide entropy values based on the relative predictability of multiple responses. Finally, we provide the code used to collate and organize the responses to facilitate additional analyses and future research projects.