Test perplexity: 107.95Params: 14.9M
Papers with Code · CC BY-SA 4.0
Language Modelling
A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.
Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.
Source: Wikipedia
논문
Validation perplexity: 77.9Test perplexity: 75.2
Bit per Character (BPC): 1.175Number of params: 13.8M
Test perplexity: 78.93
Bit per Character (BPC): 1.190Number of params: 27M
Bit per Character (BPC): 1.083Number of params: 24M
Test perplexity: 64.0Params: 25M
Test perplexity: 84.38
Validation perplexity: 82.2Test perplexity: 78.4
Bit per Character (BPC): 1.160Number of params: 10.7M
Bit per Character (BPC): 1.18
Validation perplexity: 60.8Test perplexity: 58.6Params: 24M
Bit per Character (BPC): 1.219Number of params: 14.4M
Bit per Character (BPC): 1.169Number of params: 13.8M
Bit per Character (BPC): 1.19
Bit per Character (BPC): 1.03
Validation perplexity: 67.9Test perplexity: 65.4Params: 23M
Bit per Character (BPC): 1.147Number of params: 16.0M
Bit per Character (BPC): 1.158Number of params: 13.4M
Validation perplexity: 68.1Test perplexity: 66.0