Perplexity: 12.45
Papers with Code · CC BY-SA 4.0
Language Modelling
A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.
Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.
Source: Wikipedia
논문
Steps: 1MTPUv3 Hours: 17.3KPerplexity: 12.35
Test perplexity: 16.61
Validation perplexity: 15.81Test perplexity: 15.79Number of params: 247M
Test perplexity: 11.33Number of params: 10000M
Test perplexity: 10.6Number of params: 2700M
Perplexity: 14.79
Validation perplexity: 13.11Test perplexity: 13.29
Bit per Character (BPC): 1.31
Bit per Character (BPC): 1.38
Validation perplexity: 15.8Test perplexity: 16.4Number of params: 257M
Test perplexity: 15.8
Test perplexity: 13.4Number of params: 125M
Bit per Character (BPC): 1.30
Test perplexity: 2.4Number of params: 7532M
Bit per Character (BPC): 1.01
Test perplexity: 10.81Number of params: 8300M
Bit per Character (BPC): 1.31Number of params: 5.9M
10-stage average accuracy: 10
Validation perplexity: 15.72Test perplexity: 15.5Number of params: 247M