https://github.com/Liuhong99/Sophia
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.
A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.
Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.
Source: Wikipedia
관련 논문과 평가 결과에 연결된 구현입니다. 출처에서 확인된 저자 구현에는 공식 코드 표시가 있습니다.
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
VideoBERT: A Joint Model for Video and Language Representation Learning
Recurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning
Universal Language Model Fine-tuning for Text Classification
word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
Generalization through Memorization: Nearest Neighbor Language Models
LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
An empirical study of LLaMA3 quantization: from LLMs to MLLMs
OpenELM: An Efficient Language Model Family with Open Training and Inference Framework
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
XLNet: Generalized Autoregressive Pretraining for Language Understanding
출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.