ClimateBert: A Pretrained Language Model for Climate-Related Text
원문 초록
Over the recent years, large pretrained language models (LM) have revolutionized the field of natural language processing (NLP). However, while pretraining on general language has been shown to work very well for common language, it has been observed that niche language poses problems. In particular, climate-related texts include specific language that common LMs can not represent accurately. We argue that this shortcoming of today's LMs limits the applicability of modern NLP to the broad field of text processing of climate-related texts. As a remedy, we propose CLIMATEBERT, a transformer-based language model that is further pretrained on over 2 million paragraphs of climate-related texts, crawled from various sources such as common news, research articles, and climate reporting of companies. We find that CLIMATEBERT leads to a 48% improvement on a masked language model objective which, in turn, leads to lowering error rates by 3.57% to 35.71% for various climate-related downstream tasks like text classification, sentiment analysis, and fact-checking.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Narrative Shift Detection: A Hybrid Approach of Dynamic Topic Models and Large Language Models
Intent Factored Generation: Unleashing the Diversity in Your Language Model
Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content
Quantum Graph Transformer for NLP Sentiment Classification
DIBERT: Dependency Injected Bidirectional Encoder Representations from Transformers
Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks
토론
이 논문과 연결된 포럼 글이 아직 없습니다.