Provable In-context Learning for Mixture of Linear Regressions using Transformers
원문 초록
We theoretically investigate the in-context learning capabilities of transformers in the context of learning mixtures of linear regression models. For the case of two mixtures, we demonstrate the existence of transformers that can achieve an accuracy, relative to the oracle predictor, of order $\mathcal{\tilde{O}}((d/n)^{1/4})$ in the low signal-to-noise ratio (SNR) regime and $\mathcal{\tilde{O}}(\sqrt{d/n})$ in the high SNR regime, where $n$ is the length of the prompt, and $d$ is the dimension of the problem. Additionally, we derive in-context excess risk bounds of order $\mathcal{O}(L/\sqrt{B})$, where $B$ denotes the number of (training) prompts, and $L$ represents the number of attention layers. The order of $L$ depends on whether the SNR is low or high. In the high SNR regime, we extend the results to $K$-component mixture models for finite $K$. Extensive simulations also highlight the advantages of transformers for this task, outperforming other baselines such as the Expectation-Maximization algorithm.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Assay2Mol: large language model-based drug design using BioAssay context
Universal pre-training by iterated random computation
Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Conquering the Retina: Bringing Visual in-Context Learning to OCT
Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context
PRO-V: An Efficient Program Generation Multi-Agent System for Automatic RTL Verification
토론
이 논문과 연결된 포럼 글이 아직 없습니다.