arXiv:2502.03327
Is In-Context Universality Enough? MLPs are Also Universal In-Context
원문 초록
The success of transformers is often linked to their ability to perform in-context learning. Recent work shows that transformers are universal in context, capable of approximating any real-valued continuous function of a context (a probability measure over $\mathcal{X}\subseteq \mathbb{R}^d$) and a query $x\in \mathcal{X}$. This raises the question: Does in-context universality explain their advantage over classical models? We answer this in the negative by proving that MLPs with trainable activation functions are also universal in-context. This suggests the transformer's success is likely due to other factors like inductive bias or training stability.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Relational reasoning and inductive bias in transformers trained on a transitive inference task
Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex
RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selection
BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models
Assay2Mol: large language model-based drug design using BioAssay context
Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts
토론
이 논문과 연결된 포럼 글이 아직 없습니다.