Attention Learning is Needed to Efficiently Learn Parity Function
원문 초록
Transformers, with their attention mechanisms, have emerged as the state-of-the-art architectures of sequential modeling and empirically outperform feed-forward neural networks (FFNNs) across many fields, such as natural language processing and computer vision. However, their generalization ability, particularly for low-sensitivity functions, remains less studied. We bridge this gap by analyzing transformers on the $k$-parity problem. Daniely and Malach (NeurIPS 2020) show that FFNNs with one hidden layer and $O(nk^7 \log k)$ parameters can learn $k$-parity, where the input length $n$ is typically much larger than $k$. In this paper, we prove that FFNNs require at least $\Omega(n)$ parameters to learn $k$-parity, while transformers require only $O(k)$ parameters, surpassing the theoretical lower bound needed by FFNNs. We further prove that this parameter efficiency cannot be achieved with fixed attention heads. Our work establishes transformers as theoretically superior to FFNNs in learning parity function, showing how their attention mechanisms enable parameter-efficient generalization in functions with low sensitivity.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Generative Latent Kernel Modeling for Blind Motion Deblurring
Differentially Private Relational Learning with Entity-level Privacy Guarantees
EtiCor++: Towards Understanding Etiquettical Bias in LLMs
Fragility in Average Treatment Effect on the Treated under Limited Covariate Support
Assessing parameter identifiability of a hemodynamics PDE model using spectral surrogates and dimension reduction
ACCESS DENIED INC: The First Benchmark Environment for Sensitivity Awareness
토론
이 논문과 연결된 포럼 글이 아직 없습니다.