On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition
원문 초록
We establish connections between the problem of learning a two-layer neural network and tensor decomposition. We consider a model with feature vectors $\boldsymbol x \in \mathbb R^d$, $r$ hidden units with weights $\{\boldsymbol w_i\}_{1\le i \le r}$ and output $y\in \mathbb R$, i.e., $y=\sum_{i=1}^r \sigma( \boldsymbol w_i^{\mathsf T}\boldsymbol x)$, with activation functions given by low-degree polynomials. In particular, if $\sigma(x) = a_0+a_1x+a_3x^3$, we prove that no polynomial-time learning algorithm can outperform the trivial predictor that assigns to each example the response variable $\mathbb E(y)$, when $d^{3/2}\ll r\ll d^2$. Our conclusion holds for a `natural data distribution', namely standard Gaussian feature vectors $\boldsymbol x$, and output distributed according to a two-layer neural network with random isotropic weights, and under a certain complexity-theoretic assumption on tensor decomposition. Roughly speaking, we assume that no polynomial-time algorithm can substantially outperform current methods for tensor decomposition based on the sum-of-squares hierarchy. We also prove generalizations of this statement for higher degree polynomial activations, and non-random weight vectors. Remarkably, several existing algorithms for learning two-layer networks with rigorous guarantees are based on tensor decomposition. Our results support the idea that this is indeed the core computational difficulty in learning such networks, under the stated generative model for the data. As a side result, we show that under this model learning the network requires accurate learning of its weights, a property that does not hold in a more general setting.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MultiHU-TD: Multifeature Hyperspectral Unmixing Based on Tensor Decomposition
Revisit CP Tensor Decomposition: Statistical Optimality and Fast Convergence
D-Tracker: Modeling Interest Diffusion in Social Activity Tensor Data Streams
Multiscale Tensor Summation Factorization as a New Neural Network Layer (MTS Layer) for Multidimensional Data Processing
Knowledge Graph Completion with Mixed Geometry Tensor Factorization
Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.