On Predictive Information in RNNs
원문 초록
Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work, we investigate if the same can be said of artificial neurons in recurrent neural networks (RNNs) trained with maximum likelihood. Empirically, we find that RNNs are suboptimal in the information plane. Instead of optimally compressing past information, they extract additional information that is not relevant for predicting the future. We show that constraining past information by injecting noise into the hidden state can improve RNNs in several ways: optimality in the predictive information plane, sample quality, heldout likelihood, and downstream classification performance.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Cauchy-Schwarz Divergence Information Bottleneck for Regression
End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training
HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
Information flows of diverse autoencoders
Malicious Network Traffic Detection via Deep Learning: An Information Theoretic View
The Dual Information Bottleneck
토론
이 논문과 연결된 포럼 글이 아직 없습니다.