Home / Events / Feature Learning in Two-layer Neural Networks
Applied and computational Analysis Seminars

Feature Learning in Two-layer Neural Networks

With Murat A. Erdogdu (University of Toronto)

Feature Learning in Two-layer Neural Networks: The Effect of Data Covariance

We study the effect of gradient-based optimization on feature learning in two-layer neural networks. We consider a setting
where the number of samples is of the same order as the input
dimension and show that, when the input data is isotropic, gradient descent always improves upon the initial random features model in terms of prediction risk, for a certain class of targets. Further leveraging the practical observation that data often contains additional structure, i.e., the input covariance has non-trivial alignment with the target, we prove that the class of learnable targets can be significantly extended, demonstrating a clear separation between kernel methods and two-layer neural networks in this regime.

Add to your calendar or Include in your list