Gradient Descent in Overparameterized Deep Linear Neural Networks: Adapting to Structure to Generalize

Date:

Abstract: A central question in the mathematics of modern machine learning is why heavily overparameterized models generalize well. In this talk, I will present theoretical results showing that regularization is key to enabling generalization in the overparameterized setting of deep linear neural networks trained to solve an underdetermined linear inverse problem. I will show that, when trained via gradient descent with practical step-sizes and initialization schemes, and without explicit guidance, deep linear neural networks are able to automatically adapt to low-dimensional structure in training data, resulting in improved robustness against noise. These results shed light on how neural networks generalize well in practice by naturally capturing structure in data.