ReLU Neural Networks with Linear Layers Are Biased towards Single- and Multi-index Models
Published in SIAM Journal on Mathematics of Data Science, 2025
Connects the representation cost of neural networks with 1 ReLU layer and many linear layers to the spectrum of the expected gradient outer product matrix (EGOP), showing that this architecture is biased towards single- and multi-index models.
Joint work with Greg Ongie and Rebecca Willett.