The Library
Learning a neuron by a shallow ReLU network : dynamics and implicit bias for correlated inputs
Tools
Chistikov, Dmitry, Englert, Matthias and Lazic, Ranko (2024) Learning a neuron by a shallow ReLU network : dynamics and implicit bias for correlated inputs. In: 37th Conference on Neural Information Processing Systems (NeurIPS 2023)., New Orleans, USA, 10-16 Dec 2023. Published in: Advances in Neural Information Processing Systems, 36 pp. 23748-23760.
|
PDF
WRAP-Learning-Neuron-Shallow-ReLU-Network-23.pdf - Accepted Version - Requires a PDF viewer. Download (1754Kb) | Preview |
Official URL: https://openreview.net/forum?id=xgY4QcOiEZ
Abstract
We prove that, for the fundamental regression task of learning a single neuron, training a one-hidden layer ReLU network of any width by gradient flow from a small initialisation converges to zero loss and is implicitly biased to minimise the rank of network parameters. By assuming that the training points are correlated with the teacher neuron, we complement previous work that considered orthogonal datasets. Our results are based on a detailed non-asymptotic analysis of the dynamics of each hidden neuron throughout the training. We also show and characterise a surprising distinction in this setting between interpolator networks of minimal rank and those of minimal Euclidean norm. Finally we perform a range of numerical experiments, which corroborate our theoretical findings.
Item Type: | Conference Item (Paper) | ||||||
---|---|---|---|---|---|---|---|
Divisions: | Faculty of Science, Engineering and Medicine > Science > Computer Science | ||||||
Journal or Publication Title: | Advances in Neural Information Processing Systems | ||||||
Official Date: | 2024 | ||||||
Dates: |
|
||||||
Volume: | 36 | ||||||
Page Range: | pp. 23748-23760 | ||||||
Status: | Peer Reviewed | ||||||
Publication Status: | Published | ||||||
Access rights to Published version: | Restricted or Subscription Access | ||||||
Date of first compliant deposit: | 24 November 2023 | ||||||
Date of first compliant Open Access: | 1 May 2024 | ||||||
Conference Paper Type: | Paper | ||||||
Title of Event: | 37th Conference on Neural Information Processing Systems (NeurIPS 2023). | ||||||
Type of Event: | Conference | ||||||
Location of Event: | New Orleans, USA | ||||||
Date(s) of Event: | 10-16 Dec 2023 | ||||||
Related URLs: | |||||||
Open Access Version: |
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |
Downloads
Downloads per month over past year