The Library
Adaptive learning rate clipping stabilizes learning
Tools
Tools
Tools
Ede, Jeffrey M. and Beanland, Richard (2020) Adaptive learning rate clipping stabilizes learning. Machine Learning : Science and Technology, 1 (1). 015011. doi:10.1088/2632-2153/ab81e2 ISSN 2632-2153.
|
PDF
WRAP-adaptive-learning-rate-clipping-stabilizes-learning-Ede-2020.pdf - Published Version - Requires a PDF viewer. Available under License Creative Commons Attribution 4.0. Download (4Mb) | Preview |
Official URL: http://dx.doi.org/10.1088/2632-2153/ab81e2
Abstract
Artificial neural network training with gradient descent can be destabilized by 'bad batches' with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (ALRC) to limit backpropagated losses to a number of standard deviations above their running means. ALRC is designed to complement existing learning algorithms: Our algorithm is computationally inexpensive, can be applied to any loss function or batch size, is robust to hyperparameter choices and does not affect backpropagated gradient distributions. Experiments with CIFAR-10 supersampling show that ALCR decreases errors for unstable mean quartic error training while stable mean squared error training is unaffected. We also show that ALRC decreases unstable mean squared errors for scanning transmission electron microscopy supersampling and partial scan completion. Our source code is available at https://github.com/Jeffrey-Ede/ALRC.
| Item Type: | Journal Article | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Subjects: | Q Science > Q Science (General) Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software Q Science > QC Physics |
|||||||||
| Divisions: | Faculty of Science, Engineering and Medicine > Science > Physics | |||||||||
| Library of Congress Subject Headings (LCSH): | Machine learning , Mathematical optimization , Electron microscopy, Neural networks (Computer science) | |||||||||
| Journal or Publication Title: | Machine Learning : Science and Technology | |||||||||
| Publisher: | IOP Publishing Ltd | |||||||||
| ISSN: | 2632-2153 | |||||||||
| Official Date: | 28 April 2020 | |||||||||
| Dates: |
|
|||||||||
| Volume: | 1 | |||||||||
| Number: | 1 | |||||||||
| Article Number: | 015011 | |||||||||
| DOI: | 10.1088/2632-2153/ab81e2 | |||||||||
| Status: | Peer Reviewed | |||||||||
| Publication Status: | Published | |||||||||
| Access rights to Published version: | Open Access (Creative Commons) | |||||||||
| Date of first compliant deposit: | 1 July 2020 | |||||||||
| Date of first compliant Open Access: | 1 July 2020 | |||||||||
| RIOXX Funder/Project Grant: |
|
|||||||||
| Is Part Of: | 1 |
Request changes or add full text files to a record
Repository staff actions (login required)
 |
View Item |
Downloads
Downloads per month over past year
