Skip to content Skip to navigation
University of Warwick
  • Study
  • |
  • Research
  • |
  • Business
  • |
  • Alumni
  • |
  • News
  • |
  • About

University of Warwick
Publications service & WRAP

Highlight your research

  • WRAP
    • Home
    • Search WRAP
    • Browse by Warwick Author
    • Browse WRAP by Year
    • Browse WRAP by Subject
    • Browse WRAP by Department
    • Browse WRAP by Funder
    • Browse Theses by Department
  • Publications Service
    • Home
    • Search Publications Service
    • Browse by Warwick Author
    • Browse Publications service by Year
    • Browse Publications service by Subject
    • Browse Publications service by Department
    • Browse Publications service by Funder
  • Help & Advice
University of Warwick

The Library

  • Login
  • Admin

Adaptive learning rate clipping stabilizes learning

Tools
- Tools
+ Tools

Ede, Jeffrey M. and Beanland, Richard (2020) Adaptive learning rate clipping stabilizes learning. Machine Learning : Science and Technology, 1 (1). 015011. doi:10.1088/2632-2153/ab81e2 ISSN 2632-2153.

[img]
Preview
PDF
WRAP-adaptive-learning-rate-clipping-stabilizes-learning-Ede-2020.pdf - Published Version - Requires a PDF viewer.
Available under License Creative Commons Attribution 4.0.

Download (4Mb) | Preview
Official URL: http://dx.doi.org/10.1088/2632-2153/ab81e2

Request Changes to record.

Abstract

Artificial neural network training with gradient descent can be destabilized by 'bad batches' with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (ALRC) to limit backpropagated losses to a number of standard deviations above their running means. ALRC is designed to complement existing learning algorithms: Our algorithm is computationally inexpensive, can be applied to any loss function or batch size, is robust to hyperparameter choices and does not affect backpropagated gradient distributions. Experiments with CIFAR-10 supersampling show that ALCR decreases errors for unstable mean quartic error training while stable mean squared error training is unaffected. We also show that ALRC decreases unstable mean squared errors for scanning transmission electron microscopy supersampling and partial scan completion. Our source code is available at https://github.com/Jeffrey-Ede/ALRC.

Item Type: Journal Article
Subjects: Q Science > Q Science (General)
Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software
Q Science > QC Physics
Divisions: Faculty of Science, Engineering and Medicine > Science > Physics
Library of Congress Subject Headings (LCSH): Machine learning , Mathematical optimization , Electron microscopy, Neural networks (Computer science)
Journal or Publication Title: Machine Learning : Science and Technology
Publisher: IOP Publishing Ltd
ISSN: 2632-2153
Official Date: 28 April 2020
Dates:
DateEvent
28 April 2020Published
20 March 2020Accepted
Volume: 1
Number: 1
Article Number: 015011
DOI: 10.1088/2632-2153/ab81e2
Status: Peer Reviewed
Publication Status: Published
Access rights to Published version: Open Access (Creative Commons)
Date of first compliant deposit: 1 July 2020
Date of first compliant Open Access: 1 July 2020
RIOXX Funder/Project Grant:
Project/Grant IDRIOXX Funder NameFunder ID
EP/N035437/1 [EPSRC] Engineering and Physical Sciences Research Councilhttp://dx.doi.org/10.13039/501100000266
1917382[EPSRC] Engineering and Physical Sciences Research Councilhttp://dx.doi.org/10.13039/501100000266
Is Part Of: 1

Request changes or add full text files to a record

Repository staff actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics

twitter

Email us: wrap@warwick.ac.uk
Contact Details
About Us