The Library
Adaptive Learning Rate Clipping Stabilizes Learning [pre-print]
Tools
Ede, Jeffrey M. and Beanland, Richard (2019) Adaptive Learning Rate Clipping Stabilizes Learning [pre-print]. Working Paper. arXiv. (Unpublished)
Research output not available from this repository.
Request-a-Copy directly from author or use local Library Get it For Me service.
Official URL: https://arxiv.org/abs/1906.09060
Abstract
Artificial neural network training with stochastic gradient descent can be destabilized by "bad batches" with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (ALRC) to limit backpropagated losses to a number of standard deviations above their running means. ALRC is designed to complement existing learning algorithms: Our algorithm is computationally inexpensive, can be applied to any loss function or batch size, is robust to hyperparameter choices and does not affect backpropagated gradient distributions. Experiments with CIFAR-10 supersampling show that ALCR decreases errors for unstable mean quartic error training while stable mean squared error training is unaffected. We also show that ALRC decreases unstable mean squared errors for partial scanning transmission electron micrograph completion. Our source code is publicly available at this https URL [https://github.com/Jeffrey-Ede/ALRC].
Item Type: | Working or Discussion Paper (Working Paper) | ||||
---|---|---|---|---|---|
Divisions: | Faculty of Science, Engineering and Medicine > Science > Physics | ||||
Journal or Publication Title: | arXiv preprint arXiv:1906.09060 | ||||
Publisher: | arXiv | ||||
Official Date: | 2019 | ||||
Dates: |
|
||||
Number: | 1906.09060 | ||||
Institution: | University of Warwick | ||||
Status: | Not Peer Reviewed | ||||
Publication Status: | Unpublished | ||||
Access rights to Published version: | Open Access (Creative Commons) | ||||
Description: | Preprint on adaptive learning rate clipping. This is an algorithm developed to stabilize neural network training. Particularly for small training batch sizes. |
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |