Li, Junyu (2020) Accelerating the processing of deep neural networks. PhD thesis, University of Warwick.

Preview

PDF WRAP_Theses_Li_J_2020.pdf - Submitted Version - Requires a PDF viewer. Download (1239Kb) | Preview

Request Changes to record.

Abstract

Artificial Intelligent (AI) has become the most potent and forward-looking force in the technologies over the past decade. It allowed breakthrough applications that have truly changed the world for the better. Deep Neural Network (DNN), the back-end of the AI, plays a significant role to bestow computers mind-boggling abilities in many tasks from computer vision, natural language processing, audio understanding to complex signal processing. Despite the rapid development of AI, training increasingly complex DNN is constrained by many limiting factors including learning algorithms, data sizes and computing resources. These issues have attracted a range of research on algorithm optimisation and efficient usage of computing resources. In this thesis, we aim to accelerate the processing of DNN from three aspects: data pruning, distributed processing and network pruning. Particularly, we first develop a real-time data pruning approach together with an automatic strategy for adjusting learning rate. The approach temporarily prunes specific training data items based on real-time analysis and automatically adjust the learning rate according to analysed learning trends. The work reduces training time for popular neural networks by around 24.46% without sacrificing the training accuracy. Second, we propose a novel distributed learning strategy to speed up the training process. It uses the predictors residing in the parameter server to forecast the loss of the model, which is used to compensate for the delayed model updating in distributed learning methods. Our strategy outperforms other remarkable methods in the accuracy. Lastly, we propose a fine-grained pruning method together with a feature-aware weight pruning method to reduce the complexity of the trained neural network. The methods prune less important feature maps and weights for each convolutional layer. The proposed methods can prune 74.2% of operations on average while maintaining the accuracy close to the original model.

Item Type:	Thesis (PhD)
Subjects:	Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software
Library of Congress Subject Headings (LCSH):	Machine learning, Neural networks (Computer science)
Official Date:	October 2020
Dates:	Date Event October 2020 UNSPECIFIED
Institution:	University of Warwick
Theses Department:	Department of Computer Science
Thesis Type:	PhD
Publication Status:	Unpublished
Supervisor(s)/Advisor:	He, Ligang
Format of File:	pdf
Extent:	xiv, 126 leaves : colour illustrations
Language:	eng

Request changes or add full text files to a record

Repository staff actions (login required)

View Item

Downloads