The Library
Developing normalization schemes for data isolated distributed deep learning
Tools
Zhou, Yujue, He, Ligang and Yang, Shuang‐Hua (2021) Developing normalization schemes for data isolated distributed deep learning. IET Cyber-Physical Systems: Theory & Applications, 6 (3). pp. 105-115. doi:10.1049/cps2.12004 ISSN 2398-3396.
|
PDF
WRAP-developing-normalization-schemes-data-isolated-distributed-deep-learning-2021.pdf - Published Version - Requires a PDF viewer. Available under License Creative Commons Attribution 4.0. Download (1538Kb) | Preview |
Official URL: http://dx.doi.org/10.1049/cps2.12004
Abstract
Distributed deep learning is an important and indispensable direction in the field of deep learning research. Earlier research has proposed many algorithms or techniques on accelerating distributed neural network training. This study discusses a new distributed training scenario, namely data isolated distributed deep learning. Specifically, each node has its own local data and cannot be shared for some reasons. However, in order to ensure the generalization of the model, the goal is to train a global model that required learning all the data, not just based on data from a local node. At this time, distributed training with data isolation is needed. An obvious challenge for distributed deep learning in this scenario is that the distribution of training data used by each node could be highly imbalanced because of data isolation. This brings difficulty to the normalization process in neural network training, because the traditional batch normalization (BN) method will fail under this kind of data imbalanced scenario. At this time, distributed training with data isolation is needed. Aiming at such data isolation scenarios, this study proposes a comprehensive data isolation deep learning scheme. Specifically, synchronous stochastic gradient descent algorithm is used for data exchange during training, and provides several normalization approaches to the problem of BN failure caused by data imbalance. Experimental results show the efficiency and accuracy of the proposed data isolated distributed deep learning scheme.
Item Type: | Journal Article | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Subjects: | Q Science > Q Science (General) Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software T Technology > TA Engineering (General). Civil engineering (General) |
||||||||||||
Divisions: | Faculty of Science, Engineering and Medicine > Engineering > Engineering | ||||||||||||
SWORD Depositor: | Library Publications Router | ||||||||||||
Library of Congress Subject Headings (LCSH): | Machine learning, Neural networks (Computer science), Computer vision | ||||||||||||
Journal or Publication Title: | IET Cyber-Physical Systems: Theory & Applications | ||||||||||||
Publisher: | Institution of Engineering and Technology (IET) | ||||||||||||
ISSN: | 2398-3396 | ||||||||||||
Official Date: | September 2021 | ||||||||||||
Dates: |
|
||||||||||||
Volume: | 6 | ||||||||||||
Number: | 3 | ||||||||||||
Page Range: | pp. 105-115 | ||||||||||||
DOI: | 10.1049/cps2.12004 | ||||||||||||
Status: | Peer Reviewed | ||||||||||||
Publication Status: | Published | ||||||||||||
Access rights to Published version: | Open Access (Creative Commons) | ||||||||||||
Date of first compliant deposit: | 13 July 2021 | ||||||||||||
Date of first compliant Open Access: | 14 July 2021 | ||||||||||||
RIOXX Funder/Project Grant: |
|
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |
Downloads
Downloads per month over past year