The Library
Exploiting network compressibility and topology in zero-cost NAS
Tools
Xiang, Lichuan, Hunter, Rosco, Dudziak, Łukasz, Xu, Minghao and Wen, Hongkai (2023) Exploiting network compressibility and topology in zero-cost NAS. In: International Conference on Automated Machine Learning (AutoML 2023), Potsdam/Berlin, Germany, 12–15 Sep 2023 (In Press)
|
PDF
WRAP-exploiting-network-compressibility-topology-zero-cost-NAS-2023.pdf - Accepted Version - Requires a PDF viewer. Available under License Creative Commons Attribution 4.0. Download (411Kb) | Preview |
Abstract
Neural Architecture Search (NAS) has been widely used to discover high-performance neural network architectures over manually designed approaches. Despite their success, current NAS approaches often require extensive evaluation of candidate architectures in the search space, or the training of large super networks. To reduce the search cost, zerocost proxies have recently been proposed as a way to effciently predict the performance of an architecture. Though many novel proxies have been put forward in recent years, relatively little attention has been dedicated to pushing our understanding of the existing ones. Contrary to that trend, in our work, we argue that it is worth revisiting and analysing the existing proxies in order to further push the boundaries of zero-cost NAS. Towards that goal, we propose to view the existing proxies through a common lens of network compressibility, trainability, and expressivity. Notably, doing so allows us to build a better understanding of the high-level relationship between different proxies as well as refine some of them into their more informative variants. We leverage these insights to design a novel saliency and metric aggregation method informed by compressibility, orthogonality, and network topology. We show that our proposed methods are simple but powerful and yield state-of-the-art results across popular NAS benchmarks.
Item Type: | Conference Item (Paper) | ||||||
---|---|---|---|---|---|---|---|
Subjects: | Q Science > Q Science (General) Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software T Technology > TA Engineering (General). Civil engineering (General) |
||||||
Divisions: | Faculty of Science, Engineering and Medicine > Science > Computer Science Faculty of Science, Engineering and Medicine > Engineering > Engineering |
||||||
Library of Congress Subject Headings (LCSH): | Neural networks (Computer science), Computer vision , Image processing , Machine Learning | ||||||
Official Date: | 2023 | ||||||
Dates: |
|
||||||
Status: | Peer Reviewed | ||||||
Publication Status: | In Press | ||||||
Access rights to Published version: | Open Access (Creative Commons) | ||||||
Date of first compliant deposit: | 4 August 2023 | ||||||
Date of first compliant Open Access: | 7 August 2023 | ||||||
Conference Paper Type: | Paper | ||||||
Title of Event: | International Conference on Automated Machine Learning (AutoML 2023) | ||||||
Type of Event: | Conference | ||||||
Location of Event: | Potsdam/Berlin, Germany | ||||||
Date(s) of Event: | 12–15 Sep 2023 |
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |
Downloads
Downloads per month over past year