The Library
Zero-cost operation scoring in differentiable architecture search
Tools
Xiang, Lichuan, Dudziak, Łukasz, Abdelfattah, Mohamed S., Chau, Thomas, Lane, Nicholas D. and Wen, Hongkai (2023) Zero-cost operation scoring in differentiable architecture search. In: Thirty-Seventh AAAI Conference on Artificial Intelligence, Washington, DC, USA, 7-14 Feb 2023. Published in: Proceedings of the AAAI Conference on Artificial Intelligence, 37 (9). pp. 10453-10463. doi:10.1609/aaai.v37i9.26243
|
PDF
WRAP-zero-cost-operation-scoring-differentiable-architecture-search-Wen-2022.pdf - Accepted Version - Requires a PDF viewer. Download (1700Kb) | Preview |
Official URL: https://doi.org/10.1609/aaai.v37i9.26243
Abstract
We formalize and analyze a fundamental component of dif- ferentiable neural architecture search (NAS): local “opera- tion scoring” at each operation choice. We view existing operation scoring functions as inexact proxies for accuracy, and we find that they perform poorly when analyzed empir- ically on NAS benchmarks. From this perspective, we intro- duce a novel perturbation-based zero-cost operation scor- ing (Zero-Cost-PT) approach, which utilizes zero-cost prox- ies that were recently studied in multi-trial NAS but de- grade significantly on larger search spaces, typical for dif- ferentiable NAS. We conduct a thorough empirical evalu- ation on a number of NAS benchmarks and large search spaces, from NAS-Bench-201, NAS-Bench-1Shot1, NAS- Bench-Macro, to DARTS-like and MobileNet-like spaces, showing significant improvements in both search time and accuracy. On the ImageNet classification task on the DARTS search space, our approach improved accuracy compared to the best current training-free methods (TE-NAS) while be- ing over 10× faster (total searching time 25 minutes on a single GPU), and observed significantly better transferabil- ity on architectures searched on the CIFAR-10 dataset with an accuracy increase of 1.8 pp. Our code is available at: https://github.com/zerocostptnas/zerocost operation score.
Item Type: | Conference Item (Paper) | ||||||
---|---|---|---|---|---|---|---|
Subjects: | Q Science > Q Science (General) Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software |
||||||
Divisions: | Faculty of Science, Engineering and Medicine > Engineering > Engineering | ||||||
Library of Congress Subject Headings (LCSH): | Neural networks (Computer science), Computer architecture | ||||||
Journal or Publication Title: | Proceedings of the AAAI Conference on Artificial Intelligence | ||||||
Publisher: | Association for the Advancement of Artificial Intelligence | ||||||
Official Date: | 26 June 2023 | ||||||
Dates: |
|
||||||
Volume: | 37 | ||||||
Number: | 9 | ||||||
Page Range: | pp. 10453-10463 | ||||||
DOI: | 10.1609/aaai.v37i9.26243 | ||||||
Status: | Peer Reviewed | ||||||
Publication Status: | Published | ||||||
Access rights to Published version: | Free Access (unspecified licence, 'bronze OA') | ||||||
Copyright Holders: | Copyright © 2023, Association for the Advancement of Artificial Intelligence | ||||||
Date of first compliant deposit: | 19 December 2022 | ||||||
Date of first compliant Open Access: | 27 October 2023 | ||||||
Conference Paper Type: | Paper | ||||||
Title of Event: | Thirty-Seventh AAAI Conference on Artificial Intelligence | ||||||
Type of Event: | Conference | ||||||
Location of Event: | Washington, DC, USA | ||||||
Date(s) of Event: | 7-14 Feb 2023 | ||||||
Related URLs: |
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |
Downloads
Downloads per month over past year