The Library
Efficient local linearity regularization to overcome catastrophic overfitting
Tools
Rocamora, Elias Abad, Liu, Fanghui, Chrysos, Grigorios, Olmos, Pablo M. and Cevher, Volkan (2024) Efficient local linearity regularization to overcome catastrophic overfitting. In: The Twelfth International Conference on Learning Representations, Vienna, Austria, 7-11 May 2024
PDF
WRAP-Efficient-local-linearity-regularization-overcome-catastrophic-overfitting-24.pdf - Accepted Version Embargoed item. Restricted access to Repository staff only - Requires a PDF viewer. Download (4Mb) |
Official URL: https://openreview.net/forum?id=SZzQz8ikwg
Abstract
Catastrophic overfitting (CO) in single-step adversarial training (AT) results in abrupt drops in the adversarial test accuracy (even down to
%). For models trained with multi-step AT, it has been observed that the loss function behaves locally linearly with respect to the input, this is however lost in single-step AT. To address CO in single-step AT, several methods have been proposed to enforce local linearity of the loss via regularization. However, these regularization terms considerably slow down training due to Double Backpropagation. Instead, in this work, we introduce a regularization term, called ELLE, to mitigate CO effectively and efficiently in classical AT evaluations, as well as some more difficult regimes, e.g., large adversarial perturbations and long training schedules. Our regularization term can be theoretically linked to curvature of the loss function and is computationally cheaper than previous methods by avoiding Double Backpropagation. Our thorough experimental validation demonstrates that our work does not suffer from CO, even in challenging settings where previous works suffer from it. We also notice that adapting our regularization parameter during training (ELLE-A) greatly improves the performance, specially in large
setups. Our implementation is available in https://github.com/LIONS-EPFL/ELLE.
Item Type: | Conference Item (Paper) | ||||||
---|---|---|---|---|---|---|---|
Divisions: | Faculty of Science, Engineering and Medicine > Science > Computer Science | ||||||
Official Date: | 21 April 2024 | ||||||
Dates: |
|
||||||
Status: | Peer Reviewed | ||||||
Publication Status: | Published | ||||||
Access rights to Published version: | Free Access (unspecified licence, 'bronze OA') | ||||||
Date of first compliant deposit: | 2 April 2024 | ||||||
Conference Paper Type: | Paper | ||||||
Title of Event: | The Twelfth International Conference on Learning Representations | ||||||
Type of Event: | Conference | ||||||
Location of Event: | Vienna, Austria | ||||||
Date(s) of Event: | 7-11 May 2024 | ||||||
Related URLs: | |||||||
Open Access Version: |
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |