The Library
An optimization-centric view on Bayes’ rule : reviewing and generalizing variational inference
Tools
Tools
Tools
Knoblauch, Jeremias, Jewson, Jack E. and Damoulas, Theodoros (2022) An optimization-centric view on Bayes’ rule : reviewing and generalizing variational inference. Journal of Machine Learning Research, 23 (123). pp. 1-109. ISSN 1532-4435.
|
PDF
WRAP-optimization-centric-view-Bayes-rule-2022.pdf - Published Version - Requires a PDF viewer. Available under License Creative Commons Attribution 4.0. Download (1793Kb) | Preview |
|
![[img]](https://wrap.warwick.ac.uk/style/images/fileicons/application_pdf.png) |
PDF
WRAP-optimization-centric-view-Bayes’-rule-2022.pdf - Accepted Version Embargoed item. Restricted access to Repository staff only - Requires a PDF viewer. Download (2143Kb) |
Official URL: http://jmlr.org/papers/v23/19-1047.html
Abstract
We advocate an optimization-centric view of Bayesian inference. Our inspiration is the representation of Bayes’ rule as infinite-dimensional optimization (Csiszár, 1975; Donsker and Varadhan, 1975; Zellner, 1988). Equipped with this perspective, we study Bayesian inference when one does not have access to (1) well-specified priors, (2) well-specified likelihoods, (3) infinite computing power. While these three assumptions underlie the standard Bayesian paradigm, they are typically inappropriate for modern Machine Learning applications. We propose addressing this through an optimization-centric generalization of Bayesian posteriors that we call the Rule of Three (RoT). The RoT can be justified axiomatically and recovers Bayesian, PAC-Bayesian and VI posteriors as special cases. While the RoT is primarily a conceptual and theoretical device, it also encompasses a novel sub-class of tractable posteriors which we call Generalized Variational Inference (GVI) posteriors. Just as the RoT, GVI posteriors are specified by three arguments: a loss, a divergence and a variational family. They also possess a number of desirable properties, including modularity, Frequentist consistency and an interpretation as approximate ELBO. We explore applications of GVI posteriors, and show that they can be used to improve robustness and posterior marginals on Bayesian Neural Networks and Deep Gaussian Processes.
| Item Type: | Journal Article | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Alternative Title: | |||||||||||||||||||||||||
| Subjects: | Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software | ||||||||||||||||||||||||
| Divisions: | Faculty of Science, Engineering and Medicine > Science > Computer Science | ||||||||||||||||||||||||
| Library of Congress Subject Headings (LCSH): | Neural networks (Computer science), Bayesian statistical decision theory -- Data processing, Machine learning, Artificial intelligence, Gaussian processes | ||||||||||||||||||||||||
| Journal or Publication Title: | Journal of Machine Learning Research | ||||||||||||||||||||||||
| Publisher: | M I T Press | ||||||||||||||||||||||||
| ISSN: | 1532-4435 | ||||||||||||||||||||||||
| Official Date: | 1 May 2022 | ||||||||||||||||||||||||
| Dates: |
|
||||||||||||||||||||||||
| Volume: | 23 | ||||||||||||||||||||||||
| Number: | 123 | ||||||||||||||||||||||||
| Page Range: | pp. 1-109 | ||||||||||||||||||||||||
| Status: | Peer Reviewed | ||||||||||||||||||||||||
| Publication Status: | Published | ||||||||||||||||||||||||
| Access rights to Published version: | Open Access (Creative Commons) | ||||||||||||||||||||||||
| Copyright Holders: | ©2022 Jeremias Knoblauch, Jack Jewson and Theodoros Damoulas | ||||||||||||||||||||||||
| Date of first compliant deposit: | 6 January 2022 | ||||||||||||||||||||||||
| Date of first compliant Open Access: | 5 July 2022 | ||||||||||||||||||||||||
| Funder: | JK and JJ are funded by EPSRC grant EP/L016710/1 as part of the Oxford-Warwick Statistics Programme (OxWaSP)., JK is additionally funded by the Facebook Fellowship Programme and the London Air Quality project at the Alan Turing Institute for Data Science and AI., TD is funded by the UKRI Turing AI Fellowship EP/V02678X/1, EPSRC grant EP/T004134/1 and the Lloyd’s RegisterFoundation programme on Data Centric Engineering at The Alan Turing Institute., This work was furthermore supported by The Alan Turing Institute for Data Science and AI under EPSRC grant EP/N510129/1 in collaboration with the Greater London Authority. | ||||||||||||||||||||||||
| RIOXX Funder/Project Grant: |
|
||||||||||||||||||||||||
| Related URLs: | |||||||||||||||||||||||||
| Open Access Version: |
Request changes or add full text files to a record
Repository staff actions (login required)
 |
View Item |
Downloads
Downloads per month over past year
