The Library
The frequentist implications of optional stopping on Bayesian hypothesis tests
Tools
Tools
Tools
Sanborn, Adam N. and Hills, Thomas Trenholm (2014) The frequentist implications of optional stopping on Bayesian hypothesis tests. Psychonomic Bulletin & Review , Volume 21 (Number 2). pp. 283-300. doi:10.3758/s13423-013-0518-9
|
PDF
WRAP-frequentist_implications-Sanborn-2017.pdf - Accepted Version - Requires a PDF viewer. Download (1164Kb) | Preview |
Official URL: http://dx.doi.org/10.3758/s13423-013-0518-9
Abstract
Null hypothesis significance testing (NHST) is the most commonly used statistical methodology in psychology. The probability of achieving a value as extreme or more extreme than the statistic obtained from the data is evaluated, and if it is low enough, the null hypothesis is rejected. However, because common experimental practice often clashes with the assumptions underlying NHST, these calculated probabilities are often incorrect. Most commonly, experimenters use tests that assume that sample sizes are fixed in advance of data collection but then use the data to determine when to stop; in the limit, experimenters can use data monitoring to guarantee that the null hypothesis will be rejected. Bayesian hypothesis testing (BHT) provides a solution to these ills because the stopping rule used is irrelevant to the calculation of a Bayes factor. In addition, there are strong mathematical guarantees on the frequentist properties of BHT that are comforting for researchers concerned that stopping rules could influence the Bayes factors produced. Here, we show that these guaranteed bounds have limited scope and often do not apply in psychological research. Specifically, we quantitatively demonstrate the impact of optional stopping on the resulting Bayes factors in two common situations: (1) when the truth is a combination of the hypotheses, such as in a heterogeneous population, and (2) when a hypothesis is composite—taking multiple parameter values—such as the alternative hypothesis in a t-test. We found that, for these situations, while the Bayesian interpretation remains correct regardless of the stopping rule used, the choice of stopping rule can, in some situations, greatly increase the chance of experimenters finding evidence in the direction they desire. We suggest ways to control these frequentist implications of stopping rules on BHT.
| Item Type: | Journal Article | ||||||
|---|---|---|---|---|---|---|---|
| Subjects: | B Philosophy. Psychology. Religion > BF Psychology | ||||||
| Divisions: | Faculty of Science > Psychology | ||||||
| Library of Congress Subject Headings (LCSH): | Bayesian statistical decision theory, Statistical hypothesis testing, Psychology--Methodology | ||||||
| Journal or Publication Title: | Psychonomic Bulletin & Review | ||||||
| Publisher: | Psychonomic Society | ||||||
| ISSN: | 1069-9384 | ||||||
| Official Date: | April 2014 | ||||||
| Dates: |
|
||||||
| Volume: | Volume 21 | ||||||
| Number: | Number 2 | ||||||
| Page Range: | pp. 283-300 | ||||||
| DOI: | 10.3758/s13423-013-0518-9 | ||||||
| Status: | Peer Reviewed | ||||||
| Publication Status: | Published | ||||||
| Access rights to Published version: | Restricted or Subscription Access |
Request changes or add full text files to a record
Repository staff actions (login required)
 |
View Item |
Downloads
Downloads per month over past year
