Skip to content Skip to navigation
University of Warwick
  • Study
  • |
  • Research
  • |
  • Business
  • |
  • Alumni
  • |
  • News
  • |
  • About

University of Warwick
Publications service & WRAP

Highlight your research

  • WRAP
    • Home
    • Search WRAP
    • Browse by Warwick Author
    • Browse WRAP by Year
    • Browse WRAP by Subject
    • Browse WRAP by Department
    • Browse WRAP by Funder
    • Browse Theses by Department
  • Publications Service
    • Home
    • Search Publications Service
    • Browse by Warwick Author
    • Browse Publications service by Year
    • Browse Publications service by Subject
    • Browse Publications service by Department
    • Browse Publications service by Funder
  • Help & Advice
University of Warwick

The Library

  • Login
  • Admin

The frequentist implications of optional stopping on Bayesian hypothesis tests

Tools
- Tools
+ Tools

Sanborn, Adam N. and Hills, Thomas Trenholm (2014) The frequentist implications of optional stopping on Bayesian hypothesis tests. Psychonomic Bulletin & Review , Volume 21 (Number 2). pp. 283-300. doi:10.3758/s13423-013-0518-9 ISSN 1069-9384.

[img]
Preview
PDF
WRAP-frequentist_implications-Sanborn-2017.pdf - Accepted Version - Requires a PDF viewer.

Download (1164Kb) | Preview
Official URL: http://dx.doi.org/10.3758/s13423-013-0518-9

Request Changes to record.

Abstract

Null hypothesis significance testing (NHST) is the most commonly used statistical methodology in psychology. The probability of achieving a value as extreme or more extreme than the statistic obtained from the data is evaluated, and if it is low enough, the null hypothesis is rejected. However, because common experimental practice often clashes with the assumptions underlying NHST, these calculated probabilities are often incorrect. Most commonly, experimenters use tests that assume that sample sizes are fixed in advance of data collection but then use the data to determine when to stop; in the limit, experimenters can use data monitoring to guarantee that the null hypothesis will be rejected. Bayesian hypothesis testing (BHT) provides a solution to these ills because the stopping rule used is irrelevant to the calculation of a Bayes factor. In addition, there are strong mathematical guarantees on the frequentist properties of BHT that are comforting for researchers concerned that stopping rules could influence the Bayes factors produced. Here, we show that these guaranteed bounds have limited scope and often do not apply in psychological research. Specifically, we quantitatively demonstrate the impact of optional stopping on the resulting Bayes factors in two common situations: (1) when the truth is a combination of the hypotheses, such as in a heterogeneous population, and (2) when a hypothesis is composite—taking multiple parameter values—such as the alternative hypothesis in a t-test. We found that, for these situations, while the Bayesian interpretation remains correct regardless of the stopping rule used, the choice of stopping rule can, in some situations, greatly increase the chance of experimenters finding evidence in the direction they desire. We suggest ways to control these frequentist implications of stopping rules on BHT.

Item Type: Journal Article
Subjects: B Philosophy. Psychology. Religion > BF Psychology
Divisions: Faculty of Science, Engineering and Medicine > Science > Psychology
Library of Congress Subject Headings (LCSH): Bayesian statistical decision theory, Statistical hypothesis testing, Psychology--Methodology
Journal or Publication Title: Psychonomic Bulletin & Review
Publisher: Psychonomic Society
ISSN: 1069-9384
Official Date: April 2014
Dates:
DateEvent
April 2014Published
8 October 2013Available
Volume: Volume 21
Number: Number 2
Page Range: pp. 283-300
DOI: 10.3758/s13423-013-0518-9
Status: Peer Reviewed
Publication Status: Published
Access rights to Published version: Restricted or Subscription Access
Date of first compliant deposit: 6 April 2017
Date of first compliant Open Access: 6 April 2017

Request changes or add full text files to a record

Repository staff actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics

twitter

Email us: wrap@warwick.ac.uk
Contact Details
About Us