Bayesian evidence and the stopping rule

A team testing a redesigned filter finds, after ten participants drawn from whoever was to hand, that the difference has not reached the conventional threshold. Two participants later, it has. Nothing changed except the moment the analyst stopped looking. A p value is defined against a sampling plan that includes the intention to stop. A result can therefore cross the conventional line because collection continued rather than because the effect grew. The Bayes factor was proposed as the alternative that does not depend on that intention. This seventh of ten units turns from the report to the Bayes factor, and argues that it buys freedom from the stopping rule and evidence of absence. It pays for them by making the analyst's model part of the claim.

A p value answers a question about a plan

Conventional significance testing cannot state evidence for a null hypothesis, because it models only the null. It asks how surprising the data are under the null, never how well the alternative predicted them (Rouder et al., 2009). Their Bayes factor alternative compares the predictive adequacy of the two, so it can favour either. Dienes and McLatchie give the same move as a three-way distinction. The data can support the null, support the alternative, or fall in a middle band that supports neither, placed between Bayes factors of one third and three (Dienes and McLatchie, 2018).

The evidence does not depend on the intention behind the data

Bayesian hypothesis testing allows a researcher to quantify evidence and monitor its progression as the data come in, without needing to know the intention with which the data were collected (Wagenmakers et al., 2018). A Bayes factor is a ratio of the probabilities that the two models assign to the observed data, so the stopping rule does not enter the calculation. A p value, by contrast, is a probability computed over outcomes that were never observed under a plan that fixes when to stop (Dienes, 2011). Sequential testing is licensed, though the design still needs a planned maximum and a computed probability of misleading evidence (Schonbrodt and Wagenmakers, 2018). Dienes presents the situations in which the two approaches diverge, including the decision about when to stop running participants, and he argues that confusing them misleads (Dienes, 2011).

Evidence of absence is the capability the classic test lacks

A non-significant p value is silent about the null. It records that the data were not unlikely enough under the null, not that the null is supported. It cannot separate absence of evidence from evidence of absence (Keysers et al., 2020). Keysers, Gazzola and Wagenmakers show how Bayesian t tests and analysis of variance in JASP return evidence for the null as well as against it (Keysers et al., 2020). Etz and Vandekerckhove (2016) recomputed Bayes factors for 72 original studies and their replications: 75 per cent gave qualitatively similar results, 64 per cent provided no strong evidence for either hypothesis, and no replication provided strong evidence for the null.

The number is a ratio against a model the analyst chooses

The Bayes factor is the ratio of how well two models predicted the data, so its size depends on which alternative the analyst specified. The alternative is a choice rather than a fact, and it encodes an expectation about whom the effect holds for, since a small sample drawn by convenience rarely contains an older participant, an assistive-technology user or a second-language reader (Ly et al., 2016). Ly, Verhagen and Wagenmakers set out the default tests for the nullity of a normal mean, the Bayesian equivalent of the t test, and for a correlation. They show how the default prior grades the evidence (Ly et al., 2016). Rouder and colleagues' default is a Cauchy distribution whose tails are so heavy that neither its mean nor its variance exists (Rouder et al., 2009). Kass and Raftery (1995) made the ratio legible by applying it five times over, to genetics, sports, ecology, sociology and psychology. They set out the guidelines by which a magnitude becomes weak, moderate or strong evidence. The same number means different things in different applications, because the alternative model and the external information differ.

Why this fits small iterative HCI studies

Kay, Nelson and Hekler argue that this fits HCI better than most fields. They searched the ACM Digital Library and found only 56 meta-analyses in HCI, three at CHI, so results rarely accumulate into more precise estimates (Kay et al., 2016). They then simulate publication worlds under frequentist and Bayesian practice. They report that Bayesian analysis yields more precise effects with each study and supports more principled conclusions from small samples (Kay et al., 2016).

The price is an explicit model, and an explicit model can be wrong

The freedom from the stopping rule holds for a fixed model and prior, and both are fallible. Gelman and Shalizi argue that the most successful Bayesian work does not support the view of Bayesian inference as automatic rationality. Its usefulness depends on the model, and the checking and revising of models falls outside Bayesian confirmation theory (Gelman and Shalizi, 2012). Schad, Betancourt and Vasishth turn that into a workflow. It runs prior predictive checks that test whether the priors imply plausible data, checks of computational faithfulness, sensitivity analysis, and posterior predictive checks that test whether the model captures the data (Schad et al., 2021). Van Doorn and colleagues convert the same discipline into reporting rules. Specify the prior, justify it, and report the robustness of the result (van Doorn et al., 2021).

Contested question: is the Bayes factor the right Bayesian answer?

The disagreement runs inside Bayesian practice, and three positions are on the table. One position holds that the Bayes factor is the right instrument. It grades the evidence that the data provide for one hypothesis against another, and it is not tied to a stopping rule (Rouder et al., 2009; Keysers et al., 2020). A second position, argued most sharply by Kruschke, holds that estimation should replace the test. He shows that Bayesian estimation for two groups yields complete distributions of credible values for the effect size, the group means and their difference, and the standard deviations. A decision rule on the posterior can accept the null value when certainty is high, unlike Bayes factor model comparison (Kruschke, 2013). On his account the Bayes factor discards the distribution that a reader needs.

A third position cuts across both. Gelman and Shalizi argue that neither number is the point, because the usefulness of a Bayesian analysis turns on the model and on the checking and revising of it (Gelman and Shalizi, 2012). Report the prior, assess how far the answer depends on it, and check the model against data (van Doorn et al., 2021; Schad et al., 2021). What would settle the dispute is a body of analyses robust to the choice. That means Bayes factors reported alongside a sensitivity analysis across the priors that reasonable analysts would choose, and posterior intervals showing how far the result moves with the prior. Where the answer is stable across priors, the choice of statistic is a matter of taste. Where it is not, no single number was doing the work.

The evidence is uneven. Freedom from the stopping rule is a property of the arithmetic, so that part is settled. The claimed gain in practice is not. In the largest batch of reanalyses in the unit's sources, most studies provided no strong evidence either way.

The strongest objection is that the freedom rarely arrives. A sensible Bayesian design still needs a planned maximum sample and a computed probability of misleading evidence, which restores the plan the Bayes factor was said to make unnecessary.

What to do while the dispute is open

While the dispute is open, state the prior and justify it before the data are seen, and report how far the conclusion moves when the prior changes.

Where the argument is heading

The direction of travel is toward workflows rather than a single statistic. Schad, Betancourt and Vasishth put prior predictive checks, sensitivity analysis and posterior predictive checks in sequence. Van Doorn and colleagues have turned the same requirements into reporting rules, which is where a method becomes teachable and auditable (Schad et al., 2021; van Doorn et al., 2021). The capability that started the movement, evidence of absence, is being carried into fields that need to rule out effects, as Keysers and colleagues do for neuroscience (Keysers et al., 2020). HCI's own case rests on the accumulation of small studies (Kay et al., 2016). A Bayesian answer is only as good as the model it is conditioned on (Gelman and Shalizi, 2012).

The sources this unit rests on

• Dienes, Z. (2011). Bayesian Versus Orthodox Statistics: Which Side Are You On?. Perspectives on Psychological Science. doi:10.1177/1745691611406920 Dienes separates Bayesian from orthodox inference, arguing that confusion arises when Bayesian intuitions are applied to significance tests, which answer a different question. He shows where the two diverge, including the decision about when to stop running participants. It carries the unit's distinction between what each method can conclude.
• Dienes, Z. and Mclatchie, N. (2018). Four reasons to prefer Bayesian analyses over significance testing. Psychonomic Bulletin & Review. doi:10.3758/s13423-017-1266-z They compare significance testing and Bayes factors across five case studies, giving four reasons to prefer Bayesian analysis, including its treatment of non-significant results. Their figure places the uninformative middle band of evidence between Bayes factors of one third and three.
• Etz, A. and Vandekerckhove, J. (2016). A Bayesian Perspective on the Reproducibility Project: Psychology. PLOS ONE. doi:10.1371/journal.pone.0149794 They recompute Bayes factors for 72 original studies and their replications. They report that 75 per cent gave qualitatively similar results, that 64 per cent provided no strong evidence for either hypothesis, and that no replication provided strong evidence for the null.
• Gelman, A. and Shalizi, C. R. (2012). Philosophy and the practice of Bayesian statistics. British Journal of Mathematical and Statistical Psychology. doi:10.1111/j.2044-8317.2011.02037.x They argue that successful Bayesian statistics do not support the view of Bayesian inference as automatic rationality, and examine the role of priors and the model checking and revision that fall outside Bayesian confirmation theory. The internal critique keeps the argument arguable.
• Kay, M., Nelson, G. L. and Hekler, E. B. (2016). Researcher-Centred Design of Statistics. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. doi:10.1145/2858036.2858465 They found only 56 meta-analyses in HCI in the ACM Digital Library, three at CHI, and treat the failure of results to accumulate as a design problem. A simulation comparing frequentist and Bayesian publication worlds shows Bayesian analysis yielding more precise effects with each study.
• Keysers, C., Gazzola, V. and Wagenmakers, E. (2020). Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence. Nature Neuroscience. doi:10.1038/s41593-020-0660-4 They argue that p value approaches can establish that an effect exists but not that it does not, leaving non-significant findings hard to read. Tutorial Bayesian t tests and analysis of variance in JASP show how to establish both evidence of absence and absence of evidence.
• Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology General. doi:10.1037/a0029146 Kruschke presents Bayesian estimation for two groups, yielding distributions of credible values for the effect size, the group means and their difference, and the standard deviations. The method handles outliers and its decision rule can accept the null value when certainty is high, unlike Bayes factor comparison.
• Ly, A., Verhagen, J. and Wagenmakers, E. (2016). Harold Jeffreys's default Bayes factor hypothesis tests: Explanation, extension, and application in psychology. Journal of Mathematical Psychology. doi:10.1016/j.jmp.2015.06.004 They explain Jeffreys's default Bayes factor tests for testing the nullity of a normal mean, the Bayesian equivalent of the t test, and for a correlation, and show how the default prior grades the evidence. It supplies the default the unit's examples use.
• Rouder, J. N., Speckman, P. L., Sun, D., et al. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review. doi:10.3758/pbr.16.2.225 They present a Bayes factor alternative to the conventional t test that compares the predictive adequacy of the null and the alternative, so it can express a preference for either. Their default prior is a Cauchy distribution whose tails are so heavy that neither its mean nor its variance exists.
• Schad, D. J., Betancourt, M. and Vasishth, S. (2021). Toward a principled Bayesian workflow in cognitive science. Psychological Methods. doi:10.1037/met0000275 They set out a principled Bayesian workflow covering prior predictive checks, computational faithfulness, model sensitivity and posterior predictive checks, worked through on reading-time data. It shows how domain knowledge informs priors and how a fitted model can overfit noise.
• Schonbrodt, F. D. and Wagenmakers, E. (2018). Bayes factor design analysis: Planning for compelling evidence. Psychonomic Bulletin & Review. doi:10.3758/s13423-017-1230-y They extend design analysis to Bayesian testing, setting out a fixed-sample design, an open-ended sequential design in which the analyst may stop once the evidence is strong, and a sequential design with a maximum sample size. They show how to compute the expected strength of evidence and the probability of misleading evidence.
• van Doorn, J., van den Bergh, D., Bohm, U., et al. (2021). The JASP guidelines for conducting and reporting a Bayesian analysis. Psychonomic Bulletin & Review. doi:10.3758/s13423-020-01798-5 They give guidelines for planning, executing, interpreting and reporting a Bayesian analysis, asking authors to specify a prior and justify it, to assess how far the inferences depend on prior choices, and to discuss the result's robustness. It turns the unit's argument into a reporting standard.
• Wagenmakers, E., Marsman, M., Jamil, T., et al. (2018). Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications. Psychonomic Bulletin & Review. doi:10.3758/s13423-017-1343-3 The first part of the series outlines ten advantages of Bayesian over classical inference, including the ability to quantify evidence for the null and to monitor evidence as data accumulate without knowing the intention behind the collection. It states the case in the terms the unit's thesis uses.
• Kass, R. E. and Raftery, A. E. (1995). Bayes Factors. Journal of the American Statistical Association. doi:10.1080/01621459.1995.10476572 They review the Bayes factor as a summary of evidence and work it through five applications in genetics, sports, ecology, sociology and psychology, giving guidelines for interpreting its magnitude and emphasising that conclusions should be assessed for sensitivity to the prior.