Effect size, intervals and the fate of the p value

Year 2 / Module 13 Quantitative methods and evaluation statistics / Unit 5 / Effect size, intervals and the fate of the p value

HCI2105 · Unit 5 of 10

A p value answers a question about data given a null hypothesis and cannot express how large an effect is or how precise an estimate has become; the case for reporting effects with intervals is settled by the evidence, but whether the significance test should be abandoned rather than supplemented is a judgement about scientific practice on which competent statisticians disagree.

Effect size, intervals and the fate of the p value

A p value is a statement about data under a null hypothesis. It is not the size of an effect, and it is not the precision of an estimate. The American Statistical Association said so in 2016 (Wasserstein and Lazar, 2016), after a generation of critics had said it (Cohen, 1994). This fifth of ten units takes the powered design into its report, and argues that what follows is not settled. That effects should be reported with intervals is close to consensus. The interval is not self-interpreting, however, and the quantity it surrounds is a family rather than one number (Kelley and Preacher, 2012; Hoekstra et al., 2014). Whether the significance test should be supplemented or abandoned is a judgement on which competent statisticians disagree (Lakens et al., 2018; McShane et al., 2019).

A p value answers a question about data, not about an effect

After four decades of severe criticism the ritual persisted: mechanical dichotomous decisions around a .05 criterion (Cohen, 1994). Readers take p as the probability that the null is false, its complement as the probability of replication, and a rejection as an affirmation of the theory that produced the test.

Greenland and colleagues count 25 misinterpretations of p values, confidence intervals and power. They argue that no reading of these statistics is at once simple, intuitive, correct and free of assumptions. The shortcut that makes a p value easy to state is usually the one that makes it false (Greenland et al., 2016).

The ASA statement sets out six principles. One holds that a p value does not measure the size of an effect or the importance of a result. Another holds that by itself a p value is not a good measure of evidence about a model or hypothesis (Wasserstein and Lazar, 2016).

The p value is too unstable to carry a decision

A harder objection is that the p value is at least a graded measure of evidence. Halsey and colleagues show that it is not. For a fixed true effect the p value varies widely between samples. A study obtaining p = .03 leaves only a 56 per cent chance that a replicate is also below .05 (Halsey et al., 2015). A statistic that moves this far under replication cannot replace the estimate in a report.

The remedy is to report the estimate, not the test

Cumming calls the package the new statistics: effect sizes, confidence intervals and meta-analysis. He organises it as an eight-step strategy with no place for null hypothesis significance testing (Cumming, 2013). A reader given an interval can see how large the effect plausibly is and how much uncertainty remains. Cohen had recommended the same from inside the establishment (Cohen, 1994).

Fritz, Morris and Richler surveyed articles in the Journal of Experimental Psychology: General in 2009 and 2010, a journal whose manual calls for effect sizes and intervals. Effect sizes appeared for fewer than half of the analyses. No article reported a confidence interval for an effect size, and two thirds of the t test articles reported none (Fritz et al., 2012).

The estimate is not one number: effect size is a family

The phrase effect size has been asked to do too much. Kelley and Preacher show that it is used inconsistently, because authors disagree about what it defines. They propose a general definition with three facets: dimension, measure or index, and value (Kelley and Preacher, 2012). A standardised mean difference, a correlation, an odds ratio and a partial eta squared answer different questions.

The interval is the stronger half of the remedy. Two studies can report a claim below .05 while their intervals differ so much that only one licenses a practical decision (Cumming, 2013).

The interval is not self-interpreting

Hoekstra and colleagues gave 120 researchers and 442 psychology students six statements about a 95 per cent confidence interval, all of them false. Both groups endorsed on average more than three of the six, and the correct interpretation was chosen in about 11 per cent of cases (Hoekstra et al., 2014).

Morey and colleagues press the point technically. Intervals are promoted as indexing precision and as showing which values are plausible. The authors show by example that a confidence interval need not have those properties (Morey et al., 2015). Cumming and Finch (2005) reviewed how psychologists read figures with error bars, and they proposed seven rules of eye. They established that the overlap of two confidence intervals on independent means does not map onto a significance decision in the way readers assume. Intervals can overlap while the difference is statistically reliable, and a glance at a figure is not a substitute for the arithmetic, least of all for an older reader who must enlarge the type or who reads the labels in a second language.

In HCI the report is part of the method

HCI adds a practical problem: the report in this field is often a figure. Dragicevic treats the chart as part of the method. Error bars are ambiguous unless the reader knows whether they show standard deviations, standard errors or confidence intervals. A chart is also a barrier, because a red and green pair is invisible to a colour-blind reader, and a screen reader cannot voice an error bar. Standard errors should not be plotted. The interval overlap rule must not be applied in a binary way, and within-subject intervals must account for the correlation between conditions (Dragicevic, 2016).

Contested question: clarify, redefine or abandon the test?

Agreement ends at the test itself, and the literature holds four positions.

The first keeps the test and asks for context. The ASA statement sets out how p values may be used. It insists that they not be the sole basis of a decision, and it places them among other evidence (Wasserstein and Lazar, 2016).

The second keeps the test but moves the threshold. Benjamin and colleagues propose changing the default criterion for new discoveries from .05 to .005. At prior odds of 1:10 the false positive rate at .05 exceeds 33 per cent regardless of power. At .005 it falls to about 5 per cent (Benjamin et al., 2017).

The third rejects a new universal threshold. Lakens and colleagues argue that a single new convention repeats the mistake of the old one. They hold that researchers should report and justify every design choice, the alpha level included (Lakens et al., 2018).

The fourth abandons the term. Amrhein, Greenland and McShane, writing in Nature with more than 800 signatories, argue that statistically significant should be abandoned, and that authors should report estimates with their uncertainty instead (Amrhein et al., 2019). McShane and colleagues widen the case. The problems are not solved by moving the threshold, by using intervals or by using Bayes factors. The paradigm should be dropped, and the p value demoted to one piece of evidence among prior evidence and design quality (McShane et al., 2019).

What would settle this is a comparison of the inferences that the competing practices license on the same body of research. It would report the decisions each produces and the errors each tolerates. None of these sources runs that comparison. The fate of the p value therefore remains a judgement about how science is organised rather than a finding about the arithmetic.

The four positions rest on evidence of different kinds. The threshold proposal carries computations of the error rate under stated prior odds. The abandonment case is a signed comment. The estimation case rests on a demonstration that intervals are misread.

The strongest objection is that the remedy has no record of changing decisions. Fritz, Morris and Richler found a journal whose manual required intervals and almost no intervals reported. Hoekstra and colleagues found the interval misread by researchers and students alike.

What to do while the dispute is open

While the dispute is open, report the estimate and its interval, and state the alpha which was chosen and why.

Where the argument is heading

The direction of travel is toward reporting the whole function rather than a single verdict. Cumming already treats the interval and the meta-analytic combination as the primary output (Cumming, 2013). The redefinition proposal (Benjamin et al., 2017) met the justify-your-alpha reply (Lakens et al., 2018), and the abandonment position widened from the term (Amrhein et al., 2019) to the paradigm (McShane et al., 2019). In HCI the nearest-term work is on communication (Dragicevic, 2016).

The sources this unit rests on

Amrhein, V., Greenland, S. and McShane, B. B. (2019). Scientists rise up against statistical significance. Nature. doi:10.1038/d41586-019-00857-9 A comment with more than 800 signatories urging that statistically significant be abandoned and that estimates with their uncertainty be reported instead of a threshold deciding what counts as a discovery. The unit treats it as the strongest statement of the abandonment position.
Benjamin, D. J., Berger, J. O., Johannesson, M., et al. (2017). Redefine statistical significance. Nature Human Behaviour. doi:10.1038/s41562-017-0189-z They propose lowering the default threshold for new discoveries from .05 to .005, reporting that at prior odds of 1:10 the false positive rate at .05 exceeds 33 per cent and falls to about 5 per cent at .005. It is the convention-adjusting position set against abandonment.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist. doi:10.1037/0003-066x.49.12.997 He reviews four decades of criticism and argues that the ritual of decisions around a .05 criterion persists, naming the misreadings of p as the probability that the null is false and of a rejection as an affirmation of the theory. He recommends estimating effect sizes with intervals.
Cumming, G. (2013). The New Statistics. Psychological Science. doi:10.1177/0956797613504966 He argues for a shift from dichotomous decisions to estimation, with effect sizes, confidence intervals and meta-analysis, and sets out an eight-step strategy that has no place for null hypothesis significance testing. It states the estimation position the unit builds on.
Dragicevic, P. (2016). Fair Statistical Communication in HCI. Modern Statistical Methods for HCI. Springer. doi:10.1007/978-3-319-26633-6_13 He examines how statistics are communicated in HCI papers and argues that charts, error bars and reported tests should let a reader judge the claim without the authors' interpretation; unlabelled error bars are ambiguous and standard errors should not be plotted. It supplies the unit's HCI-specific conclusion that the report is part of the method.
Fritz, C. O., Morris, P. E. and Richler, J. J. (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General. doi:10.1037/a0024338 They surveyed articles in that journal from 2009 and 2010 and found effect sizes in fewer than half of the analyses, no confidence interval for an effect size in any article, and none for two thirds of the t tests. It documents the practice gap between a standing standard and what is published.
Greenland, S., Senn, S. J., Rothman, K. J., et al. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. doi:10.1007/s10654-016-0149-3 They catalogue 25 misinterpretations of p values, confidence intervals and power, arguing that none of these concepts has an interpretation that is at once simple, correct and free of assumptions. It supplies the misreadings the unit expects students to correct.
Halsey, L. G., Curran-Everett, D., Vowler, S. L., et al. (2015). The fickle P value generates irreproducible results. Nature Methods. doi:10.1038/nmeth.3288 They show that for a fixed effect the p value varies widely between samples, and that a study obtaining p = .03 has only a 56 per cent chance that a replicate is also below .05. It explains why a p value cannot measure evidence strength.
Hoekstra, R., Morey, R. D., Rouder, J. N., et al. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin and Review. doi:10.3758/s13423-013-0572-3 They asked 120 researchers and 442 psychology students to judge six false statements about a 95 per cent confidence interval and found both groups endorsing more than three of the six, with the correct interpretation chosen in about 11 per cent of cases. It shows the proposed remedy is itself widely misunderstood.
Kelley, K. and Preacher, K. J. (2012). On effect size. Psychological Methods. doi:10.1037/a0028086 They survey the confusion over how effect size is defined and propose a general definition with three facets, dimension, measure or index, and value, noting that estimates should be reported with an interval. It shows the remedy the unit recommends is itself contested in detail.
Lakens, D., Adolfi, F., Albers, C. J., et al. (2018). Justify your alpha. Nature Human Behaviour. doi:10.1038/s41562-018-0311-x They reply to proposals to redefine significance at .005 by arguing that researchers should transparently report and justify all their design choices, including the alpha level, rather than adopt a new universal convention. It is the main critique of the redefinition proposal.
McShane, B. B., Gal, D., Gelman, A., et al. (2019). Abandon Statistical Significance. The American Statistician. doi:10.1080/00031305.2018.1527253 They argue that the problems of null hypothesis significance testing are not solved by moving the threshold, using intervals or using Bayes factors, and that the paradigm should be abandoned, with the p value demoted to one piece of evidence among prior evidence, design quality and costs. It closes the contested question against the ASA position.
Morey, R. D., Hoekstra, R., Rouder, J. N., et al. (2015). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin and Review. doi:10.3758/s13423-015-0947-8 They argue that confidence intervals cannot be given the Bayesian interpretation most researchers place on them, showing by example that an interval need not index precision or plausible parameter values. It states why the estimation remedy is not automatically more informative.
Wasserstein, R. L. and Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician. doi:10.1080/00031305.2016.1154108 The statement sets out six principles, two of which the unit rests on: a p value does not measure the size of an effect or the importance of a result, and by itself it is not a good measure of evidence about a model or hypothesis. It stops short of calling for abandonment.
Cumming, G. and Finch, S. (2005). Inference by Eye: Confidence Intervals and How to Read Pictures of Data. American Psychologist. doi:10.1037/0003-066x.60.2.170 They analyse how error bars are read and propose seven rules of eye, showing that the overlap of confidence intervals on independent means does not correspond to a significance decision in the way readers assume.

Check your understanding & apply it · 4 questions, 1 essay

1A paper reports that the new interface significantly improved task completion, p = .04, and the authors treat the p value as the size of the improvement. Which diagnosis does the unit support?

2A study obtains p = .03 and the authors treat it as a graded measure of evidence against the null. What does the unit say follows under replication?

3A team plans to report a 95 per cent confidence interval and expects readers to take it as the probability that the true value lies inside. Which caution does the unit support?

4Four sources hold different positions on the fate of the significance test. Which reading does the unit support as the judgement that arithmetic cannot settle?

5Essay Take a published claim of the form 'the new interface significantly improved task completion (p = .04)'. Rewrite the claim in the estimation idiom, saying what an effect size and an interval would add and how you would compute an interval that matches the design. Then take a position on the unit's contested question: should the significance test be supplemented by estimation or abandoned, and why? Use Cumming (2013) or Cohen (1994) for the estimation position; Hoekstra and colleagues (2014) or Morey and colleagues (2015) for the limits of intervals; Fritz and colleagues (2012) for the practice gap; Benjamin and colleagues (2017) or Lakens and colleagues (2018) for the threshold debate; Amrhein and colleagues (2019) or McShane and colleagues (2019) for the abandonment position; and Dragicevic (2016) for what a fair report in HCI must show.

Begin by separating what the p value licenses from what the claim asserts, because the claim treats a decision as a magnitude. The p value answers a question about data under a null hypothesis and does not measure the size or importance of the effect (Wasserstein and Lazar, 2016), so the honest rewrite reports the difference in completion rates as an effect with an interval: for example, a rise of four percentage points with a 95 per cent interval from two to six points. That interval does work the verdict cannot, because a reader can see whether the whole interval clears the threshold that would matter for a shipping decision (Cumming, 2013). Say how the interval was computed. A between-subjects comparison of two proportions needs an interval from the difference of independent proportions, whereas a within-subjects comparison needs an interval that accounts for the correlation between conditions, and Dragicevic (2016) warns that the ordinary calculation is wrong in within-subject designs and that unlabelled error bars mislead. State the effect size measure explicitly, since a difference in proportions, a risk ratio and a standardised difference answer different questions (Kelley and Preacher, 2012). Add a caveat from the interval literature: an interval is not a Bayesian statement of plausibility, and readers systematically over-interpret it as the probability that the true value lies inside (Hoekstra et al., 2014; Morey et al., 2015), so the report must define it. Then take a position. The better argument is for supplementation rather than abandonment at the level of an individual study, because the p value carries information about compatibility with a null when the design is prespecified, and because the practice gap Fritz and colleagues (2012) document shows that even the modest step of reporting effect sizes with intervals has not been taken. Abandoning the test while intervals are computed wrongly and read wrongly would trade one misread statistic for another that is misread in a new way. But the abandonment case has force at the level of the literature rather than the single paper: McShane and colleagues (2019) are right that a threshold screening role distorts what gets published, so the field should stop treating p below .05 as the criterion of a discovery, even while an individual prespecified test is reported. That position is closest to Lakens and colleagues (2018), who ask for a justified alpha rather than a new universal convention, and it is against Benjamin and colleagues (2017), whose .005 proposal fixes the threshold instead of questioning it.