Statistics, effect sizes and what a result supports

A significance test returns a verdict on a question your reader is not asking. A reader wants to know how large the effect was, which way it points, and how much of the estimate is uncertainty. A p value answers none of these. It reports how surprising the data would be if the null model, and only the null model, were true.

The sixth of ten units argues that a defensible result reports an estimate with an interval instead, and the seventh asks how far that estimate holds. It states what the design could have detected, and it treats any threshold as a stated convention.

A p value answers a question nobody asked

Wasserstein and Lazar (2016) present the American Statistical Association's statement on p values, which sets out six principles. A p value does not measure the probability that the studied hypothesis is true, nor the size of an effect or the importance of a result. A small p value can accompany a trivial effect; a large one a substantial effect.

Greenland and colleagues (2016) supply the catalogue a results section needs: 25 documented misinterpretations of p values, confidence intervals and power. They demonstrate that unstated analysis protocols can produce small p values when the hypothesis is correct, and large ones when it is wrong. The sentence "the effect was significant, so the interface is better" commits two of their errors at once.

A single p value is unstable

Even read correctly, one p value is a fragile summary. Halsey and colleagues (2015) show through simulation that the p value varies widely across samples drawn from the same populations, so a repeated experiment yields a wide range of values. Drawing repeatedly from two populations that differ by 0.5 units, the effect estimates they obtain include 1.46 with p = 0.005 and 0.74 with p = 0.09, and a study returning p = 0.03 leaves a 90 per cent chance that a repeat lands anywhere between 0 and 0.6.

An estimate with an interval answers the reader's question

Cumming (2014) names the alternative: estimation based on effect sizes, confidence intervals and meta-analysis. He sets out an eight-step strategy for research with integrity that begins with questions phrased in estimation terms.

Lakens (2013) makes the reporting concrete. He gives the effect-size measures for t tests and analyses of variance and supplies a spreadsheet, so the calculation is no barrier. Effect sizes matter, he argues, because they let the next study set its sample and a meta-analysis combine studies.

Ho and colleagues (2019) make the report legible. They note that the limits of significance testing have been discussed for at least 77 years without agreement on a replacement, and that the test confines the analyst to asking whether an effect exists, while estimation asks how large it is. Their estimation graphic plots the full distribution of the data with the effect size and its interval, placing the raw values for each group as a column of points, the estimated difference beneath them, and the confidence interval as a whisker around that estimate. A table of the same values carries the estimate to a reader who cannot see the graphic or uses a screen reader.

State what the design could detect

Cohen (1992) provides the standard tool: tables giving the sample size required for 0.80 power at conventional small, medium and large effect sizes across eight common tests. A project need not promise 0.80 power, but it should say what effect its sample could have detected.

Button and colleagues (2013) show why the number belongs in the paper. They estimate the median statistical power of studies in the neurosciences at between about 8 and 31 per cent. Low power reduces the chance of detecting a true effect, and it lowers the probability that a significant result reflects a true one, because the selection itself inflates the effects that pass a low bar.

The contested question: abandon, lower, or justify the threshold?

McShane and colleagues (2019) argue that thresholded significance should be abandoned. The p value should be demoted from a screening role and treated, continuously, as one piece of evidence among prior evidence, plausibility, design quality and cost.

Benjamin and colleagues (2018) keep a threshold and move it. They propose 0.005 instead of 0.05 for claims of new discoveries, because the current level admits too many false positives. Lakens and colleagues (2018) reject a fixed number, replying that each study should justify the level against its purpose and report the justification. Cumming (2014) and Wasserstein, Schirm and Lazar (2019) take a fourth position. Estimation and meta-analysis, not any threshold, are the route to cumulative knowledge.

Ioannidis (2018) sits inside the dispute. He signed the 0.005 proposal, yet he calls lowering the threshold a temporizing measure. He names collateral harms: bias may escalate as researchers work to reach the lower bar, and the effects that survive may be more exaggerated by regression to the mean. He reports that 96 per cent of articles reporting p values include one at or below 0.05, and that moving to 0.005 would shift about one third of past significant biomedical results into a merely suggestive category. What would settle the question is a within-field test of whether a rule change lowers false claims without discarding true ones. Ioannidis prices the cost, but no paper here shows the claims he preserves are truer.

Read by kind, no side has tested a rule change against the claims it keeps. The case for abandoning the threshold is an argument. The case for a lower threshold rests on a proposal priced by a count, and the case for estimation on a simulation and an eight-step method. While the dispute is open, a project should report its estimate with an interval, state the smallest effect its design could detect, name the population the sample was drawn from, and declare any threshold with the reason for it. The strongest objection to this unit's thesis is that an interval is not free of judgement, since it rests on the same decisions that make a threshold fragile, and a reader may treat it as a verdict.

What this means for an MSci results section

A project compares two interfaces on one task with 24 participants from one population and reports a difference significant at p = 0.03, in a paragraph ending "the new interface improves efficiency". The writer can revise the paragraph without deleting the finding. Report the mean difference and its confidence interval, drawn as an estimation graphic in the style of Ho and colleagues (2019). State the smallest effect the design could detect, computed as Cohen (1992) describes. Report the effect size, as Lakens (2013) sets out. The p value may stay, but as one number among these, not as the verdict.

If the project keeps a threshold, it should say which level and why, following Lakens and colleagues (2018). If it reports none, it should say that too, in the manner of Cumming (2014). What it may not do is present a p value as strong or weak evidence for the hypothesis tested. Wasserstein and Lazar (2016) and Greenland and colleagues (2016) list that reading among the documented misinterpretations.

Where this is heading

The direction is towards analyses and displays whose native output is an estimate. Kruschke (2013) sets out Bayesian estimation for two groups, which returns distributions of credible values for the effect size and the group difference. Ho and colleagues (2019) supply the matching display, and Wasserstein, Schirm and Lazar (2019) collect 43 papers whose tenor is that the replacement for the threshold is a practice, not a new single number. Bogdan (2025) finds the reform has already changed what is reported: across 240,355 psychology articles from 2004 to 2024, the share of significant p values in the fragile band fell from 32 to 26 per cent. Whether it changes what readers conclude is still open.

The sources this unit rests on

Each note states what the cited work did and found and what it adds to the argument. Identifiers and metadata were checked against the OpenAlex and Semantic Scholar records and the publisher pages on 31 October 2026.

• Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7-29. doi:10.1177/0956797613504966 Argues that estimation based on effect sizes, confidence intervals and meta-analysis should replace significance testing, and sets out an eight-step strategy that begins with questions phrased in estimation terms. It supplies the unit's governing proposal.
• Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, Article 863. doi:10.3389/fpsyg.2013.00863 Gives worked guidance on computing and reporting effect sizes for t tests and analyses of variance. Effect sizes matter, he argues, because they let the next study set its sample and a meta-analysis combine studies.
• Wasserstein, R. L. and Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. doi:10.1080/00031305.2016.1154108 Presents the American Statistical Association's six principles on p values, including that a p value measures neither the probability a hypothesis is true nor the size or importance of an effect. It is the basis for the claim that a threshold crossing is not a result.
• Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N. and Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350. doi:10.1007/s10654-016-0149-3 Catalogues 25 misinterpretations of p values, confidence intervals and power, showing how unstated analysis protocols can yield small p values when the hypothesis is correct and large ones when it is wrong. It is the unit's checklist of errors to avoid.
• Halsey, L. G., Curran-Everett, D., Vowler, S. L. and Drummond, G. B. (2015). The fickle P value generates irreproducible results. Nature Methods, 12(3), 179-185. doi:10.1038/nmeth.3288 Uses simulation to show that the p value varies widely across samples drawn from the same populations, so a repeated experiment yields a range of values. That is why the unit treats a threshold crossing as an unstable summary.
• McShane, B. B., Gal, D., Gelman, A., Robert, C. P. and Tackett, J. L. (2019). Abandon statistical significance. The American Statistician, 73(S1), 235-245. doi:10.1080/00031305.2018.1527253 Proposes dropping thresholded significance as the default and treating the p value as continuous evidence alongside prior evidence, plausibility and design quality. It is the strongest statement that no threshold should decide inference.
• Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J. and colleagues (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6-10. doi:10.1038/s41562-017-0189-z A large coalition proposes changing the default threshold from 0.05 to 0.005 for claims of new discoveries, arguing that the current level admits too many false positives. It is the reformist middle position the unit sets against abandoning the threshold.
• Lakens, D., Adolfson, F. G., Albers, C. J., Anvari, F., Apps, M. A. J. and colleagues (2018). Justify your alpha. Nature Human Behaviour, 2(3), 168-171. doi:10.1038/s41562-018-0311-x Responds to the 0.005 proposal by arguing that researchers should report and justify every design choice, the alpha level included, against the study's purpose rather than inherit a convention. It carries the unit's argument that the decision rule must be defended.
• Ioannidis, J. P. A. (2018). The proposal to lower P value thresholds to .005. JAMA, 319(14), 1429-1430. doi:10.1001/jama.2018.1536 Examines the 0.005 proposal and estimates that lowering the threshold would shift about one third of past significant biomedical results into a suggestive category, noting that 96 per cent of articles reporting p values include one at or below 0.05. He endorses the change as a temporizing measure while naming collateral harms.
• Wasserstein, R. L., Schirm, A. L. and Lazar, N. A. (2019). Moving to a world beyond p < 0.05. The American Statistician, 73(S1), 1-19. doi:10.1080/00031305.2019.1583913 Introduces a special issue of 43 papers on what should replace the significance threshold, arguing for estimation, uncertainty and a stronger role for judgement. It carries the unit's constructive programme and its caution that the replacement is a practice, not a number.
• Ho, J., Tumkaya, T., Aryal, S., Choi, H. and Claridge-Chang, A. (2019). Moving beyond P values: Data analysis with estimation graphics. Nature Methods, 16(7), 565-566. doi:10.1038/s41592-019-0470-3 Presents the estimation graphic, a plot showing the full distribution of the data with the effect size and its interval. The authors argue the test confines the analyst to asking whether an effect exists while estimation asks how large it is.
• Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J. and Munafo, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376. doi:10.1038/nrn3475 Estimates the median statistical power of neuroscience studies at between about 8 and 31 per cent, and shows that low power reduces both the chance of detecting a true effect and the probability that a significant result is true. An underpowered study does not merely miss effects; it misreports them.
• Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. doi:10.1037/0033-2909.112.1.155 Provides tables of the sample size needed for 0.80 power at conventional small, medium and large effect sizes across eight standard tests. It is the reference a project uses to state, in advance, the smallest effect its design could detect.
• Bogdan, P. C. (2025). One decade into the replication crisis, how have psychological results changed? Advances in Methods and Practices in Psychological Science, 8(2). doi:10.1177/25152459251323480 Examined 240,355 empirical psychology articles from 2004 to 2024 and found the share of significant p values in the fragile band between .01 and .05 fell from 32 to 26 per cent, with typical sample sizes rising after 2015. It shows the reform changed reporting, and measures p values rather than conclusions.
• Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573-603. doi:10.1037/a0029146 Sets out Bayesian estimation for two groups, which returns distributions of credible values for the effect size and the group difference and allows the null to be accepted when the estimate is certain. It is the unit's alternative analysis, whose output is an estimate rather than a decision.