A paper is a claim, its evidence and the inference between them

The opening unit asked what counts as a contribution; this one asks what a claim rests on. Read the abstract and you have a paper's claim; read the results and you have its evidence. The step between them is the inference, and a paper rarely states it. This unit argues that the distance between a claim and its evidence is not a rhetorical accident but a systematic product of how results are analysed and reported. A reader who cannot re-derive the inference cannot tell a warranted claim from an inflated one.

An argument's three parts, and the warrant left unwritten

Toulmin (1958) supplies the vocabulary. An argument has a claim, the data offered for it, and a warrant, the principle licensing the step from data to conclusion. "The technique is faster" is a claim. The selection times of eighteen participants are the data. The warrant is that this sample and this task stand in for the users the technique is for. Those users are assumed to be able-bodied, to read English and to have an uninterrupted connection, and the warrant says nothing about the difference. The warrant, not the data, carries the generalisation, and it is what papers most often leave implicit. Keshav (2007) turns the anatomy into a routine. It has three passes, from title, abstract and conclusions, through the figures and method, to a close reading that reconstructs the argument.

A published claim before it was tested

Bem (2011) reported nine experiments with more than a thousand participants. He read them as evidence of anomalous retroactive influence, a future event affecting a response already given. Eight of the nine produced significant effects, the mean effect size was 0.22, and in five the individual-difference measure correlated with performance. That is what a published claim can look like before it is tested: many studies, a consistent direction, a modest effect, and a warrant running from the pattern to a general claim.

Re-analysis and replication: two tests of the same inference

Wagenmakers, Wetzels, Borsboom and van der Maas (2011) answered with a re-analysis rather than a new study. They argued that part of Bem's analysis had been exploratory, and that one-sided p values overstated the evidence against the null. A default Bayesian test of the same data gave evidence for precognition that was weak to nonexistent. Ritchie, Wiseman and French (2012) answered with data. They ran three preregistered replications of Bem's ninth experiment, the largest effect, at three laboratories, with the sample the original power analysis required. All three failed; the combined sample was 150 and the combined one-tailed p was .83. Re-analysis tests whether the numbers licensed the claim, replication whether the result appeared again, and it is the replication that closed the case.

Spin widens the gap between result and claim

Not every gap is an accident of design. Boutron and colleagues (2010) examined 616 published reports of randomised controlled trials and found 72 with statistically nonsignificant results for the primary outcome. They coded each for spin, which they define as

specific reporting strategies, whatever their motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome. (Boutron et al., 2010)

Spin appeared in 13 titles (18.0 per cent), in the results section of 27 abstracts (37.5 per cent) and in the conclusions of 42 (58.3 per cent) (Boutron et al., 2010).

Misreading is measured, not assumed

The other half of the thesis, that routine misreading opens the gap, has been tested rather than assumed. Hoekstra, Morey, Rouder and Wagenmakers (2014) asked 120 researchers and 442 psychology students to judge six statements about a confidence interval, all of which were false. Both groups endorsed more than three on average, and experience with statistics did not help the researchers. One of the six read the interval as a range in which the true mean probably lies, which treats a fixed quantity as though it varied; the correct reading concerns the intervals an experiment would produce if repeated. Greenland, Senn, Rothman and colleagues (2016) catalogue 25 misreadings that recur in applied work. These run from treating a p value as the probability that the null hypothesis is true to reading a confidence interval as a range of probable values. The American Statistical Association states six principles, among them that a p value does not measure the size or importance of an effect (Wasserstein and Lazar, 2016).

The gap is also a property of the analysis

Button and colleagues (2013) estimated the median power of studies in the neurosciences at between about 8 per cent and about 31 per cent. Low power means a real effect is often missed, and that a significant result is more likely to be an overestimate, since a small study must have drawn an unusually large difference to reach significance at all. Ioannidis (2005) modelled the probability that a published claim is true as a function of power, bias and the ratio of true to false relationships tested. Under plausible values for small studies, it can fall below one half. Caine (2016) analysed every manuscript published at CHI 2014 and found sample sizes from 1 to 916,000, with 12 the most common. Ortloff and colleagues supply the reference a power analysis needs, extracting statistics from 1,692 quantitative CHI papers published from 2019 to 2023 and reporting effect-size distributions for twelve CHI research fields (Ortloff et al., 2025).

Contested question: a reporting failure or a property of the statistics?

The literature agrees that the gap exists and disagrees about what produces it. One position holds that the gap is made in the writing. Boutron and colleagues (2010) locate it in reporting strategies. On this view reporting standards protect the reader. An abstract that says "no significant difference" where the spin says "promising trend" is a different document. A second position holds that the gap is in the statistics. Greenland and colleagues (2016) argue that correct interpretation demands more attention to detail than working scientists can spare. Button and colleagues (2013) and Ioannidis (2005) locate the problem in design and analysis, where no reporting standard can reach. On that view the small, flexible study yields a claim that no careful sentence can make honest. Wasserstein and Lazar (2016) and Nickerson (2000) sit between the positions, holding that the tools can be used well, so the defect is practice rather than instrument. What would settle it is a measurement nobody has run. Take a corpus of papers. Have readers who did not write them re-derive each inference from the data alone. Code each departure of the stated claim for whether it follows from a design feature or a reporting feature. Boutron's corpus carries the reporting code and Button's the power calculation, and the two have not been applied to the same papers.

The two positions rest on different kinds of evidence. The reporting position rests on a coded audit of 616 published trial reports. The statistical position rests on a simulation, a survey of power estimates and a catalogue of recurring misreadings. The middle position rests on two reviews and a statement. Only the reporting side measures what authors wrote. While the dispute is open, a designer should re-derive the inference from the data before acting on the claim, and ask for the estimate and its interval rather than the verdict.

The strongest counter-argument: the evidence is borrowed

The strongest objection is that the thesis borrows its heaviest evidence from other fields and predicts a crisis in ours that nobody has measured. The median power of 8 to 31 per cent is neuroscience. The spin audit is clinical trials. The HCI evidence is thinner. Caine (2016) reports the field's ordinary samples rather than its errors, and Nickerson (2000) concludes that significance testing is misunderstood but usable, which would make the gap a correctable error of practice. In reply, the mechanism does not wait for a crisis. If small samples inflate effects, they inflate HCI effects too. The absence of an HCI spin audit is a gap in the evidence, not evidence of its absence.

Where the argument is heading

The direction of travel is away from the reported number and towards the inference it licenses. Wasserstein and Lazar (2016) present their statement as a step towards treating a result as something to be described rather than admitted or rejected. Caine (2016) contributes the field's own reference point, the local standard. A sample can then be judged against what CHI studies actually do rather than against a rule imported from medicine. Lakens (2022) turns that into an obligation at the point of design. Amrhein, Greenland and McShane (2019) published a comment in Nature asking that the significance threshold be abandoned as a decision rule. It carries no data, so it shows the reading is mainstream rather than correct.

The sources this unit rests on

Each note summarises what the cited document did and found, not a quotation, and every record was checked against the publisher's registered metadata on 1 November 2026. Where a published title contains a spelling the house style forbids, the British form is given.

• Toulmin, S. E. (1958). The uses of argument. doi:10.1017/CBO9780511840005 Sets out a layout for arguments that separates a claim, the data supporting it, the warrant licensing the step from data to claim, and the qualifier that bounds it.
• Keshav, S. (2007). How to read a paper. doi:10.1145/1273445.1273458 Gives a three-pass method for reading a research paper, from title, abstract and conclusions, through the figures and method, to a close reading. He states that the skill is rarely taught.
• Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. doi:10.1037/a0021524 Reports nine experiments with more than a thousand participants that he read as evidence of retroactive influence, with a mean effect size of 0.22 and significant results in eight of the nine.
• Wagenmakers, E. J., Wetzels, R., Borsboom, D., et al. (2011). Why psychologists must change the way they analyse their data: the case of psi. doi:10.1037/a0022790 Re-analysed Bem's data with a default Bayesian test, arguing that part of the original analysis was exploratory and one-sided p values overstated the evidence, and concluding that the evidence for precognition was weak to nonexistent.
• Ritchie, S. J., Wiseman, R. and French, C. C. (2012). Failing the future: three unsuccessful attempts to replicate Bem's retroactive facilitation of recall effect. doi:10.1371/journal.pone.0033423 Ran three preregistered replications of Bem's ninth experiment, the one with the largest effect at d = 0.42, at three laboratories. All three failed, with a combined sample of 150 and a combined one-tailed p of .83.
• Boutron, I., Dutton, S., Ravaud, P., et al. (2010). Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. doi:10.1001/jama.2010.651 Examined 616 published reports of randomised controlled trials and found 72 with statistically nonsignificant primary outcomes, coding each for spin. Spin appeared in 13 titles (18.0 per cent), 27 abstract results sections (37.5 per cent) and 42 abstract conclusions (58.3 per cent).
• Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. doi:10.1037/1082-989x.5.2.241 Reviews the long controversy over null hypothesis significance testing, cataloguing the objections and the common misinterpretations. He concludes that the test is easily misunderstood but that, applied with judgement, it can still aid interpretation.
• Ioannidis, J. P. A. (2005). Why most published research findings are false. doi:10.1371/journal.pmed.0020124 Models the probability that a published claim is true as a function of power, bias and the ratio of true to false relationships tested. Simulations show that in small studies a claim is more likely to be false than true.
• Button, K. S., Ioannidis, J. P. A., Mokrysz, C., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. doi:10.1038/nrn3475 Argues that low power reduces both the chance of detecting a real effect and the chance that a significant result reflects one, estimating the median power of neuroscience studies at between about 8 and about 31 per cent.
• Caine, K. (2016). Local standards for sample size at CHI. doi:10.1145/2858036.2858498 Analysed every manuscript published at CHI 2014 and reports sample sizes ranging from 1 to 916,000, with 12 the most common. She proposes judging a sample against the community's own local standard.
• Hoekstra, R., Morey, R. D., Rouder, J. N., et al. (2014). Robust misinterpretation of confidence intervals. doi:10.3758/s13423-013-0572-3 Asked 120 researchers and 442 psychology students to judge six statements about a confidence interval, all of which were false. Both groups endorsed more than three on average, and experience with statistics did not help the researchers.
• Greenland, S., Senn, S. J., Rothman, K. J., et al. (2016). Statistical tests, p values, confidence intervals, and power: a guide to misinterpretations. doi:10.1007/s10654-016-0149-3 Catalogues 25 recurring misinterpretations of p values, confidence intervals and power, including reading a p value as the probability that the null hypothesis is true.
• Wasserstein, R. L. and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. doi:10.1080/00031305.2016.1154108 States six principles for the use of p values, including that a p value does not measure the size or importance of an effect and that conclusions should not rest on whether a value crosses a threshold.
• Ortloff, A. M., Martius, F., Meier, M., et al. (2025). Small, Medium, Large? A meta-study of effect sizes at CHI to aid interpretation of effect sizes and power calculation. doi:10.1145/3706598.3713671 Extracted statistics from 1,692 quantitative CHI papers published from 2019 to 2023 and reports effect-size distributions for twelve CHI research fields, giving a power analysis a field-specific target. It supplies the reference the unit otherwise says is missing.
• Lakens, D. (2022). Sample size justification. doi:10.1525/collabra.33267 Sets out six approaches to justifying a sample size, including an a priori power analysis, planning for accuracy, resource constraints and an explicit acknowledgement that none is available. He argues a study is justified only if it says which applies.
• Amrhein, V., Greenland, S. and McShane, B. (2019). Scientists rise up against statistical significance. Nature. doi:10.1038/d41586-019-00857-9 Argues, in a comment signed by more than eight hundred researchers, that treating statistical significance as a decision rule is a cause of the reproducibility problem and asks that the threshold be abandoned. It carries no data of its own.