Replication and generalisation in HCI

An HCI project result is narrow by construction: one interface, one task, one group of people, one afternoon, one connection.

The fifth of ten units argues that the narrowness is not a defect to hide but a boundary to declare, and the sixth asks what the numbers inside it support. The field's evidence shows that its results are conditional on who takes part and what they are asked to do, and that it has rarely checked a finding with a second study. A project therefore contributes a bounded claim, and the bound belongs in the sentences that make the claim.

The field has counted itself, and the count is small

The strongest evidence is the field's own audit. Hornbaek, Sander, Bargas-Avila and Grue Simonsen (2014) examined four outlets covering 891 papers and found that 3 per cent attempted to replicate an earlier result. Wilson, Mackay, Chi, Bernstein, Russell and Thimbleby (2011) had argued the case from practice, convening the RepliCHI panel. CHI rewards novelty, the community does not value replication, and the field treats a single study on 20 participants as true.

Replication arrives after the fact, and reproduction is not replication

Hornbaek and colleagues (2014) also found that most authors agreed their studies were replications yet had rarely planned them as such. The field treated replication as a confirm or do-not-confirm decision without effect sizes, and many non-replication studies could have corroborated earlier work at almost no cost, given a different analysis. Replication in HCI is scarce not because it is hard but because it is unplanned.

Echtler and Haussler (2018) analysed every CHI 2016 and CHI 2017 paper for open-source releases and found HCI had not followed the open-source trend. Iarygina, Hornbaek and Mottelson (2026) tried to reproduce the results of every CHI paper that had shared data and analysis code, fully reproducing 49 per cent, and argue that reproducibility, not replicability, is the achievable goal. The sample is small for a reason: of 9,435 CHI papers published between 2007 and 2024, only 1 per cent carried both data and analysis code, leaving seventy-six eligible.

HCI's measures do not travel as the field assumes

Generalisation also fails at the level of measurement. Hornbaek and Law (2007) meta-analysed correlations among usability measures from the raw data of 73 studies. Effectiveness and efficiency correlated at 0.247, efficiency and satisfaction at 0.196, and effectiveness and satisfaction at 0.164. Those are weak relationships among measures that usability models treat as facets of one construct, so the measures are not interchangeable and perceptions are generally uncorrelated with objective measures. If usability is not one quantity, an effect on one measure is not a general result about the construct. Ahsan, Shi, Becker, Ramos and Wilkinson (2026) find the same misalignment in a newer area, reviewing how co-creativity is measured in human-AI research and building a taxonomy of instruments that does not map onto the constructs named.

What an experiment can establish, and what its statistics do not

The reader judges a bounded claim against the experiment's design, not its test's significance. Hornbaek (2013) set out heuristics for good experiments: build hypotheses on existing work, select measures deliberately, craft challenging comparisons rather than biased win-lose setups, and rule out alternative explanations. Cairns (2007) shows how often that standard was missed. Of 80 papers across two BCS HCI conferences and two HCI journals, 41 used inferential statistics and all but one had a reporting or analysis problem. Cockburn, Dragicevic, Besancon and Gutwin (2020) report significance used as the evidentiary criterion in 61 of the 410 most-downloaded articles from 41 ACM Transactions journals. They argue that computing including HCI faces the threats seen elsewhere.

The sample is small, and its size is rarely argued for

The scale of a typical study bears on what it can support. Caine (2016) analysed every manuscript published at CHI 2014 and found sample sizes from 1 to 916,000, most commonly 12. A field whose modal study recruits twelve people can resolve only large effects, so null results are weak evidence and positive ones uncertain. Caine's remedy is local standards: a number justified against the study, not inherited.

A result is conditional on who took part and how the work was framed

Sonderegger and Sauer (2013) make that conditionality measurable. Sixty-four participants in Switzerland and East Germany carried out five typical tasks with a coffee machine, with the product presented as high or low value by manipulating its price. In Switzerland the participants rated the high-value product higher in usability. In East Germany they rated it lower, and the same interaction appeared in the emotion measures. Their conclusion is that usability outcomes do not transfer across cultures and that perceived product value mediates the result. The same interface produced opposite conclusions depending on who took part and how it was framed, and age, ability and language vary a sample the same way.

The default design bounds the reach of the claim

Two papers argue that the default evaluation tests the wrong question for many claims. Greenberg and Buxton (2008) accept that usability evaluation matters when conditions warrant it, but they argue that done by rule rather than by thought it can be harmful: it can kill creative ideas during design brainstorming, and suggest a prototype's worthiness rather than critique its adoption. Bargas-Avila and Hornbaek (2011) reviewed 51 user-experience publications from 2005 to 2009 reporting 66 empirical studies, finding a shift from work to leisure and from the desktop to consumer products, with context of use rarely researched and many studies using unvalidated self-developed questionnaires. A study built to the default template can be sound and still answer a question narrower than the one the paper implies.

The contested question: replicate more, or generalise less?

One position holds that the remedy is more replication. Wilson and colleagues (2011) call it a cornerstone of good science that CHI fails to reward. Echtler and Haussler (2018) show that releasing materials is the mechanism, and Cockburn and colleagues (2020) argue for statistical reform and openness. Iarygina and colleagues (2026) hold the measured version. They argue the field should secure reproducibility first, because the value of replicability remains contested.

The opposing position holds that more replication would reproduce the wrong thing. Hornbaek and Law (2007) show that usability is not one generalisable quantity, so a repeat of a usability finding repeats a local measurement. Greenberg and Buxton (2008) argue that by-rule evaluation tests the wrong question. On this reading the problem is too much generalisation. What would settle it is a pre-registered repetition of a named HCI effect that deliberately changes the interface, the task or the population. If the effect survives, the replication camp is right; if not, the bounded-claim position is.

Read by kind, the replication camp rests on one conference panel, one audit of open-source releases and one reproduction attempt that recovered half the eligible papers. The bounded-claim camp rests on a meta-analysis of seventy-three studies and an argument from practice. Until recently neither camp had run a repeat that deliberately changes a condition. Vanderdonckt and Vatavu (2025) ran four, each with a new sample of thirty participants in a different setting. Agreement rates barely differed across the replications, but thinking times did, so whether the effect travelled depended on which measure was read. While the dispute is open, a project should state its interface, task and sample as conditions inside the claim, report the smallest effect its design could detect, and decline to generalise past them. The strongest objection to this unit's thesis is that a boundary declared in every sentence can conceal a weak result, because a claim narrowed to its exact conditions may hold and still say nothing a reader can use.

What this means for a project

Take the common case: a laboratory comparison of two interfaces on one task with 24 participants. The paper may report the difference with an interval and the smallest effect the design could detect, and it may name the interface, task and sample as conditions. It may decline to generalise to users or to the design (Hornbaek, 2013; Caine, 2016). It may not claim that people prefer the new interface if preference was not measured, because Hornbaek and Law (2007) found perception measures largely uncorrelated with objective ones. The bound belongs in the abstract, not in a limitation at the end.

Where this is heading

The direction is towards tracking what can be checked. Reproducibility auditing is the newest development, with Iarygina and colleagues (2026) finding 49 per cent of eligible CHI papers reproducible. The mechanism arrives in parallel: Echtler and Haussler (2018) show the gap is one of released materials, and Cockburn and colleagues (2020) carry reform into computing venues. The measurement problem is met with a familiar instrument, as Ahsan and colleagues (2026) map where constructs and metrics diverge. The change-of-condition test is no longer missing: Vanderdonckt and Vatavu (2025) ran four such replications, and the result split by measure. Hornbaek and colleagues' (2014) evidence that cheap corroboration went unplanned suggests it should become routine.

The sources this unit rests on

Each note states what the cited work did and found and what it adds to the argument. Identifiers and metadata were checked against the Crossref, OpenAlex and Semantic Scholar records on 31 October 2026.

• Hornbaek, K., Sander, S. S., Bargas-Avila, J. A. and Grue Simonsen, J. (2014). Is once enough? On the extent and content of replications in human-computer interaction. Proceedings of CHI '14, 3523-3532. doi:10.1145/2556288.2557004 Examined four publication outlets covering 891 papers and found 3 per cent attempting replication. Replications were rarely planned in advance and were treated as confirm or do-not-confirm decisions, with effect sizes rarely analysed.
• Hornbaek, K. and Law, E. L.-C. (2007). Meta-analysis of correlations among usability measures. Proceedings of CHI '07, 617-626. doi:10.1145/1240624.1240722 Meta-analysed correlations among usability measures from the raw data of 73 studies. Effectiveness and efficiency correlated at 0.247, efficiency and satisfaction at 0.196 and effectiveness and satisfaction at 0.164, so the measures are not interchangeable.
• Hornbaek, K. (2013). Some whys and hows of experiments in human-computer interaction. Foundations and Trends in Human-Computer Interaction, 5(4), 299-373. doi:10.1561/1100000043 Sets out heuristics for designing, running and reporting experiments in HCI, including challenging comparisons rather than biased win-lose setups and designs that rule out rival explanations. Supplies the standard for a general conclusion.
• Cockburn, A., Dragicevic, P., Besancon, L. and Gutwin, C. (2020). Threats of a replication crisis in empirical computer science. Communications of the ACM, 63(8), 70-79. doi:10.1145/3360311 Analysed the ten most-downloaded articles from 41 ACM Transactions journals and found statistical significance used as an evidentiary criterion in 61 articles across 21 journals. Argues that computer science, including HCI, faces the same threats to replication as elsewhere.
• Cairns, P. (2007). HCI... not as it should be: Inferential statistics in HCI research. Proceedings of BCS-HCI '07, 195-201. doi:10.14236/ewic/hci2007.20 Surveyed inferential statistics across two BCS HCI conferences and two HCI journals, covering 80 papers of which 41 used them. All but one had a reporting or analysis problem that undermined the test.
• Caine, K. (2016). Local standards for sample size at CHI. Proceedings of CHI '16, 981-992. doi:10.1145/2858036.2858498 Analysed every manuscript published at CHI 2014 and found sample sizes from 1 to 916,000, most commonly 12, varying by setting and method. Argues for local standards, a size justified against the study rather than inherited.
• Wilson, M. L., Mackay, W., Chi, E., Bernstein, M., Russell, D. and Thimbleby, H. (2011). RepliCHI: CHI should be replicating and validating results more. CHI '11 Extended Abstracts, 463-466. doi:10.1145/1979742.1979491 The panel that opened the replication debate in HCI, arguing that CHI rewards novelty, that the community does not reward replication, and that a single study on 20 participants is treated as true. Proposes how CHI should facilitate replication.
• Greenberg, S. and Buxton, B. (2008). Usability evaluation considered harmful (some of the time). Proceedings of CHI '08, 111-120. doi:10.1145/1357054.1357074 Argues that usability evaluation done by rule rather than by thought can be harmful, naming design brainstorming, early system design and prototype verification as settings where it misfires. Calls for critique fitted to the design claim instead.
• Sonderegger, A. and Sauer, J. (2013). The influence of socio-cultural background and product value in usability testing. Applied Ergonomics, 44(3), 341-349. doi:10.1016/j.apergo.2012.09.004 Ran a coffee-machine study with 64 participants in Switzerland and East Germany, varying the product's apparent price. High-value products were rated higher in usability in Switzerland and lower in East Germany, so outcomes do not transfer across cultures.
• Bargas-Avila, J. A. and Hornbaek, K. (2011). Old wine in new bottles or novel challenges? A critical analysis of empirical studies of user experience. Proceedings of CHI '11, 2689-2698. doi:10.1145/1978942.1979336 Reviewed 51 user-experience publications from 2005 to 2009 reporting 66 empirical studies, finding a shift from work to leisure and from the desktop to consumer products, with context of use rarely researched and many studies using unvalidated questionnaires.
• Echtler, F. and Haussler, M. (2018). Open source, open science, and the replication crisis in HCI. CHI EA '18, Paper alt03. doi:10.1145/3170427.3188395 Analysed the entire body of CHI 2016 and CHI 2017 papers for open-source releases and compared HCI with bioinformatics. Finds HCI has not followed the open-source trend and proposes publication practices to improve rigour.
• Iarygina, O., Hornbaek, K. and Mottelson, A. (2026). On the computational reproducibility of human-computer interaction research. Proceedings of CHI '26. doi:10.1145/3772318.3791129 Identified every CHI paper that shared study data and analysis code and attempted to reproduce the results, fully reproducing 49 per cent. The authors argue that reproducibility, not replicability, is the achievable goal.
• Vanderdonckt, J. and Vatavu, R.-D. (2025). Context is key for reproducibility of empirical studies in human-computer interaction. Proceedings of the 3rd ACM Conference on Reproducibility and Replicability (ACM REP '25), 41-50. doi:10.1145/3736731.3746142 Ran four new replications of a smart-ring gesture elicitation study, each with thirty new participants, a different team and a different setting. Average agreement rates barely differed, but thinking times did, so the finding's transfer depended on the measure. It is the unit's change-of-condition test.
• Ahsan, M., Shi, W., Becker, D. V., Ramos, G. and Wilkinson, D. (2026). How do we measure creativity? A systematic review and taxonomy of evaluative methods in human-AI co-creativity. Human-Computer Interaction, 1-32. doi:10.1080/07370024.2026.2720237 Systematically reviews how co-creativity is measured in human-AI research and synthesises the instruments into a taxonomy. Finds constructs and measures often conflated and calls for context-sensitive measurement.