Quantitative methods and evaluation statistics

A number is not evidence until you can say what it measures. That is the first discipline of quantitative work in human-computer interaction, and the one most often skipped. A team reports that task completion rose by four percentage points after a redesign. The result can be real and still not mean what the team claims: completion rate, completion time and satisfaction are different measures, and a change can move one alone.

The order of the work matters as much as the arithmetic: choose what to measure, how to arrange the conditions, how many people to observe, which test fits, and how to report it so a stranger can judge it.

Measurement comes before the test

A scale is a set of rules for assigning numbers to observations, and those rules decide what arithmetic is legitimate. Counting errors gives a true zero and equal intervals, so a mean is meaningful. Ranking satisfaction as poor, fair, good, excellent gives order without equal spacing, so the median is honest and the mean is a fiction with a decimal point.

Validity asks whether the instrument measures the construct you named. Campbell and Fiske set out the standard test in 1959, the multitrait-multimethod matrix. Measure at least two constructs, each by at least two methods. Agreement between methods targeting the same construct is convergent validity, and those correlations should exceed correlations between different constructs measured by the same method (Campbell and Fiske, 1959). Reliability is necessary and not sufficient: a stopwatch that runs slow is reliable and wrong.

The gap between a proxy and the thing itself is where studies fail most often. Take a constructed case. A streaming team defines engagement as clicks per session and redesigns the home screen to invite clicking. Clicks rise and watch time falls, because the new screen interrupts playback. Clicks were a proxy for attention, and the proxy moved against the construct. The repair is a decision made earlier: measure the construct in two ways that can disagree.

Design decides which comparisons the data can support

A between-subjects design gives each participant one condition. It is immune to order effects and needs more people, because variability between participants stays in the error term. A within-subjects design gives each participant every condition. It removes that between-person variability and is more sensitive, but it brings a new problem: order and carry-over effects.

Order effects include practice, fatigue and learning; carry-over effects include anything a condition leaves behind, such as a remembered layout. Counterbalancing addresses both. With two conditions you alternate the order; with more, you use a Latin square so each condition appears equally often in each position. Counterbalancing does not remove carry-over, it spreads it evenly so it no longer favours one condition.

A control condition holds everything constant except the thing you are testing. Cairns and Cox chose controlled experiments to open their methods collection for this reason (Cairns and Cox, 2008). Take a constructed case: does animation improve comprehension? If the animated version also has larger text and a longer reading time, the control differs from the treatment in three ways, and a difference between them is uninterpretable.

Power is decided before data collection, not after

Power is the probability that a study detects an effect of a given size, if that effect is real. It is a function of four quantities: sample size, the significance criterion, the effect size, and power itself. Any one follows from the other three, which is why the calculation belongs at design time (Cohen, 1992).

Cohen reports the neglect he was trying to repair. Of 54 articles reviewed, only two mentioned power, and seven studies treating a null result as a finding had a median power of .25 (Cohen, 1992). One chance in four of detecting a real medium effect means the null result says very little.

Cohen's conventions for small, medium and large effects are widely used and widely abused. For the standardised difference between two means they are 0.20, 0.50 and 0.80; for a correlation they are .10, .30 and .50 (Cohen, 1992). He described medium as an effect visible to the naked eye of a careful observer, and said the values were subjective. A convention is not a finding about your system. If a change must deliver a five per cent gain to be worth shipping, design against that figure.

The rule is short: decide the smallest effect that would change a decision, estimate its variability from a pilot, then compute the sample size. If the number is unaffordable, change the study rather than the threshold.

Interaction data do not arrive normally shaped

Descriptive statistics are the argument, not the decoration, so report the shape before the summary. Task times, session durations and error counts are typically skewed, with a long right tail from a few slow or heavy users.

A mean drawn from such a distribution sits where almost no user is. The median is sturdier, and the tail often carries more useful information, because the slowest users are the ones who abandon the product. Summary and analysis are linked: means of skewed data let outliers drive the test, and deleting outliers after seeing the result is one of the analytic freedoms described below. Choose a summary in advance that fits the distribution.

Match the test to the design, then check what it assumes

The test follows from the design rather than from taste. Two independent groups on a continuous outcome give the t test for independent samples. Two conditions measured on the same people give the paired t test. Three or more conditions give analysis of variance, with a repeated-measures form for within-subjects work. Regression predicts a continuous outcome from several variables and will carry covariates and interactions (Field and Hole, 2003).

Every parametric test carries assumptions. Observations should be independent; the t test and analysis of variance also assume roughly normal distributions within groups and comparable variances across them. Repeated-measures analysis adds sphericity, the assumption that differences between conditions have equal variance. When sphericity fails, the F ratio is biased and the p value too small.

Non-parametric alternatives trade assumptions for a weaker question. The Mann-Whitney U test compares distributions rather than means, the Wilcoxon signed-rank test does the same for paired data, and the Kruskal-Wallis and Friedman tests extend this beyond two conditions (Field and Hole, 2003). The common mistake is to switch after a failed assumption check and then report the result as though it answered the original question. It answers a different question, about ranks, and the report must say so. A team comparing three layouts that finds unequal variances and switches to Kruskal-Wallis can conclude that the layouts differ in the ordering of times, which is weaker than the claim that one layout is fastest.

Effect sizes and intervals, and what a p value answers

A p value is the probability of data at least this extreme if the null hypothesis is true. It is not the probability that the null is true, and it is not the size of the effect. Cumming's case for the new statistics is that the field should move from a dichotomous decision to estimation: report the effect size, an interval around it, and how precise that interval is (Cumming, 2014).

The interval does work a p value cannot. It says how large the effect plausibly is and how much uncertainty remains, so a reader can judge whether the effect would matter. Two studies can both return p below .05 with intervals of very different width, and only the interval separates a solid estimate from a lucky small sample (Cumming, 2014). His eight-step strategy runs from a question phrased in estimation terms to a point estimate, an interval and open data.

The contested question is how far to go. Many researchers treat estimation as an addition and keep tests for the decision; Cumming argues the test should go. The evidence supports the first half strongly, that an interval is more informative than a verdict. Whether the test should be abandoned is a judgement about practice rather than a finding, and competent statisticians disagree.

The replication problem, and HCI's inherited findings

The Open Science Collaboration replicated 100 studies from three psychology journals. Ninety-seven per cent of the originals reported significant results and 36 per cent of the replications did. Replication effects were about half the size of the originals, and 47 per cent of original effects fell inside the 95 per cent interval of the replication (Open Science Collaboration, 2015). Success was better predicted by the strength of the original evidence than by the two teams.

The limits matter. These were psychology studies, and the replications used high-powered designs and original materials where available, so transferring the result to HCI is an argument rather than a measurement. Effect sizes can also decline for legitimate reasons, including publication bias in the originals.

Part of the explanation is not fraud but flexibility. Simmons, Nelson and Simonsohn showed that four ordinary analytic choices, which outcome to report, when to stop collecting, which covariate to include and which conditions to compare, can lift the false positive rate from the nominal 5 per cent to 60.7 per cent in their simulations (Simmons, Nelson and Simonsohn, 2011). They demonstrated it with real data, reporting that a Beatles song made participants about a year and a half younger by their own birth dates. The remedy is prespecification: state the plan, the stopping rule and the analyses beforehand.

For HCI the implication is uncomfortable. The field produces many small studies, few direct replications, and a habit of turning one laboratory result into a design rule. Read a single finding as one estimate with a wide interval, not as a settled fact.

A first look at Bayesian estimation

Bayesian analysis treats the parameter as uncertain, combining a prior distribution with the data into a posterior distribution. Wagenmakers and colleagues list ten advantages, and two matter most here. Bayesian hypothesis testing quantifies evidence for and against a hypothesis rather than only rejecting a null, and it lets the evidence be monitored as data accumulate without needing to know the intention behind the data collection (Wagenmakers et al., 2018).

A frequentist p value depends on the stopping rule, so checking results repeatedly and stopping at the first significant one inflates the error rate, which is the flexibility described above. Bayesian evidence does not depend on that intention in the same way. The costs are genuine, though: the answer depends on the prior, so the analyst must state and defend it, and a reader who rejects the prior will read the result differently.

Logs, funnels, and the industry A/B test

Interaction logs are observational rather than experimental, so they answer a different question. A session is a period of activity attributed to one user, and a funnel counts how many reach each step of a flow. The funnel's central confound is that its denominators are not comparable. Sessions that never reached the first step are absent, so a change that moves many users into the start of the flow can lower the conversion rate while raising the number who complete.

Kaplan and Meier built their product-limit estimate for exactly this class of problem: estimating how long something survives when some observations are incomplete because follow-up ended (Kaplan and Meier, 1958). A user who has not abandoned yet is censored, not a failure, and treating them as a failure biases every survival curve downwards. Their assumption that the lifetime is independent of the loss time deserves checking, since heavy users are often observed for longer.

The industry A/B test is the same controlled experiment at scale. Kohavi, Tang and Xu describe companies that each run more than 20,000 experiments a year, and set out the machinery: a primary metric and an overall evaluation criterion, trustworthiness checks that flag violated assumptions, and the pitfalls of carry-over effects (Kohavi, Tang and Xu, 2020). Twyman's law, that an interesting figure is usually wrong, is the discipline that keeps a large platform honest.

Two habits would improve most of the tests that ship: report the effect with an interval rather than only a verdict, and treat the ethics as part of the design. Which users receive the weaker version, is the optimised metric the one users would choose, and could the test be described to its participants without embarrassment?

Where a measurement concerns people whose access is already unequal, an average effect is the wrong instrument; the argument shifts to who a system excludes, and how the standards and the law require that exclusion to be tested.