Interaction data are not normal: skew, tails and robust methods

A team times a checkout task. Most participants finish in under two minutes. One leaves the tab open and answers a call, and another takes ninety-four minutes on an older device over a metered connection. Task times, error counts and session durations are bounded below and open above, so they skew right with a long tail.

This sixth of ten units asks what an estimate is worth when the data are not normal, and argues that rank-based, trimmed and mixed-effects methods are not a fallback for data that failed an assumption. Each answers a different question from a difference of means, and the choice decides what claim the study can make. The workflow that tests for normality and switches method when it fails gets that wrong twice: it decides after seeing the data, and it treats a change of question as a repair.

Interaction data are skewed with heavy tails

Micceri examined 440 large-sample achievement and psychometric measures and found every one significantly non-normal at the .01 level (Micceri, 1989). Tail weights ran from the uniform to the double exponential, asymmetry was exponential in scale, and the shapes included digit preferences and multimodalities. His conclusion was that the tenets of normality-assuming statistics appear fallacious for such data.

The measures are achievement and psychometric tests, not task times from an interface, so applying the result to HCI is an argument rather than a measurement. Sauro and Lewis (2010) supply the direct measurement: across 61 large-sample usability datasets they found task times positively skewed, and reported that the geometric mean estimated the centre better than the median.

The assumption check is the wrong trigger

When the assumption is usually false, the usual response is to test for it and change method when the test fails. Erceg-Hurn and Mirosevich show why that understates the problem: analysis of variance and least squares regression are accurate only when their assumptions hold, and normality and homogeneity of variance are rarely met in real data, so the tests yield inaccurate p values, effect sizes and intervals (Erceg-Hurn and Mirosevich, 2008).

Leys and colleagues argue that the rule for handling outliers must be specified before the data are seen, because it affects validity and reproducibility (Leys et al., 2019), covering how outliers are classified, detected and retained.

A rank test claims an ordering, not a mean

Wilcoxon's procedure for two related samples compares the sum of signed ranks rather than the raw observations (Wilcoxon, 1945). Mann and Whitney extended the idea to independent samples with a statistic based on relative ranks (Mann and Whitney, 1947), and Kruskal and Wallis generalised it to several groups (Kruskal and Wallis, 1952).

What these tests license is a claim about stochastic ordering, whether observations from one group tend to be larger than those from another. That ordering is not a safer version of a difference of means: a team comparing three layouts can conclude that they differ in the ordering of times, but not that the first is faster by some amount.

The trimmed mean is an estimand the analyst defines

Trimming keeps the language of the mean and discards the observations that make it untrustworthy. Wilcox's reference work sets out the family, trimmed means, Winsorised variances, M-estimators and bootstrap methods, with the conditions under which each outperforms the classical procedures (Wilcox, 2012). The trimmed mean removes a fixed proportion from each tail, conventionally 20 per cent, and the Winsorised variance replaces the discarded values with the nearest retained value to give the standard error. Trimming discards the slowest fifth, often the poorest-connected users and those who work through assistive input.

Yuen built the two-sample test on that foundation, a trimmed t using Winsorised variances that tests location while remaining robust to non-normality and unequal population variances (Yuen, 1974). It estimates the population trimmed mean, a quantity defined by the rule the analyst chose in advance.

The bootstrap gives an interval without a distribution

An interval is the honest companion to these estimators, and the bootstrap supplies one without assuming a distribution. Efron's method estimates the sampling distribution of a statistic by resampling the observed data with replacement rather than assuming a parametric form (Efron, 1979), which gives intervals for quantities with no textbook standard error. Kirby and Gerlanc (2013) supply BootES, which computes bootstrap intervals for effect sizes of robust estimators such as the trimmed mean, though it does not supply the trimming rule: the interval is for the estimand that rule defines.

Mixed models: modelling the individual instead of averaging it away

Many interaction studies cannot treat observations as exchangeable, because each participant contributes several trials and each stimulus is seen by many. Baayen, Davidson and Bates set out mixed-effects models with crossed random effects for subjects and items. Their simulations showed the cost of ignoring that structure: models that omit item variability present in the data can have unacceptably high Type I error rates (Baayen et al., 2008).

Judd, Westfall and Kenny make the same case for stimuli, showing that averaging over them, the standard practice, produces anti-conservative tests (Judd et al., 2012). The mixed model changes the claim as well as the arithmetic: instead of one mean difference for a hypothetical average user, it estimates an effect while allowing participants and items to differ.

Contested question: is non-normality the right problem?

One position holds that the problem is settled: change the default method. Erceg-Hurn and Mirosevich argue that the assumptions of the classic tests are rarely met, so the robust alternatives should be in ordinary use (Erceg-Hurn and Mirosevich, 2008). Delacre, Lakens and Leys give the low-cost version. Welch's t controls the Type I error rate better than Student's t when the variances differ and loses almost nothing when they are equal, so they argue it should be the default (Delacre et al., 2017). A later simulation narrows that default: where the distributions are skewed and the sample sizes unequal, Welch's test inflates the Type I error rate, reaching about 9 per cent at a skewness of 3 against a nominal 5 per cent, and Student's t is no remedy (de Winter, 2026).

A second position holds that robustness is weaker than the first assumes. Bradley found the Type I error rate under assumption violation to depend on many interacting factors, so claims of robustness are particularistic and many would be false under any halfway reasonable quantitative definition (Bradley, 1978). The shapes Micceri found do not match the theoretical contaminations prior robustness research simulated, so the simulations that justify the robust methods may not describe the data they protect against (Micceri, 1989).

The third position comes from HCI itself. Elkin and colleagues validated ART-C, a contrast procedure for the aligned rank transform, on 72,000 synthetic datasets. Contrasts computed from ART inflate the Type I error rate, and ART-C does not. ART-C had more power than the t test, the Mann-Whitney U test, the Wilcoxon signed-rank test and ART itself (Elkin et al., 2021).

What would settle the dispute is a comparison on the distributions interaction research produces: all three positions rest on simulations with theoretical shapes, and Micceri's finding is that those shapes are not the shapes of real measures.

The evidence is thicker on one side: Micceri's audit covers real measures, Bradley's analysis is an argument about error rates, and the HCI validation is a large simulation with its own stated limits.

The strongest objection is that the unit's alternative rests on the evidence it criticises: robust methods are calibrated on theoretical distributions Micceri found in no real measure, so a designer who rejects the classical test has no stronger warrant for the replacement.

What to do while the dispute is open

While the dispute is open, state the estimator and the outlier rule before the data are seen, and name the quantity each test estimates and the assumption the chosen method takes on.

Where the argument is heading

The direction of travel is toward methods that model the structure of the data rather than discard it: mixed models with crossed random effects have moved from psycholinguistics into general practice (Baayen et al., 2008; Judd et al., 2012), and the HCI contribution is the aligned rank transform extended to contrasts (Elkin et al., 2021). The unresolved strand is Micceri's: the distributions used to justify robust methods are not the ones they handle (Micceri, 1989).

The sources this unit rests on

• Baayen, R., Davidson, D. and Bates, D. (2008). Mixed-effects modelling with crossed random effects for subjects and items. Journal of Memory and Language. doi:10.1016/j.jml.2007.12.005 They introduce mixed-effects models with crossed random effects for subjects and items and compare them by simulation with quasi-F, by-subjects and by-items analyses. Models that ignore item variability can have unacceptably high Type I error rates.
• Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology. doi:10.1111/j.2044-8317.1978.tb00581.x Bradley examines how the Type I error rate behaves under assumption violation and finds that it depends on many interacting factors, so claims of robustness are particularistic. Under any reasonable quantitative definition, many claims of robustness would be demonstrably false.
• de Winter, J. (2026). Neither the t-test nor Welch's test: a case for robust alternatives in two-sample comparisons with non-ideal data. Quality & Quantity. doi:10.1007/s11135-026-02909-5 He simulated two-sample comparisons across sample-size imbalance, variance and skewness and found Welch's test inflating the Type I error rate under skew, to about 9 per cent at skewness 3, while permutation-based Welch held the nominal rate. Limits: simulation.
• Delacre, M., Lakens, D. and Leys, C. (2017). Why Psychologists Should by Default Use Welch's t-test Instead of Student's t-test. International Review of Social Psychology. doi:10.5334/irsp.82 They report that Welch's test controls the Type I error rate better when variances differ and loses little when they are equal, and argue that testing the variances to choose between the tests is unreliable.
• Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics. doi:10.1214/aos/1176344552 Efron introduces the bootstrap, which estimates the sampling distribution of a statistic by resampling the observed data with replacement instead of assuming a parametric form. It is the tool that lets a robust estimator be reported with an interval rather than a bare point.
• Elkin, L. A., Kay, M., Higgins, J. J., et al. (2021). An Aligned Rank Transform Procedure for Multifactor Contrast Tests. ACM Symposium on User Interface Software and Technology (UIST). doi:10.1145/3472749.3474784 They present ART-C, a contrast procedure within the aligned rank transform, validated on 72,000 synthetic datasets: contrasts computed from ART inflate the Type I error rate while ART-C does not. Limits: six distribution types and no random slopes.
• Erceg-Hurn, D. M. and Mirosevich, V. M. (2008). Modern robust statistical methods. American Psychologist. doi:10.1037/0003-066x.63.7.591 They argue that analysis of variance and least squares regression are accurate only when their assumptions hold, that normality and homogeneity of variance are rarely met, and that using the tests anyway yields inaccurate p values, effect sizes and confidence intervals.
• Judd, C. M., Westfall, J. and Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology. doi:10.1037/a0028347 They argue that experimental stimuli should be treated as a random factor and show that averaging over stimuli produces anti-conservative tests, extending the independence problem from participants to the materials presented.
• Kruskal, W. H. and Wallis, W. A. (1952). Use of Ranks in One-Criterion Variance Analysis. Journal of the American Statistical Association. doi:10.1080/01621459.1952.10483441 They generalise rank-based testing to several samples by ranking the pooled observations and computing a statistic approximately chi-square distributed with one fewer degree of freedom than there are groups.
• Leys, C., Delacre, M., Mora, Y. L., et al. (2019). How to Classify, Detect, and Manage Univariate and Multivariate Outliers, With Emphasis on Pre-Registration. International Review of Social Psychology. doi:10.5334/irsp.289 Leys and colleagues classify outliers, argue that common detection methods are problematic, and recommend the median absolute deviation for univariate cases and the Mahalanobis-MCD distance for multivariate ones, their central claim being that the outlier rule must be pre-specified.
• Mann, H. B. and Whitney, D. R. (1947). On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics. doi:10.1214/aoms/1177730491 They propose a statistic based on the relative ranks of two independent samples for testing whether one variable is stochastically larger than the other, and derive its mean, variance and limiting normal form.
• Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin. doi:10.1037/0033-2909.105.1.156 Micceri examined the distributional characteristics of 440 large-sample achievement and psychometric measures and found every one significantly non-normal at the .01 level, with tail weights from the uniform to the double exponential. His findings did not support the theoretical distributions used in prior robustness research.
• Sauro, J. and Lewis, J. R. (2010). Average task times in usability tests. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1753326.1753679 They found task times in 61 large-sample usability datasets positively skewed and reported that the geometric mean estimated the centre with less error and bias than the median.
• Wilcox, R. R. (2012). Introduction to Robust Estimation and Hypothesis Testing. doi:10.1016/c2010-0-67044-1 Wilcox's reference work sets out the robust alternatives to mean-based inference, including trimmed means, Winsorised variances, M-estimators and bootstrap methods, with the conditions under which each outperforms the classical procedures.
• Wilcoxon, F. (1945). Individual Comparisons by Ranking Methods. Biometrics Bulletin. doi:10.2307/3001968 Wilcoxon proposes a rank-based procedure for two related samples that works from the sum of signed ranks rather than the raw observations, so the comparison does not require the measurements to be normally distributed.
• Yuen, K. K. (1974). The Two-Sample Trimmed t for Unequal Population Variances. Biometrika. doi:10.2307/2334299 Yuen develops a two-sample test based on trimmed means and Winsorised variances, robust both to non-normality and to unequal population variances. It shows that discarding the tails is a principled estimator with a defined population quantity rather than a way of removing inconvenient values.
• Kirby, K. N. and Gerlanc, D. (2013). BootES: An R package for bootstrap confidence intervals on effect sizes. Behaviour Research Methods. doi:10.3758/s13428-013-0330-5 They present an R package that computes bootstrap confidence intervals for effect sizes, including for robust estimators such as the trimmed mean, giving a robust point estimate an interval without a derived standard error.