The design decides which comparison the data can support

Two teams run the same experiment. One gives every participant both layouts, and the other gives each participant one. The first reports a comfortably significant difference from half the participants. The second, with the same hypothesis, is inconclusive. The first study is not the stronger one. This third of ten units moves from the instrument to the arrangement, and argues that the design decides which comparison the data can support.

A within-subjects arrangement removes the variability between people from the error term. That buys the sensitivity (Greenwald, 1976). The same arrangement exposes every participant to every condition. That buys practice and carry-over effects whose size is rarely measured (Greenwald, 1976; Charness, Gneezy and Kuhn, 2012). A long session also burdens every participant, including an older one and one who works the task through a screen reader. Counterbalancing spreads those effects over the conditions rather than cancelling them (Bradley, 1958; Brooks, 2012). The test that follows then arrives carrying an assumption that the between-subjects version does not need (O'Brien and Kaiser, 1985). The between/within decision is not an efficiency trade settled by the smaller p value. It fixes which comparison the data are entitled to make. No later test recovers a comparison that the design never licensed.

A design is an argument about what can be compared

Campbell (1957) specified the categories of extraneous variable and distinguished seven of them. He used them to evaluate the standard designs of social experimentation, and he separated internal from external validity. The main effects threaten the internal claim, and their interactions threaten how far the result travels. West and Thoemmes (2010) read the same territory through two traditions. Rubin's framework specifies the possible outcomes for each participant and the assumptions sufficient to estimate a causal effect.

Rubin (1974) concluded that randomisation should be used whenever possible. He added that carefully controlled nonrandomised data remain reasonable and necessary in many cases. Imai, King and Stuart (2008) single out the improper use of hypothesis tests on covariate balance between treated and control groups. They separate the consequences of randomisation from those of blocking and matching. Post-hoc adjustment reaches only the variables that were measured. Assignment handles the rest.

Repetition buys sensitivity and costs context

Greenwald (1976) states the trade in one sentence. Within-subjects designs gain statistical efficiency by removing subject variance from the error terms used to test treatment effects. They are faulted for practice, sensitisation and carry-over effects that may limit the interpretation of results. A between-subjects design is not free of context, but it has the context that a single treatment affords itself. Charness, Gneezy and Kuhn (2012) find the two arrangements sometimes yielding the same result and sometimes not. They call between-subjects designs more conservative, and they note that within-subjects designs support more powerful techniques and often match the theory under test more closely.

Counterbalancing distributes the order effects; it does not remove them

Counterbalancing is the standard answer to the order problem. Bradley (1958) showed that when the number of conditions is even, a Latin square can be constructed in which every condition is preceded by a different condition in every row. That is complete counterbalancing of immediate sequential effects. Brooks (2012) treats carry-over as the reason counterbalancing is not optional. He generates sequences by traversing an Euler circuit through a graph of conditions, and he reaches cases that a plain Latin square does not cover, including a condition following itself. One concrete instance is a task whose stimulus is chosen from the response to the last one. A plain Latin square can place each condition before every other, but it never places a condition immediately after itself. It therefore cannot produce the orders that design needs, and the Euler-circuit construction can.

A balanced Latin square neutralises the effect of what preceded a condition. It cannot touch an effect that depends on the set of conditions rather than their sequence. Poulton (1973) called these range effects. Exposure to a range of values lends contextual comparison to every scenario other than the first, so a treatment's effect depends on the other treatments offered alongside it. A counterbalanced order leaves the range as it was.

The repeated-measures test arrives with an assumption

A within-subjects design yields repeated measures. The ordinary univariate analysis then carries the sphericity assumption, that the differences between conditions have comparable variance, and interaction data rarely satisfy it. O'Brien and Kaiser (1985) set out a multivariate analysis of variance for repeated-measures designs, because the case is a family of routes rather than one procedure. Maxwell and Delaney (2003) supply the general form. The model comparison is derived from the design's structure, so a design that repeats measures carries its assumption set into whichever test is chosen. The multivariate route answers violated sphericity. It does not repair the design.

What a well-designed HCI experiment still cannot fix

MacKenzie (1992) reviewed Fitts' law as a research and design tool. He examined six Fitts' law studies using devices including the mouse, trackball, joystick, touchpad, helmet-mounted sight and eye tracker. His finding is that the performance measures were so inconsistent that comparisons across the studies were difficult, and that the sources of experimental variation had to be identified before the differences could be reconciled. The six studies he read differed in their tasks, their selection techniques, the range of conditions they ran and their handling of errors. They ran from Card, English and Burr's 1978 comparison of pointing devices to Ware and Mikaelian's 1987 work on head orientation and eye tracking. He traced the inconsistent throughput values to those choices rather than to the devices. The case shows a credible model defeated by measurement drift across studies. It does not show that Fitts' law fails. MacKenzie's own conclusion is that the sources of variation have to be reconciled before any cross-device claim can be made.

External validity: the design's second boundary

Campbell (1957) located the external-validity problem in the interactive effects of the extraneous variables and the experimental arrangements. The problem lies in the specifics of the setting, not the size of the sample. Mook (1983) is the counter-argument that this unit has to answer. Many investigations, he observed, are accused of failing to generalise because of sample bias or artificiality of setting, when such generalisations are often not intended. A laboratory study may test what ought to happen there, and an artificial finding can show what can occur even if it rarely does. The boundary between what a design licenses and what it does not is a fact. The fault lies in the claim that ignores it.

Contested question: is a within-subjects design the better default?

The literature disagrees about the size and tractability of the costs, not about their existence. The efficiency position holds that repetition pays. Greenwald (1976) sets out the gain from removing subject variance. He notes that practice, sensitisation and carry-over can be minimised by procedure when they are not wanted. He adds that ecological validity depends on how closely the research context matches the generalisation context. Charness, Gneezy and Kuhn (2012) support the power half of this position, while still calling between-subjects designs more conservative.

The caution position holds that the costs are structural rather than procedural. Poulton (1973) concluded that range effects are expected in within-subjects designs, so they should ordinarily be avoided. He allowed them only when between-subjects results bolster them, the conclusion Greenwald set out to moderate. Brooks (2012) treats serial carry-over as intrinsic to a system with memory, which is why counterbalancing is essential rather than remedial.

What would settle it has not been run. The test is a single study that builds both arrangements for the same hypothesis, estimates carry-over and range effects directly rather than assuming that counterbalancing has handled them, and reports the two estimates side by side. Washout trials and a calibration condition at two stimulus ranges would do it. Neither Greenwald (1976) nor Charness and colleagues (2012) reports that measurement. The claim that within-subjects is the better default is therefore a judgement about typical studies rather than a finding.

The evidence is uneven. Greenwald (1976) argues from the arithmetic of the error term, which is silent about the size of a carry-over. Poulton (1973) names a threat without measuring it, and Brooks (2012) supplies a construction rather than a magnitude.

The strongest objection is that the rule has no threshold. No source here puts a number on a carry-over, so a designer cannot tell a survivable order effect from a fatal one, and the demand to state what the design licenses becomes a reporting duty rather than a protection.

What to do while the dispute is open

While the dispute is open, fix the arrangement and the counterbalancing method before the data are collected. Where conditions repeat, measure the order effect rather than assume that counterbalancing removed it.

Where the argument is heading

The named work points one way. Maxwell and Delaney (2003) derive the analysis from the design's structure, which makes the arrangement of conditions the object of scrutiny. Brooks (2012) turns counterbalancing from a recipe into a computational problem with a general solution. O'Brien and Kaiser (1985) show that the repeated-measures case carries a choice of analytic routes. Cairns and Cox (2008) place the obligation on the researcher to match method to question. The design is the argument. The statistics are its arithmetic.

The sources this unit rests on

• Bradley, J. V. (1958). Complete Counterbalancing of Immediate Sequential Effects in a Latin Square Design. Journal of the American Statistical Association. doi:10.1080/01621459.1958.10501456 Constructed a Latin square for any even number of conditions in which each condition is preceded by a different condition in every row, with a proof. It supplies the counterbalancing the unit describes, which distributes sequential effects rather than removing them.
• Brooks, J. L. (2012). Counterbalancing for serial order carryover effects in experimental condition orders. Psychological Methods. doi:10.1037/a0029310 Notes that reactions are rarely independent of previous inputs, so serial carry-over makes counterbalancing essential, and reframes condition ordering as a graph problem solved by traversing an Euler circuit. It reaches cases a plain Latin square cannot cover, including self-adjacencies.
• Cairns, P. and Cox, A. L. (2008). Research Methods for Human-Computer Interaction. Cambridge University Press. doi:10.1017/CBO9780511814570 Assemble HCI research methods from controlled experiments through qualitative approaches and argue that the method must answer the question the researcher is actually asking. It is the HCI-specific statement of the thesis.
• Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin. doi:10.1037/h0040950 Specified the major categories of extraneous variable, distinguished seven of them, and used them to evaluate standard designs, separating internal from external validity. It originates the claim that design features, not analysis, decide which threats a study has ruled out.
• Charness, G., Gneezy, U. and Kuhn, M. A. (2012). Experimental methods: Between-subject and within-subject design. Journal of Economic Behaviour and Organisation. doi:10.1016/j.jebo.2011.08.009 Compare within-subject and between-subject experiments and find the two arrangements sometimes agree and sometimes not, concluding that between-subjects designs are more conservative while within-subjects designs support more powerful techniques. It states the trade the unit refuses to treat as convenience.
• Greenwald, A. G. (1976). Within-subjects designs: To use or not to use? Psychological Bulletin. doi:10.1037/0033-2909.83.2.314 Sets out the efficiency of within-subjects designs, which remove subject variance from the error terms used to test treatment effects, alongside practice, sensitisation and carry-over effects, and argues that between-subjects designs are not context-free either. It is the unit's central source for the bargain.
• Imai, K., King, G. and Stuart, E. A. (2008). Misunderstandings Between Experimentalists and Observationalists About Causal Inference. Journal of the Royal Statistical Society Series A (Statistics in Society). doi:10.1111/j.1467-985X.2007.00527.x Clarify fallacies of causal inference, single out the improper use of hypothesis tests on covariate balance, and contrast randomisation, blocking before randomisation and matching after assignment. It shows why assignment handles variation that adjustment reaches only when measured.
• MacKenzie, I. S. (1992). Fitts' Law as a Research and Design Tool in Human-Computer Interaction. Human-Computer Interaction. doi:10.1207/s15327051hci0701_3 Reviews Fitts' law as a research and design tool and examines six Fitts' law studies across devices including the mouse, trackball and joystick, finding the measures so inconsistent that cross-study comparison is difficult. It is the worked HCI case for the unit's claim.
• Maxwell, S. E. and Delaney, H. D. (2003). Designing Experiments and Analysing Data. Routledge. doi:10.4324/9781410609243 Present experimental design through a model comparison approach, deriving the analysis from the structure of the design rather than offering a catalogue of tests. It is the unit's statement that the model follows the design.
• Mook, D. G. (1983). In defence of external invalidity. American Psychologist. doi:10.1037/0003-066X.38.4.379 Argues that many investigations are accused of failing to generalise because of sample bias or artificiality when such generalisations were never intended, and that artificial findings are interesting because they show what can nevertheless occur. It is the strongest counter-argument to the thesis.
• O'Brien, R. G. and Kaiser, M. K. (1985). MANOVA method for analysing repeated measures designs: An extensive primer. Psychological Bulletin. doi:10.1037/0033-2909.97.2.316 Give a primer on multivariate analysis of variance for repeated-measures designs, documenting that the case is a family of analytic routes with their own assumptions rather than one fixed procedure.
• Poulton, E. C. (1973). Unwanted range effects from using within-subject experimental designs. Psychological Bulletin. doi:10.1037/h0034731 Argues that exposure to a range of values lends contextual comparison to every scenario other than the first, so a treatment's effect depends on the other treatments offered with it, concluding that within-subjects designs should ordinarily be avoided or supported by between-subjects results. It is why counterbalancing, which fixes sequence, leaves the range untouched.
• Rubin, D. B. (1974). Estimating causal effects of treatments in randomised and nonrandomised studies. Journal of Educational Psychology. doi:10.1037/h0037350 Sets out matching, randomisation and random sampling as ways of controlling extraneous variation, specifies the benefits of randomisation, and concludes that it should be used whenever possible while controlled nonrandomised data remain reasonable and necessary in many cases. It shows what assignment buys and the limits of that claim.
• West, S. G. and Thoemmes, F. J. (2010). Campbell's and Rubin's perspectives on causal inference. Psychological Methods. doi:10.1037/a0015917 Compare Campbell's threat-and-design approach with Rubin's specification of possible outcomes and sufficient assumptions, on randomised, broken randomised and observational studies. It shows the two traditions disagree about machinery, not about the need for a design argument.