Choosing the method against the alternatives it rejects

A project defends its method twice. The first defence shows that the method answers the question. The second shows what the alternatives would have cost, and the second is the one proposals omit. A method is defensible only together with the alternatives it rules out. The evaluation literature shows the same instrument answering one question well and another badly. Method choice follows the question, and a method's goodness is visible only against a stated purpose and a named rival.

This fourth of ten units argues in three steps, and the fifth unit asks what the result will warrant. The question selects the method, and convention selects it badly (Greenberg and Buxton, 2008; Wixon, 2003; MacKenzie, 2013). Each alternative carries a cost the proposal must name (Charness, Gneezy and Kuhn, 2011). The measures and criteria by which methods are compared are choices whose meanings do not agree (Hornbaek, 2005; Hornbaek and Law, 2007; Hartson, Andre and Williges, 2003).

The question selects the method, and convention selects it badly

Greenberg and Buxton (2008) accept that usability evaluation has a significant role where conditions warrant it. They then set out four ways it fails when done by rule rather than by thought. It can kill creative ideas that defy current interface norms during brainstorming, or kill an inspired vision when run prematurely. It can suggest an academic prototype is worthy rather than critique how anybody would adopt it, or read today's cultural reluctance as a verdict. They conclude that the choice of evaluation methodology must arise from the problem under study.

Wixon (2003) gives the practitioner's version. The literature on evaluating methods is unhelpful or irrelevant, because it compares methods outside the context in which they are used and does not say which method serves which purpose. He therefore calls for field research into how methods work in practice. MacKenzie (2013) supplies the vocabulary, presenting human-computer interaction as an empirical research discipline. A study is not a method until the task, the variables and the measures are specified.

Naming the alternative means naming what it costs

Charness, Gneezy and Kuhn (2011) make the trade-off concrete. In a within-subject design each participant meets more than one treatment, so the estimate comes from how an individual's behaviour changes when the circumstances change. In a between-subject design each participant meets one treatment, and the estimate comes from comparing randomly assigned groups. Between-subject designs are more conservative but limited. Within-subject designs have more power but potentially suffer from confounds. The choice should therefore follow the question and its practical implementation.

A worked case. A project compares a wrist-worn pointing device with a mouse on a standard pointing task. The within-subject version has every participant use both devices, so fewer participants carry the comparison. The cost is using the second device after practice at the task. The between-subject version removes that carryover and pays with the conservatism the same paper names. Both versions assume a sample of working age who can wear the device.

The repertoire a proposal has to survey before it chooses

Cairns and Cox (2008) set the width of the choice. Their edited volume presents the field's methods. It has chapters on experimental design, eyetracking, qualitative methods, cognitive modelling and writing up as they apply in human-computer interaction. Each chapter comes from a researcher who uses examples from their own work, so the volume shows methods as used rather than as listed.

Kensing and Blomberg (1998) and Orlikowski and Baroudi (1991) show that the choice is not only procedural. Reviewing participatory design as a maturing research area and evolving practice, Kensing and Blomberg set out the politics of design and the nature of participation. They also set out the methods, tools and techniques for participation. Orlikowski and Baroudi examined 155 information systems research articles published between 1983 and 1988. They found the articles shared a single set of rarely explicit assumptions about what counts as valid knowledge, and they present interpretive and critical alternatives. A method carries commitments about who decides and assumptions about what counts as evidence.

Johnson and Onwuegbuzie (2004) add combination to the repertoire. They position mixed methods as the natural complement to qualitative and quantitative research, with pragmatism as its partner. They add mixed-model and mixed-method designs, an eight-step process and a claim that pluralism frequently yields superior research. Johnson, Onwuegbuzie and Turner (2007) then treat the definition of mixed methods as the problem, collecting leaders' definitions and searching for criteria of demarcation. Mixing is a design decision with a name, so answering mixed methods is not yet answering the question of method.

The measures and the criteria for comparing methods are both choices

Hornbaek (2005) reviewed the measures used in 180 studies in core human-computer interaction journals and proceedings. He found practice short of a valid account of usability as quality in use: approximately one quarter of the studies did not assess the outcome of the users' interaction.

Hornbaek and Law (2007) put a number on the assumption that measures agree. From the raw data of 73 studies they calculated the correlations among usability measures and found them generally low. The correlation was 0.247 between effectiveness and efficiency, and 0.196 between efficiency and satisfaction, and 0.164 between effectiveness and satisfaction. The 95 per cent confidence intervals were 0.188 to 0.306, 0.132 to 0.260 and 0.102 to 0.226 respectively. More complex measures attenuated the correlations, and perception measures were generally not correlated with objective measures. Changing the complexity of the task did not move the correlations at all. The standard questionnaires used to measure satisfaction held up better than the homegrown ones.

If the measures a project might use to decide that one interface is better than another are only loosely related, a comparison can be won on time and lost on satisfaction. The claim that a design is better is incomplete until the proposal names the measure.

Hartson, Andre and Williges (2003) carry the same demand to the criteria. Practitioners need to know which usability evaluation methods are more effective, in what ways and for what purposes, but the methods cannot be compared reliably because no standard set of criteria exists. They propose reliability, validity, thoroughness, effectiveness, cost effectiveness and downstream utility, and their argument is procedural: methods can be compared once the criteria are fixed in advance.

Contested question: can methods be compared, and on whose criteria?

The literature agrees that methods should be chosen against purposes and divides over whether they can be compared. Hartson, Andre and Williges (2003) hold that comparison is possible and blocked only by missing criteria they supply. Gray and Salzman (1998) examine five experiments comparing usability evaluation methods. They find design and conduct problems that call into question what the field thought it knew about their relative efficacy. Practitioners were already choosing methods on the strength of those results, so a small flaw in a comparison study carries weight it was never designed to bear. Wixon (2003) holds that the comparison literature asks the wrong question: not which method is better in the abstract but which serves the purpose at hand.

A fourth strand attacks the outcome being compared. Hornbaek and Law (2007) find the measures used to decide that one method beat another only loosely related, so the verdict depends on the measure chosen. Hornbaek (2005) reports that a quarter of the studies he reviewed did not assess the outcome of the interaction. That leaves comparative claims unsupported by their own data.

What would settle the dispute is not another head-to-head comparison of the kind Gray and Salzman faulted. It is a study that fixes the criteria in advance and states the purpose for which the methods are compared. It reports validity, thoroughness, cost and downstream utility, and pre-declares which measure carries the verdict. It also supplies the field study of methods in product development Wixon asked for. Neither source supplies that evidence, so a proposal simply inherits the duty to state its own criteria.

Read by kind, the sides differ in the evidence behind them. Hartson, Andre and Williges (2003) argue procedurally, from a gap in the criteria; Wixon (2003) argues from practitioner experience; Gray and Salzman (1998) rest on a review of five experiments; and Hornbaek and Law (2007) rest on a meta-analysis of 73 studies, the strongest single result here. While the dispute is open, a student should state the purpose of the comparison, name the rival design, and pre-declare which measure carries the verdict. The strongest objection is that Gray and Salzman remove the ground under the unit's own duty: where the comparisons the field already holds are unreliable, naming a rival may import an estimate the panel cannot use, whatever criteria are fixed in advance.

Where the argument is heading

Two moves are visible. Evaluation practice is being documented rather than assumed. Barkhuus and Rode (2007) analysed the approach to evaluation taken by CHI papers over 24 years. They reported an increase in the proportion of papers including evaluation, a decrease in the median number of subjects in quantitative studies, and an over-reliance on students. Their critique of who takes part goes further, since the samples they reviewed were also short of appropriate gender balance. A field whose default evaluation is a small study of students must argue for a method.

The second move is towards explicit combination and reporting. Johnson and Onwuegbuzie (2004) called mixed methods a third paradigm whose time had come. Johnson, Onwuegbuzie and Turner (2007) then treated the definition as the problem to be settled, because it decides what counts as a legitimate mixed design. The direction of travel is a method section that reads as an argument: a stated purpose, a named rival, declared measures and stated criteria. What remains unsettled is whether comparison will move from method against method to measure against measure.

The sources this unit rests on

Each note states what the work did, what it found and how it bears on the argument. Every identifier was resolved on 31 October 2026.

• Barkhuus, L. and Rode, J. A. (2007). From mice to men: 24 years of evaluation in CHI. doi:10.1145/1240624.2180963 Analysed 24 years of CHI evaluation against a schema for classifying type and scope. Found more papers including evaluation, a falling median number of subjects and an over-reliance on students. Shows that method practice drifts rather than settling.
• Cairns, P. and Cox, A. (eds) (2008). Research methods for human-computer interaction. doi:10.1017/cbo9780511814570 An edited volume whose chapters present experimental design, eyetracking, qualitative methods, cognitive modelling and writing up as they apply in the field. Carries the breadth requirement, that a proposal survey the repertoire before choosing.
• Charness, G., Gneezy, U. and Kuhn, M. A. (2011). Experimental methods: between-subject and within-subject design. doi:10.1016/j.jebo.2011.08.009 Compared within-subject and between-subject designs across experiments that sometimes agree and sometimes diverge. Found between-subject designs more conservative but limited and within-subject designs more powerful but exposed to confounds, so the choice follows the question.
• Gray, W. D. and Salzman, M. (1998). Damaged merchandise? A review of experiments that compare usability evaluation methods. doi:10.1207/s15327051hci1303_2 Examined five influential experiments comparing usability evaluation methods and found design and conduct problems that cast serious doubt on what the field thought it knew about their relative efficacy. Their outcomes were already advising practitioners, so comparisons must be read critically.
• Greenberg, S. and Buxton, B. (2008). Usability evaluation considered harmful (some of the time). doi:10.1145/1357054.1357074 Argue that evaluation by rule rather than thought kills ideas that defy current norms, kills an immature vision when run prematurely, and suggests worthiness instead of adoption on a prototype. The choice of methodology must arise from the problem under study.
• Hartson, H. R., Andre, T. S. and Williges, R. C. (2003). Criteria for evaluating usability evaluation methods. doi:10.1207/s15327590ijhc1501_13 Argue that methods cannot be compared reliably because no standard criteria exist, and propose reliability, validity, thoroughness, effectiveness, cost effectiveness and downstream utility. Methods become comparable once criteria are fixed in advance.
• Hornbaek, K. (2005). Current practice in measuring usability: challenges to usability studies and research. doi:10.1016/j.ijhcs.2005.06.002 Reviewed measures in 180 studies in core human-computer interaction journals and proceedings and found practice short of a valid account of usability as quality in use. About one quarter did not assess the outcome of the interaction.
• Hornbaek, K. and Law, E. L.-C. (2007). Meta-analysis of correlations among usability measures. doi:10.1145/1240624.1240722 Calculated correlations among usability measures from the raw data of 73 studies and found them generally low: effectiveness with efficiency 0.247, efficiency with satisfaction 0.196, effectiveness with satisfaction 0.164. Measures assumed to agree do not.
• Johnson, R. B. and Onwuegbuzie, A. J. (2004). Mixed methods research: a research paradigm whose time has come. doi:10.3102/0013189x033007014 Position mixed methods as the natural complement to qualitative and quantitative research, with pragmatism as its partner and designs for mixed-model and mixed-method studies. Carries the claim that the alternatives include combinations of methods.
• Johnson, R. B., Onwuegbuzie, A. J. and Turner, L. A. (2007). Toward a definition of mixed methods research. doi:10.1177/1558689806298224 Collected definitions from leaders in the field and searched for criteria of demarcation to derive a composite definition, because it decides what counts as a legitimate mixed design.
• Kensing, F. and Blomberg, J. (1998). Participatory design: issues and concerns. doi:10.1023/a:1008689307411 Review participatory design as a maturing research area and practice, setting out the politics of design, the nature of participation and methods for participation. A method embodies commitments about who decides.
• MacKenzie, I. S. (2013). Human-computer interaction: an empirical research perspective. https://archive.org/details/humancomputerint0000mack Presents human-computer interaction as an empirical research discipline, working through framing questions, designing experiments, measuring behaviour and reporting results. Gives the vocabulary in which a method decision becomes specific enough to have rivals.
• Orlikowski, W. J. and Baroudi, J. J. (1991). Studying information technology in organisations: research approaches and assumptions. doi:10.1287/isre.2.1.1 Examined 155 information systems articles published between 1983 and 1988 and found they shared a single set of unstated assumptions about what counts as valid knowledge, then presented interpretive and critical alternatives. A method choice carries assumptions a proposal must surface.
• Wixon, D. (2003). Evaluating usability methods: why the current literature fails the practitioner. doi:10.1145/838830.838870 Argues that the literature evaluating usability methods is unhelpful to practitioners because it compares methods out of context and does not say which serves which purpose. Calls for field research into methods in practice.