Research questions, decisions and the shape of a study

A study earns its place by naming the decision it will change. It also names in advance the observation that would count as an answer. That is the thesis of this first of the ten units, and it is a demand on method rather than on technique. If the decision is stated first, the method follows from it. A question about how many people do a thing points towards a survey. A question about why an error happens points towards observation. A question about whether a change worked points towards a comparison. The decision may serve a product, a clinic or a public service, and the people it affects include those with no reliable connection and no time to spare. If the method is chosen first, the study answers the question the technique can reach. The question the decision posed goes unanswered.

The menu is large, well catalogued, and stocked with instruments that feel like progress because they have a name. This unit argues that the field's own reviews show what choosing from the menu first costs.

A study is an answer to a decision, not a procedure

Lazar, Feng and Hochheiser (2017) map the range of HCI research designs onto the kinds of question each can answer. The designs run from controlled experiments through surveys to case studies and ethnography. They argue that the method is a consequence of the question. A design settled before the question is fixed produces data that cannot resolve it.

Wobbrock and Kientz (2016) set out seven contribution types that appear across HCI publication. They argue that authors and reviewers should name the type being made. Naming the type is a discipline because it is checkable. An empirical contribution claims to have measured something. An artefact contribution claims to have built something. A study that cannot say which it offers has not yet decided what its answer is.

Colusso, Jones, Munson and Hsieh (2019) add the harder half of the point. They triangulate the literature on the research-practice gap with interviews and an analysis of how HCI knowledge travels. They propose a translational science model, in which findings are deliberately adapted for practice rather than merely published. They argue that the gap is better understood as a translation problem than as practitioners failing to read. A study can name its decision and still change nothing.

The answer is fixed before the data, or it is not an answer

Stating the decision is the first half of the demand. Stating what observation would count as an answer is the second, and it is the half that disciplines a study. That second half is this unit's own and stricter demand, because a threshold chosen after the data is chosen to fit the data. Wobbrock and Kientz (2016) make the same point through the contribution type. A study that claims an empirical contribution has promised a measurement, and a measurement implies a rule for what would count against the claim.

The demand has a cost: many useful-looking studies fail it. A survey that asks whether users would like a feature can yield a number. No observation could show that number to be wrong, because no decision was named for it to inform. Greenberg and Buxton (2008) describe a real instance at the other end. An evaluation run to verify an academic prototype reports how that prototype performed there. They argue that it can suggest a design's worthiness rather than offer a critique of how it would be adopted in practice. A question that cannot fail is a description. A study that cannot report a result against itself has not been designed to answer anything yet.

The menu of methods is real, and it is uneven

A method-first habit is tempting because the menu exists and is well stocked. Vermeeren and colleagues (2010) collected 96 user experience evaluation methods from academia and industry. They analysed them by product development phase and by the period of experience studied. Their analysis found the field short of early-stage methods and of methods for social and collaborative experience. It concluded that many current methods have no established practicability or scientific quality. The inventory argues for choosing deliberately, and it warns that a large menu is not a sound one.

Roto and colleagues (2011) make the matching problem explicit. They argue that traditional usability methods cannot evaluate user experience, because experience has many dimensions. They set out a taxonomy to help practitioners match a method to the aspect of experience and the stage of design they need to study. Their contribution is a teaching device rather than an empirical study. The taxonomy asks what is being measured before a technique is picked.

What the field measures, and why it drifts from the question

If methods were chosen from the question, the measures a study carries would track the quality in use that the decision depends on. Hornbaek (2006) reviewed how usability was measured in 180 studies from core HCI journals. He found the choice of measures narrow and poorly reasoned, with weak evidence that what was measured captured usability as quality in use.

The same drift appears when a field rebrands. Bargas-Avila and Hornbaek (2011) systematically reviewed 51 empirical user experience publications from 2005 to 2009, covering 66 studies. They integrated the products studied, the dimensions of experience measured and the methods used. They found that studies labelled user experience often measure the same constructs that earlier usability work measured. Many of the 66 studies used self-developed questionnaires whose items and statistical validation were not reported. Emotions, enjoyment and aesthetics were the dimensions most often assessed. A new name on the menu does not by itself change what a study can answer.

Woolrych and colleagues (2011) diagnose the mechanism. Most usability research, they argue, proposes evaluation methods as indivisible packages. That conceals which component actually produces an effect, and it blocks cumulative knowledge. They propose decomposing methods into ingredients that can be separately evaluated and recombined. Choosing a method from a menu is therefore choosing a bundle whose active part is unknown. A question names the ingredient needed, and a menu names only the bundle.

A study can be the wrong move

The strongest objection to this unit's thesis is that a study can be the wrong move whatever the question. Greenberg and Buxton (2008) argue that the convention that every design process must include usability evaluation is harmful some of the time. Evaluation is expensive and can be premature. It can substitute for design judgement, or answer a question nobody had. Run during brainstorming, it can kill ideas that do not conform to current interface norms. Run prematurely in early design, its list of issues can kill a promising direction. It should not validate very early or culturally sensitive work. On their account the discipline of naming a decision in advance cuts both ways. If no decision turns on the answer, the honest conclusion is that the study should not be run. The thesis licenses refusing a study as much as designing one. Refusing the study is the same demand applied one step earlier, to the decision to collect at all.

Contested question: does the question choose the method, or the method the question?

The literature agrees that method and question must be matched. It disagrees about which of the two is doing the choosing. Zimmerman, Forlizzi and Evenson (2007) propose research through design as a legitimate research method for HCI. They argue that designing an artefact can produce knowledge the behavioural and technical traditions cannot reach. They set out four lenses for judging that knowledge, with three worked examples of designs that produced it. On that position a method can generate the question, because the artefact discloses a problem that was not visible before it existed. A proposal to fix the question first would exclude exactly the work the position defends.

Gaver (2012) keeps the contest live by asking what research through design should be expected to produce. He reviews two theories from the philosophy of science. He argues against judging research through design by the standards of science or of design practice alone. He proposes documented annotation of the design process as a way of making the knowledge in a design public. Gaver does not settle what the standard is. He shows that the field has not agreed one. Reeves (2015) widens the doubt to the whole discipline. He traces how HCI imported cognitive science's model of knowledge and what that import excluded. He argues that the field's coherence does not depend on settling its scientific status.

The two positions can be told apart, and the evidence that would settle them is not a further philosophical argument. A decisive study would follow a set of projects from question to outcome. For each project it would record whether the question was fixed before the method, or the method disclosed the question. It would then compare which kind changed a decision. A second study would test the translational claim directly. It would give two groups of practitioners the same finding. One group would see the finding adapted for practice, the other as published. It would then measure whether either changes what they do. If question-first projects change more decisions, the thesis holds. If made artefacts change decisions that no prior question anticipated, the rival account is the better description of design research.

The evidence on each side differs in kind. Question-first rests on method textbooks and on the field's reviews of what it measures. Artefact-first rests on a philosophical argument and a handful of documented designs. Neither has run a test, so the dispute stays open. While it is open, the rule a designer follows does not change. Fix the decision and the observation in advance for any empirical claim, because that discipline pays either way. Where a project begins from an artefact, record the question it disclosed and the decision it serves. The strongest objection is that the field has not agreed what counts as knowledge, so an answer fixed in advance may answer the wrong question.

Where the argument is heading

Two lines of work set the direction. Colusso and colleagues (2019) propose a translational science model in which findings are deliberately adapted for practice rather than merely published. That makes translation itself a research contribution. Blandford and colleagues (2018) distil lessons from interdisciplinary research on interactive digital health interventions. They argue that research quality in that setting depends on a team holding several epistemologies at once. Identifying user needs and evaluating an intervention demand different standards of evidence. Together they point away from a single correct method and towards a study that states which decision it serves.

The sources this unit rests on

• Lazar, J., Feng, J. and Hochheiser, H. (2017). Research Methods in Human-Computer Interaction. Morgan Kaufmann. https://shop.elsevier.com/books/research-methods-in-human-computer-interaction/lazar/978-0-12-805390-4 Maps HCI research designs onto the questions each can answer and argues that the method follows from the question. It is this unit's set text.
• Vermeeren, A. P. O. S. et al. (2010). User experience evaluation methods: current state and development needs. NordiCHI '10. doi:10.1145/1868914.1868973 Collected 96 user experience evaluation methods and found the field short of early-stage and social methods, many lacking established practicability. The menu is uneven.
• Roto, V., Vermeeren, A., Vaananen-Vainio-Mattila, K. and Law, E. (2011). User Experience Evaluation - Which Method to Choose? Lecture Notes in Computer Science. doi:10.1007/978-3-642-23768-3_129 Argues that traditional usability methods cannot evaluate experience and offers a taxonomy matching a method to the aspect of experience and stage of design. It makes matching an explicit decision.
• Hornbaek, K. (2006). Current practice in measuring usability. International Journal of Human-Computer Studies. doi:10.1016/j.ijhcs.2005.06.002 Reviewed how usability was measured in 180 studies and found the measures narrow and poorly reasoned, with weak evidence they captured quality in use. It shows the drift between framing and measure.
• Bargas-Avila, J. A. and Hornbaek, K. (2011). Old wine in new bottles or novel challenges. CHI '11. doi:10.1145/1978942.1979336 Reviewed 51 user experience publications covering 66 studies and found they often measure the same constructs as earlier usability work. The novelty is in the label.
• Woolrych, A., Hornbaek, K., Frokjaer, E. and Cockton, G. (2011). Ingredients and Meals Rather Than Recipes. International Journal of Human-Computer Interaction. doi:10.1080/10447318.2011.555314 Argues that methods are treated as indivisible packages, concealing which component produces an effect, and proposes decomposing them into ingredients. Choosing a method picks a bundle of unknown parts.
• Greenberg, S. and Buxton, B. (2008). Usability evaluation considered harmful (some of the time). CHI '08. doi:10.1145/1357054.1357074 Argues that obligatory usability evaluation is harmful some of the time, because it is expensive, can be premature, and can answer a question nobody had. The honest move can be not to run the study.
• Zimmerman, J., Forlizzi, J. and Evenson, S. (2007). Research through design as a method for interaction design research in HCI. CHI '07. doi:10.1145/1240624.1240704 Proposes research through design as a legitimate method, arguing that making an artefact can produce knowledge the behavioural and technical traditions cannot reach. A method can generate the question.
• Gaver, W. (2012). What should we expect from research through design? CHI '12. doi:10.1145/2207676.2208538 Asks what research through design should produce and argues against judging it by science or design practice alone. It shows the standard is unsettled.
• Wobbrock, J. O. and Kientz, J. A. (2016). Research contributions in human-computer interaction. Interactions. doi:10.1145/2907069 Sets out seven contribution types across HCI publication and argues authors should name the type they make. It gives the vocabulary for what a study claims.
• Reeves, S. (2015). Human-computer interaction as science. Aarhus Series on Human Centred Computing. doi:10.7146/aahcc.v1i1.21296 Traces how HCI imported cognitive science's model of knowledge and what that import excluded. It questions the evidence standard this unit inherits.
• Colusso, L., Jones, R., Munson, S. A. and Hsieh, G. (2019). A Translational Science Model for HCI. CHI '19. doi:10.1145/3290605.3300231 Triangulates the research-practice gap with interviews and proposes a translational model in which findings are adapted for practice. It explains why a named decision may still not move.
• Blandford, A. et al. (2018). Seven lessons for interdisciplinary research on interactive digital health interventions. DIGITAL HEALTH. doi:10.1177/2055207618770325 Distils lessons from interdisciplinary digital health research and argues quality depends on a team holding several epistemologies. It shows the thesis where disciplines must agree a question.