User research methods

Every design decision is a claim about the world, from "people will find this button" to "this will reduce missed appointments". Research is how a guess becomes evidence, and its point is the decision it changes.

No method is neutral: each sees something and is blind to something else. An interview reports what a person can say about their behaviour, an observation what they do, and a survey what they are willing to endorse on a form. Choosing a method means choosing which account counts as evidence.

The field disputes that ranking. One camp holds that only observed behaviour is evidence, since an account is coloured by memory and the wish to please. The other holds that a person describes meaning no observer can see, so the account is data in its own right. Neither has settled it. Say which evidence you offer, and never offer one kind as a substitute for the other.

A newer claim has entered that debate. A language model given a persona will answer a survey in a plausible voice. Agnew and colleagues (2024) reviewed the proposals to replace human participants with model-generated data. The review argues that their promise of cheaper, more diverse data conflicts with what participation is for. A model answers from its training data, not from a life, so a simulated participant is a hypothesis, not evidence.

A question you can answer beats one you can only admire

"How do people use our service" cannot be answered. It names no behaviour, no boundary and no outcome. A constructed case shows the shape of a better question. A council keeps losing residents part way through its online recycling form. Its question is narrower: at which step do applicants stop, and what do they do next? An answer might read "step three, where the form demands a garden-waste subscription number most households do not have". That names a population, a behaviour, a setting and a decision.

A workable question says in advance what would count as an answer. Write that down before you collect, because it guards against finding what you hoped for.

Two kinds of study follow: generative work asks what the problem is, evaluative work whether a solution works. Mixing them produces nothing usable.

Feasibility belongs to the question: who can you reach, for how long, and with whose permission?

Who you ask decides what you can learn

Sampling is the decision about who stands for the people your design will affect. Probability sampling gives every member of a defined population a known chance of selection, which allows you to generalise. Convenience sampling takes whoever is reachable. It is acceptable only if you say so, and you must not generalise.

The 1936 Literary Digest poll is the classic failure. It mailed ten million straw ballots to its subscribers, telephone directories and car registrations, and about 2.4 million came back. It forecast a Landon landslide, but Roosevelt won. The usual explanation is frame bias: the lists reached mainly wealthier households. Lusinchi (2012) reopens the case and finds telephone and car owners in fact favoured Roosevelt, so nonresponse rather than the frame carried most of the error. That revision is contested, since it rests on one retrospective poll and the frame account is still taught. Size cannot repair a list that misses the people you need.

Convenience costs you systematically: you reach the articulate, the nearby and the willing. Recruit with a screener (a short set of questions that decides who qualifies), and set the incentive so that cost is not the filter.

Henrich, Heine and Norenzayan (2010) found that broad claims about human psychology rested on Western, educated, industrialised, rich and democratic samples. Those participants are frequent outliers, not a neutral standard. The strongest counter is that the critique overstates its case, since many effects do replicate across populations and its comparisons often set one project against another. A study run only with students leaves out the people whose literacy, language or device is the reason the form fails.

How many participants is a contested question, not a rule

The shorthand is that five users are enough. The figure comes from a model, not a commandment: Nielsen and Landauer (1993) fitted a Poisson process to eleven studies and found diminishing returns. The model assumes each participant finds each problem independently, and the strongest objection is that real participants share knowledge and miss the same things.

Faulkner (2003) tested it directly. She ran sixty users and drew random sets of five from the whole. Some sets found ninety-nine per cent of the problems, others only fifty-five per cent. At ten users the worst set found eighty per cent, and at twenty users ninety-five per cent. The spread, not the average, is the finding.

Caine (2016) analysed every paper published at CHI in 2014. Sample sizes ran from one to nine hundred and sixteen thousand, and the most common was twelve. Her rule is that the number follows the claim: a quantitative claim needs a power calculation, and a qualitative claim needs saturation, where new sessions stop producing new themes.

Saturation is contested too, since some hold that it cannot be known before the study starts, so the number becomes a judgement made afterwards. State which logic you used. This is an area of weak evidence, with many small studies and few replications.

Interview about what people did, not what they say they would do

Interviews are structured, semi-structured or unstructured. A semi-structured interview uses a guide of topics rather than a script, letting the participant take the account somewhere you did not plan.

Technique is learnable: ask open questions one at a time, and leave silence after an answer, because the second sentence usually carries the weight. Avoid leading questions such as "how confusing did you find it", and double-barrelled questions. Do not ask people to predict their own future behaviour.

Goodman, Kuniavsky and Moed (2012) treat the interview as a process with its own preparation, wording and pitfalls.

A constructed exchange shows the difference. Asked whether she would use a deadline reminder, a student gives an opinion. Asked when she last missed a deadline, she describes the evening she found the portal had closed. That answer carries a behaviour and a context.

Watch the work, because workarounds do not appear in an interview

Observation in the setting, with questioning alongside it, is contextual inquiry. The participant is the expert on their own work, and the researcher learns it rather than testing them. Shadowing follows a task from start to finish, not a demonstration.

The reason to look is the workaround. A receptionist keeps a paper list beside the screen, because the search is slower than memory. She will not offer the list as a finding, because she no longer sees it as anything. An observer watching one handover sees it inside a minute.

Goodman, Kuniavsky and Moed (2012) treat the field visit as a technique with its own preparation, note taking and etiquette, and say that observation answers different questions from a usability test.

That preference for watching is disputed. A long tradition treats the interview as the participant's own interpretation, which observation cannot supply. Nor is watching free of distortion: the observer is part of the scene, so a participant can perform, and one site is one site. Use both, and treat a disagreement between them as the finding, not a tie to break.

Diaries catch what a single session loses

An interview is a snapshot of a life reconstructed after the fact, so a study that needs the event itself cannot rest on one. A diary study asks participants to record events as they happen, and experience sampling prompts people at set or random moments to report a state, a rating or a short note (Goodman, Kuniavsky and Moed, 2012).

These methods reach what one session cannot: events that are rare, private or spread across a day, and the distortion of hindsight is cut. The costs are real. Diaries are a burden, so entries go missing and participants drop out, and a prompt can arrive at an inconvenient moment and produce a guess. Self-report stays self-report.

A constructed case shows when they earn that burden. A study of medication reminders needs to know what happened at eight in the morning, not what the participant remembers at six. Design for compliance: keep each entry short, give one clear cue, pilot the burden, and decide in advance what to do about missing entries.

Surveys and focus groups manufacture findings as easily as they collect them

A survey reaches many people cheaply, which is why it is easy to do badly. Each item should measure one thing. "The app is fast and easy to use" measures two, so the answer cannot tell you which failed. Keep the wording plain, avoid negations, and hold the scale constant.

Two biases are permanent. Acquiescence is the tendency to agree with a statement whatever it says, and a mix of positively and negatively worded items can expose it. Social desirability is the tendency to answer in the way that looks good. Tourangeau, Rips and Rasinski (2000) describe survey answering as four stages: comprehension, retrieval, judgement and reporting. Social desirability sits at the last, where an answer is edited for acceptability.

A bad survey is worse than no survey, because it produces numbers, and numbers end arguments.

Focus groups trade depth for interaction, and the group rather than the individual is the unit of analysis (Goodman, Kuniavsky and Moed, 2012). The conversation surfaces shared norms, vocabulary and reasons. Group norms can silence dissent. What a group will not say in company is a finding in itself. A show of hands is no independent count of opinion.

Consent, power and data are method, not paperwork

Informed consent tells the participant the purpose, what participation involves, what is recorded, who will see it, the right to withdraw, and how to complain. Withdrawal needs a stated cut-off, since merged accounts are hard to unpick.

The Facebook emotional contagion experiment shows the failure directly. In 2014 Kramer, Guillory and Hancock altered the News Feed of 689,003 users for a week, to test whether friends' emotions changed what people posted. Nobody was asked. The authors offered Facebook's Data Use Policy, accepted at sign-up, as consent, and a board classed the work as internal company research. PNAS published it, then added an expression of concern (Kramer, Guillory and Hancock, 2014).

The authors and their defenders argued that this was ordinary industry testing under a consent users had accepted. That argument fails on the point that matters: a clause accepted once at sign-up names no study and offers no chance to refuse one. Treat routine testing on identifiable people as subject to the same rule.

Power runs through recruitment. A student recruiting classmates, a manager recruiting staff, and a researcher paying for time all create a relationship in which saying no is costly. An incentive becomes coercive when it is large for the person receiving it.

Data protection is a design task. Collect the minimum, store it securely, keep identifiers apart from the data, and set a deletion date. A small sample cannot really be anonymous.

The British Psychological Society (2021) grounds this in four principles: respect for the autonomy, privacy and dignity of individuals, groups and communities, scientific integrity, social responsibility, and the maximising of benefit and the minimising of harm.

Recording, analysis and the honest account of failure

Decide in advance what a session must capture: timestamps, quotations, observed behaviour, and the mood of the room. Record audio where consent allows, because memory reconstructs and notes select.

Analysis begins with the first pass, not the end. Read the notes, tag them by topic, group the tags into themes, and hunt for counterexamples. Blandford, Furniss and Makri (2016) describe this as a craft with its own standards, judged differently from an experiment. The field disputes its goal. Some hold that an ethnographic study must end in implications for design, while others hold that describing the context is valuable on its own. Dourish (2006) argues that case directly, and Blandford, Furniss and Makri (2016) survey the dispute.

Triangulation checks one finding against another method, source or analyst. When a survey and an observation disagree, the disagreement is often the most useful result.

Report the study as it happened. Say who took part and who refused, what you changed midway, what went wrong, and what you cannot claim. The next module takes that evidence into the studio, where it becomes a brief and a prototype that will itself be tested.