Research through design and the evidence problem

Research through design answers a question most research methods avoid. When the contribution is a thing that was made, what does it let anyone know? Frayling (1993) supplied the vocabulary, separating research into, through and for art and design. HCI borrowed the middle term. That term carries the unit's claim, that designing can produce knowledge rather than only illustrate it.

This unit, the ninth of the ten, makes a claim that is true but narrow. The previous unit examined design that refuses to be used; this one asks what designing itself can know. Making can support statements about what is possible, and about what a designed thing does. It cannot support statements about how a population will behave, because a design project does not sample one. A claim about a population is a different kind of claim, not a harder one about an artefact. The field's response has been annotation of portfolios, an evidential form with obligations of its own.

Frayling's distinction names a mode of inquiry, not a topic

Research through design is not research about design, which studies designers, and it is not research for design, whose output serves a project. It is research in which the designing is the method.

Zimmerman, Forlizzi and Evenson (2007) turned the intuition into a model. They argue HCI had struggled to integrate design into research. They propose that designers contribute by producing novel integrations of research to make the right thing, moving the world towards a preferred state while addressing under-constrained problems. They offer four lenses for evaluating a contribution, an early attempt to say what a made thing is accountable to.

Fallman (2008) supplies the map. His triangle of design practice, design studies and design exploration holds that interaction design research moves among the three, each carrying different criteria. An exploration study is not failing to be design studies when it declines to generalise. What the triangle does not supply is the criteria themselves.

The claim it supports is about possibility, not about populations

The boundary-setting essay is Gaver's (2012). He argues that research through design will produce theory that is provisional, contingent and aspirational. He argues that theory should be treated as annotation of realised examples and portfolios. He warns against convergence and standardisation. The knowledge is of a particular kind, not the population-behaviour kind, and a field that standardised its claims would sacrifice the variety that produces them.

Koskinen and colleagues (2011) add the mechanism that makes the boundary usable. They set out design research through practice across lab, field and showroom. They argue each setting supports a different knowledge claim about designed things. A showroom study supports a claim about what a design does in a framed encounter. A field study supports a claim about how it fares in ordinary use.

The annotated portfolio answers the evidence problem, and needs documentation

If the claims are not general laws, what conveys them? Gaver and Bowers (2012) propose annotated portfolios. They argue the knowledge in a body of design work is communicated by describing the individual pieces and the relations between them. Its evidential unit is a set of artefacts, not a dataset. Its annotation is an interpretation the designer offers, not a measurement. Lowgren (2013) places the portfolio among competing candidates for intermediate-level knowledge, so it is one answer among several rather than the field's settled one.

The strongest competitor is the strong concept. Hook and Lowgren (2012) argue strong concepts are an intermediate-level form of design knowledge, more abstracted than an instance but not generalised theory. They argue this is where the knowledge sits. A strong concept is not a hypothesis about populations. It is a design idea that has proved tractable across cases, so it can be taught without being tested experimentally. Their two worked examples are social navigation, moving through a space by following the traces other people left, and seamfulness, treating a system's visible gaps and breaks as resources rather than defects.

Redstrom (2017) gives the philosophical version, arguing that design theory is made through design work rather than applied to it. The evidence problem is then misstated. Asking a design project for the evidence a survey produces asks for something it never claimed to make. The unit stops short of this, because Redstrom does not show how a made theory can be checked by anyone who did not make it.

An annotation is evidence only if another person can follow it back to the work. Bardzell and colleagues (2016) analyse how research through design processes are documented. They argue documentation is necessary for translating design knowledge into broader academic knowledge. A project that reports its final artefact, but not its rejected alternatives, criteria and turning points, leaves the reader an object and no reasoning, and a reader who cannot see the artefact has nothing to reason from.

The strongest objection is imported from another standard

The difficulty is not always the field's to solve. Dourish (2006) argues the ethnographic contribution to design is not a set of implications for design, but an understanding of the reasoning that shapes practice. He argues that the demand for implications truncates the account that made the method valuable. The objection, transposed onto research through design, is that the evidence problem is manufactured by importing a standard from elsewhere. The unit answers it only partly. Something is lost when a study is compressed into findings, so the honest response is to mark which claims are made and which are waived.

Contested question: should the field specify criteria or resist convergence?

The disagreement is not whether research through design produces knowledge. It is whether the field should specify that knowledge well enough to be scored. The literature divides into three positions.

The first position holds that criteria can and should be specified. Zimmerman, Forlizzi and Evenson (2007) supply four lenses for evaluating a contribution. Prochner and Godin (2022) build a framework of quality dimensions for planning, reporting and evaluation. Hook and Lowgren (2012) give the notion of knowledge the criteria would attach to. The standards are scattered, and the remedy is to write them down.

The second position holds that the criteria are the danger. Gaver (2012) argues the theory will be provisional and aspirational, and he warns against standardisation. The annotated portfolio of Gaver and Bowers (2012) is an interpretation of one body of work, not a rubric for any. A shared standard would select for the project that satisfies it, and the field would lose the diversity that generates its ideas.

The third position holds that the knowledge is produced in the making, and that a standard cannot capture it. Redstrom (2017) argues design theory is made through design work. Dourish (2006) argues the demand for transferable implications truncates what gave the work its value.

• Criteria can be specified. Zimmerman, Forlizzi and Evenson (2007), Hook and Lowgren (2012) and Prochner and Godin (2022): assessors given the criteria should agree on which of two projects is stronger. Low agreement on the same pair would falsify this.
• Criteria endanger the practice. Gaver (2012) and Gaver and Bowers (2012): a standard should narrow the range of work the field produces. A shared standard that still produces diverse work would falsify this.
• The making is the knowledge. Redstrom (2017) and Dourish (2006): stating the claim in transferable form should lose the content that mattered. An annotation that lets a later designer reproduce the reasoning would falsify this.

What would settle it is a study none of these sources reports. It would take published projects with their annotations, give assessors a stated criterion set, and record whether they agree on a ranking that predicts what later designers take from the work. The test is not whether the projects are good. It is whether the annotation carries the reasoning to a reader who was not there.

The three positions rest on different evidence, all of it argument rather than measurement. The specify-criteria position rests on a model with four lenses, a framework of quality dimensions and an account of intermediate-level knowledge, all proposals, though the framework was built from an analysis of how quality is judged. The resist-convergence position rests on one methodological essay and one proposal for annotated portfolios. The knowledge-in-the-making position rests on two arguments, one philosophical and one drawn from ethnography. The only counting relates to documentation, and it finds that reporting the process is what lets a claim travel to a reader. Until the test exists, the designer should document the process rather than the artefact alone, naming the rejected alternatives, the criteria applied and the turning points, and should mark which part of the reasoning a reader can follow back to the work. The strongest objection to the unit's thesis is that the boundary may be drawn too narrowly, since a portfolio that annotates many projects approaches a claim about what works across settings, and the insistence on the narrow reading may be what keeps the field's knowledge local.

Where the argument is heading

Named work points one way, towards partial instrumentation rather than generalisation. Prochner and Godin (2022) turn quality in research through design projects into a framework of dimensions for evaluation. Robinson and colleagues (2024) argue mid-fidelity prototypes in progress can be assessed as formative situations. Almendra and da Silva (2025) distinguish research through design from research by design. They argue that systematic and methodical investigation, not practice alone, constitutes research. Can a reader not in the studio reach the same judgement from the annotation?

The sources this unit rests on

• Zimmerman, J., Forlizzi, J. and Evenson, S. (2007). Research through design as a method for interaction design research in HCI. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1240624.1240704 Zimmerman, Forlizzi and Evenson propose that designers contribute by producing novel integrations of research to make the right thing, addressing under-constrained problems, and offer four lenses for evaluating a contribution.
• Gaver, W. (2012). What should we expect from research through design? Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/2207676.2208538 Gaver argues from two theories in the philosophy of science that research through design will produce theory that is provisional, contingent and aspirational, and should be treated as annotation of realised examples. No test is offered.
• Frayling, C. (1993). Research in art and design. Royal College of Art Research Papers. https://researchonline.rca.ac.uk/384/ Frayling distinguishes research into art and design, research through art and design, and research for art and design, the vocabulary HCI later borrowed. It names a mode of inquiry and supplies no criteria.
• Koskinen, I., Zimmerman, J., Binder, T., Redstrom, J. and Wensveen, S. (2011). Design Research Through Practice: From the Lab, Field, and Showroom. Morgan Kaufmann. https://www.sciencedirect.com/book/9780123855022/design-research-through-practice Koskinen and colleagues set out design research through practice across lab, field and showroom settings, arguing each supports a different kind of knowledge claim about designed things. It tests none of them.
• Bardzell, J., Bardzell, S., Dalsgaard, P., Gross, S. and Halskov, K. (2016). Documenting the research through design process. Proceedings of the 2016 ACM Conference on Designing Interactive Systems. doi:10.1145/2901790.2901859 Bardzell and colleagues analyse how research through design processes are documented and argue documentation is necessary for translating design knowledge into broader academic knowledge. It proposes what to record and when. It proposes rather than tests.
• Hook, K. and Lowgren, J. (2012). Strong concepts: intermediate-level knowledge in interaction design research. ACM Transactions on Computer-Human Interaction. doi:10.1145/2362364.2362371 Hook and Lowgren propose strong concepts as an intermediate-level form of design knowledge, more abstracted than an instance without aspiring to generalised theory, illustrated with social navigation and seamfulness. It is the main competitor to the annotated portfolio. Conceptual analysis.
• Fallman, D. (2008). The interaction design research triangle of design practice, design studies, and design exploration. Design Issues. doi:10.1162/desi.2008.24.3.4 Fallman proposes a triangle of design practice, design studies and design exploration, arguing interaction design research moves among the three. It supplies the unit's map and leaves the criteria unspecified.
• Gaver, W. and Bowers, J. (2012). Annotated portfolios. Interactions. doi:10.1145/2212877.2212889 Gaver and Bowers propose annotated portfolios, describing the individual pieces and the relations between them, as the way to communicate the knowledge in a body of design work. It makes a set of artefacts the evidential unit. A proposal with no measurement.
• Lowgren, J. (2013). Annotated portfolios and other forms of intermediate-level knowledge. Interactions. doi:10.1145/2405716.2405725 Lowgren places annotated portfolios among competing candidates for intermediate-level knowledge, alongside strong concepts and experiential qualities. It shows the portfolio is one answer among several, not the field's settled one. It is a commentary.
• Redstrom, J. (2017). Making Design Theory. MIT Press. https://mitpress.mit.edu/9780262036658/making-design-theory/ Redstrom argues design theory is made through design work rather than applied to it. It is the unit's strongest version of the third position, but he does not show how a made theory can be checked by others. Philosophical essay.
• Dourish, P. (2006). Implications for design. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/1124772.1124855 Dourish argues the ethnographic contribution to design is not a set of implications for design but an understanding of the reasoning that shapes practice. It is the unit's objection from outside. It is an argument, not a study.
• Prochner, I. and Godin, D. (2022). Quality in research through design projects: recommendations for evaluation and enhancement. Design Studies. doi:10.1016/j.destud.2021.101061 Prochner and Godin analyse the problem of judging quality in research through design projects and propose a framework of quality dimensions for planning, reporting and evaluation. It is a proposal, not a validated instrument.
• Robinson, R. B., Osborne, A., Ji, C., Fey, J. C., Dagan, E. and Isbister, K. (2024). That's not good science! An argument for the thoughtful use of formative situations in research through design. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. doi:10.1145/3613905.3644063 Robinson and colleagues argue mid-fidelity prototypes in progress can be assessed as formative situations and propose criteria for judging the rigour of such assessments. The criteria are proposed rather than tested. Position piece.
• Almendra, R. and da Silva, F. M. (2025). Research through design versus research by design: contribution to design research methodologies. In Advances in Design, Music and Arts III. Springer. doi:10.1007/978-3-031-74975-9_38 Almendra and da Silva distinguish research through design from research by design and argue that systematic and methodical investigation, not practice alone, constitutes research. It shows the boundary redrawn by argument. It offers no measurement.