MSci project: delivery, evaluation and defence

A project is judged in its second half, when a plan meets a study that will not cooperate. Delivery produces something messier: sessions that ran long, a protocol revised after the first few participants, a log with a column you did not expect to need. None of that is failure. It is the raw material of the paper, provided you recorded it while it happened.

The contested question is what the work may claim. One position treats the paper as the report of a test fixed in advance: the hypothesis, the analysis and the sample size are settled before the data arrive, and anything added later is labelled as added (Cockburn, Gutwin and Dix, 2018; Chambers, 2013). The other treats the paper as a contribution of a stated kind, where the value lies in the knowledge produced and the type decides how the work is judged (Wobbrock and Kientz, 2016). The positions are compatible but pull the write-up differently: a project that produced a prototype and an unexpected finding must decide what it is claiming, then defend that choice to a reader who was not there.

A study that cannot be reconstructed was not really run

The record is the part of the work the reader never sees, and the part that makes the rest possible. Blandford, Furniss and Makri draw an analogy with a documentary film. Films were once shown as finished products, with the production hidden; the recent habit is to go behind the scenes and show the work. Qualitative research has the same gap: there are many guides to a named method and few accounts of how a method was adapted to a setting or how an unexpected problem was handled (Blandford, Furniss and Makri, 2016).

Suppose you run a think-aloud study and, two participants in, you notice your prompt invites a running commentary that answers a different question from the one you asked. You rewrite it, and the later sessions are no longer the same instrument as the earlier ones. That is not a blemish to conceal; it is a version boundary. The wording, the instrument file and the date belong in the record, as do the decision to drop a condition, the codebook revised halfway through, and the reason one session was set aside.

Two habits make the record usable: one dated decision log that gives the reason for each decision, and separate files for observation and inference.

Planned analysis and emerged analysis must both be visible

Cockburn, Gutwin and Dix give the failure a name. HARKing stands for hypothesising after the results are known, and it is what happens when an introduction is rewritten so that a finding discovered in the data appears to have been predicted all along. When intentions are preregistered, reviewers can be confident that the evidence offered for a hypothesis is not the product of that rewriting (Cockburn, Gutwin and Dix, 2018).

One mechanism is the registered report. A manuscript is reviewed before data collection, including its hypotheses, procedure, analysis pipeline and an a priori power analysis; if it survives review it is accepted in principle, so the decision to publish no longer depends on the results (Chambers, 2013). Chambers names the limit: post hoc analyses are welcome, and are distinguished from those planned in advance (Chambers, 2013). The same editorial cites a survey in which 67 per cent of researchers admitted selectively reporting experiments that produced desirable outcomes, and 72 per cent admitted hunting for a p value (Chambers, 2013). Those self-reports come from another field, so treat them as context, not as a measurement of HCI.

Hornbaek and colleagues found the same discipline missing in HCI: when asked, most authors agreed that studies they had run were replications, but they had rarely planned them as such (Hornbaek, Sander, Bargas-Avila and Simonsen, 2014). So label the boundary yourself. Your proposal planned one contrast between two conditions; the data also show that the effect depends on the input method used. Report the second finding, keep the contrast as planned, and mark the interaction as exploratory. You may not rewrite the introduction so that the interaction becomes the hypothesis you always had.

A contribution is a claim of a stated kind

Wobbrock and Kientz answer the question a reviewer asks first: what is the knowledge here, and what kind is it? They set out seven contribution types in HCI: empirical, artifact, methodological, theoretical, benchmark or dataset, survey, and opinion. Each is judged by different criteria. Empirical contributions are assessed on the importance of the findings and the soundness of the methods. Artifact contributions on what they make possible and how. Methodological ones on utility, reproducibility, reliability and validity (Wobbrock and Kientz, 2016).

Their account is also evidence about the field's shape. At CHI 2016, empirical studies of people were 28.4 per cent of submissions and had the highest acceptance rate of any type, 26.3 per cent; artifact or system papers were 24.2 per cent; dataset contributions were least submitted, at 1.3 per cent, with the lowest acceptance rate, 16.7 per cent (Wobbrock and Kientz, 2016). The authors say activity should not be confused with value: a method or theory paper may be relied on for years (Wobbrock and Kientz, 2016).

The worked case is a project that produced a toolkit and a small study of how people used it. It has two possible papers and must choose one as primary. If the toolkit is the contribution, the paper is judged on what it makes possible and on whether the design judgement is explained. If the study is the contribution, it is judged on its findings and method, and the toolkit becomes the instrument. Stating the type is how the reader knows what to accept or dispute.

The related work positions the project, it does not list it

A related work section that reads as an inventory has done no work. The CHI contribution criteria ask whether the paper clearly reviews what is already known and where that knowledge stops (ACM CHI, 2027). That is a demand for position, not coverage.

The worked case is three prior systems that do something close to yours. A list describes each in turn and stops. A positioning paragraph states what each assumes, what each leaves unresolved, and where your project stands. The same move applies to a design project. Zimmerman, Forlizzi and Evenson argue that research through design produces novel integrations of existing knowledge in the attempt to make the right thing, its value resting on the reasoning behind the artefact rather than the artefact alone (Zimmerman, Forlizzi and Evenson, 2007). A related work section for such a project must say which prior designs solved the problem differently, and why you chose the trade-off you chose.

The reproducibility package is the argument made checkable

The manifesto for reproducible science treats reproducibility as one property among several that decide whether a result can be trusted, alongside the methods, the reporting, the evaluation, and the incentives that reward them (Munafo et al., 2017). Its candour about its own evidence is worth copying: the evidence for the likely effectiveness of these measures comes, the authors write, from simulations and empirical studies, and broad adoption will require iterative evaluation (Munafo et al., 2017). A package is not a guarantee of truth. It is the part of the argument a reader can check.

Two sources define what it contains. The FAIR principles ask that data be findable, accessible, interoperable and reusable, putting weight on machine-actionable reuse rather than on prose description alone (Wilkinson et al., 2016). The ACM artifact policy defines an artifact as a digital object the authors created for the study or the study generated, naming software systems, input datasets, raw data and analysis scripts. It attaches three independent badges: artifacts available, artifacts evaluated, and results validated (ACM, 2020).

The worked case is the analysis script that regenerates your second table from the raw logs and writes the numbers the paper reports. It is the highest-value item, because it turns your analysis into something a reader can run. Ethics often prevents sharing raw data from a study of people, which is a reason to share the script, the instrument, the codebook and the derived data, and to state plainly what is withheld and why. A gap that is explained is usable; a gap that is not mentioned is not.

What the finding does not show

HCI has many small studies and few replications, and the scale is documented. Hornbaek and colleagues examined four publication outlets covering 891 papers and found 3 per cent attempting to replicate an earlier result. Where replications were done, they were usually treated as a confirm or do not confirm decision, with little analysis of effect sizes (Hornbaek, Sander, Bargas-Avila and Simonsen, 2014). A project result sits almost alone in a literature that rarely checks it.

The consequence is a duty on the paper's own prose. Consider a laboratory study of twenty-four students, comparing two interfaces on one task. The abstract may say that participants were faster with one under those conditions. It may not say that people prefer the design, because preference was not measured, nor that the design is better in general, because the setting, the sample and the task were all narrow. The viva question that exposes overreach is the shortest one: what does this finding not show?

Wobbrock and Kientz supply the general rule: the type of contribution fixes the scope of what may be claimed. An artifact contribution is judged on what it makes possible, so a project need not have produced an effect at all, but it must say where its knowledge stops (Wobbrock and Kientz, 2016).

An artefact is a contribution only when its knowledge is extracted

Where the project produced something a reader can run, download or handle, that object is a public artefact, and the temptation is to let it stand in for the argument. Zimmerman, Forlizzi and Evenson treat the artefact as the outcome of a design process whose contribution lies in the process and its reasoning, not in the object by itself (Zimmerman, Forlizzi and Evenson, 2007).

The worked case is a released prototype. A reader can install it, which is a genuine form of openness, and it settles nothing on its own. The paper has to extract the knowledge the making produced: the problem it addresses, the trade-offs resolved and left open, and what a different design would have cost. Without that extraction the artefact is a demo.

The viva is a conversation with someone who did not design the study

Defence is not a recital of the work. It is an exchange with a reader reconstructing your method, and the questions follow from the gaps that remain. Three recur. Why this sample, and what would a different one have changed? Why this analysis, and what would have led you to another? What result would have made you abandon the claim? A candidate with a decision log answers all three from the record, not from memory (Blandford, Furniss and Makri, 2016).

Cockburn, Gutwin and Dix make the boundary between planned and emerged work defensible, and that is what a viva tests. If you can point to what was registered in advance and what you added later, the examiner can tell a study that found something unexpected from one rewritten to look prescient (Cockburn, Gutwin and Dix, 2018). Chambers adds that an analysis added afterwards is legitimate once it is labelled as such.

The hardest questions are about your own study, and the honest answer is usually the strong one. Suppose an examiner suggests your measured difference could be an order effect, because every participant saw one condition first. If you counterbalanced the order, say so and point to the check. If you did not, say so, name the consequence for the claim, and say what the next study would do differently. Inventing a defence the data do not support is the one answer that loses the exchange.

What the next study would be

A project that ends with a deliverable has answered the question the proposal set. A project that ends with knowledge answers a question and opens another, so the final section of the paper should name the study that would test what this one could not. That is what separates work that produced knowledge from work that produced an artefact, and it is where the project hands on to the research career it was designed to begin.