Analysis, rigour and the honest account of what was found

Data carry no warrant of their own. What makes a finding credible is the analysis and the account given of it. In this last of the ten units, credibility in qualitative work rests on an analysis a reader can audit. The reader follows the path from raw material to claim, and sees what would have counted against it. It also rests on quality criteria stated in the study's own terms, not on the reliability apparatus of an experiment. The same demand falls on quantitative work. The replication difficulties documented in psychology and in empirical computer science show that an unreported analysis path manufactures findings whatever the data type.

Analysis is a procedure, not the residue of data collection

The phrase "the themes emerged" hides the work. Braun and Clarke (2006) wrote the paper most HCI qualitative work now follows. Its central move was to treat thematic analysis as a method in its own right rather than a residual step after data collection. They set out six phases, from familiarising yourself with the data and generating initial codes to defining and naming themes and producing the report.

The phases iterate, so reviewing a theme sends the analyst back to the codes and coding back to the transcripts. Writing is analytic: a theme is generated rather than discovered ready-made, and its name is part of its claim.

Blandford, Furniss and Makri (2016) give the HCI-specific version. They cover planning, sampling, analysis, reporting and quality, with checklists for each stage. The essential work happens behind the scenes, so if it is not reported a reader cannot tell a careful study from a careless one. Two reports of the same eight interviews differ. The first lists themes with one quotation each and no record of how they were built. The second shows the codes, the merged themes and the theme abandoned when a counterexample appeared. Only the second lets a reader disagree with the reasoning rather than the taste.

Quality is an argument, and broad criteria are prompts for one

Braun and Clarke (2021) state the position the dispute turns on. Quality in reflexive thematic analysis cannot be reduced to a checklist, they argue, because the technique is not separable from the analyst's interpretative judgement. Reviewers who demand coding reliability measures as universal requirements assume a homogeneity the method does not have. Their criteria are conceptual clarity, engagement with the data and analytic depth, supported by ten common problems and twenty questions for reviewers. Asking whether two coders agreed asks whether the coding could have been done by a machine. Asking whether the analysis earned its conclusion is a different question.

Reflexivity is a documented practice, not an assertion

One criterion runs through the scheme: the researcher's own position. Finlay (2002) gives the account this unit uses. Reflexivity is the researcher's explicit, self-aware analysis of their own role, needed to evaluate how intersubjective elements influence data collection and analysis. It must be documented to add anything. Traditions treat it in three ways. It can be a confessional account of method, or an examination of the researcher's reactions. It can also be an analysis of how researcher and participant construct the account. Recorded as a decision trail, it shows how the researcher's presence shaped who was asked and how an answer was read. It also shows what the researcher assumed about a participant's age, first language or access needs, because those assumptions decide what is coded as data and what is dismissed as noise.

Adequacy is a relation between a claim and its evidence

Sandelowski (1995) refuses both easy answers about how many participants. A sample can be too small to support a claim of redundancy or saturation. It can also be too large to permit the deep, case-oriented analysis that is the point of the method. Adequacy is a judgement weighing the information collected against the uses to which it will be put. A claim that most participants found a form confusing may be drawn from five interviews never designed to support a frequency. That is not a small-sample problem. It is a mismatch between claim and evidence.

Flexible analysis manufactures findings in any paradigm

It is tempting to treat the demand for an honest account as a concession qualitative work makes. The evidence runs the other way. Simmons, Nelson and Simonsohn (2011) simulate four common researcher degrees of freedom. They show the nominal five per cent false-positive rate rising as the freedoms are combined. One freedom is selecting among dependent variables, and another is collecting extra observations after an initial test. A third is choosing whether to control for a covariate, and a fourth is deciding which conditions to report. Combined, they reached a false-positive rate of 60.7 per cent. Their demonstration is a real study of 20 undergraduates who listened to either When I'm Sixty-Four by The Beatles or an instrumental control. The undergraduates then reported the date of birth that fixed their age. By measuring several dependent variables, checking the result after roughly every ten participants, and controlling for father's age, they obtained a significant effect. Participants were nearly a year and a half younger after the song, 20.1 years against 21.5. Without the covariate the difference shrinks and is no longer significant. Another measure in the same paper moved the other way. They propose six disclosure requirements for authors.

Munafo and colleagues (2017) generalise the remedy into a programme, and are candid that the supporting evidence is partial. For a student study the principle is proportion, not compliance. Report what you did, and make the analysis reproducible to the scale of the claim. Cockburn and colleagues (2020) carry the argument into computing. They locate the cause of replication problems in incentive structures and publication practices that reward low-power studies and selective reporting. HCI's quantitative studies sit inside those practices.

Contested question: does credibility require coders to agree?

The reliability position holds that a coding frame is an instrument, and an instrument whose output depends on who holds it is a weak one. O'Connor and Joffe (2020) give the procedural version. Intercoder reliability can improve the systematicity, communicability and transparency of coding, and help convince audiences of the trustworthiness of an analysis. They give guidance on coefficients, chance correction and thresholds, then concede the limit. The chance-corrected coefficient is Cohen's (1960), derived for nominal scales as the agreement left after chance. Reliability attests to the robustness of the coding process and guarantees neither the data collection before it nor the theme generation after it.

The reflexive position holds that the disagreements are the analysis. Braun and Clarke (2021) treat coding reliability measures as one paradigm's requirement imposed on another's. Grounded theory researchers have long argued that a recursive analytic process cannot be validated by agreement at a single point. O'Connor and Joffe (2020) record that position. Two coders converging may mean the code was easy, not that the claim is true.

The field sits between them. McDonald, Schoenebeck and Forte (2019) analysed a representative sample of CSCW and HCI papers from 2016 to 2018. They found that authors communicate reliability in various ways while inter-rater reliability (the same measure, under another name) is rare, appearing in around one in nine qualitative papers. They read the borrowing as sometimes defensive and in tension with the interpretative traditions authors claim.

What would settle it? Two designs would bear on it. The first varies the reliability evidence given for the same finding and measures whether a reader's assessment changes. The second asks whether the reliability apparatus predicts how well findings hold up. McDonald and colleagues supply the baseline such a test needs. The unevenness of current practice is the reason the comparison is hard to run.

The two positions differ over what reliability is for. O'Connor and Joffe give procedural guidance and concede its limit. Braun and Clarke argue from the logic of reflexive analysis. McDonald and colleagues supply the field's own baseline, finding inter-rater reliability in about one in nine qualitative papers. While the dispute is unresolved, show the path from data to claim. Report the codes, the merged themes and the theme abandoned against a counterexample. The strongest objection is that an auditable path is still one analyst's reading, so auditability is not the same as agreement.

If reliability is not the test, what is? Yardley (2000) proposes four flexible principles: sensitivity to context, commitment and rigour, transparency and coherence, and impact and importance. The scheme invites an objection. Criteria broad enough to fit every qualitative tradition are broad enough for a weak study that uses the vocabulary. The reply is that criteria are prompts for an argument rather than a score. A study must say which it claims and show the evidence.

Where the argument is heading

The newest pressure comes from tools that do part of the analysis. Xiao and colleagues (2023) tested a workflow for deductive coding. An expert-drafted codebook is given to a large language model, which codes a set of curiosity-driven questions. Compared with expert coding, the model agrees fairly to substantially on well-specified codes and is unreliable where a code needs contextual judgement. The labour moves to codebook construction and audit. The codebook, the model version and the audit record become part of the analysis that must be reported. The coders may be two researchers, one researcher at two times, or a researcher and a model. The question is the same. Can a reader follow the path from the data to the claim, and see what would have changed it?

The sources this unit rests on

• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37-46. doi:10.1177/001316446002000104 Introduces a coefficient of agreement for nominal scales that corrects for the agreement expected by chance, and it became the standard index for reporting reliability between coders. A measure of agreement, which is not validity.
• Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology. doi:10.1191/1478088706qp063oa Sets out thematic analysis as a method in its own right, with six phases from familiarisation and coding to defining themes and producing the report. The procedure this unit teaches, and the object of the dispute over whether quality is procedure or judgement.
• Blandford, A., Furniss, D. and Makri, S. (2016). Qualitative HCI Research: Going Behind the Scenes. Synthesis Lectures on Human-Centred Informatics. doi:10.2200/s00706ed1v01y201602hci034 Synthesises semi-structured qualitative HCI research across purpose, planning, sampling, analysis, paradigms, reporting and quality, with checklists for planning, conducting, reporting and evaluating a study. It states why unreported work cannot be judged.
• Braun, V. and Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology 18(3), 328-352. doi:10.1080/14780887.2020.1769238 Argues that quality in reflexive thematic analysis cannot be reduced to a checklist, since the technique is inseparable from the analyst's judgement. It supplies the ten common problems and twenty reviewer questions the field is split over.
• O'Connor, C. and Joffe, H. (2020). Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines. International Journal of Qualitative Methods. doi:10.1177/1609406919899220 Reviews the arguments for and against intercoder reliability and gives guidance on coefficients, chance correction and thresholds, while noting that reliability attests only to the coding process. It concedes that a coefficient guarantees neither the collection before it nor the theme generation after.
• McDonald, N., Schoenebeck, S. and Forte, A. (2019). Reliability and Inter-rater Reliability in Qualitative Research. Proceedings of the ACM on Human-Computer Interaction. doi:10.1145/3359174 Analyses how reliability is described in a sample of CSCW and HCI papers from 2016 to 2018 and finds inter-rater reliability rare, in around one in nine qualitative papers, reading the borrowing as sometimes defensive and in tension with the traditions authors claim.
• Yardley, L. (2000). Dilemmas in qualitative health research. Psychology and Health. doi:10.1080/08870440008400302 Proposes four flexible principles: sensitivity to context, commitment and rigour, transparency and coherence, and impact and importance. The standard alternative to imported psychometric criteria, whose fourth principle asks what a study changes.
• Finlay, L. (2002). Outing the Researcher: The Provenance, Process, and Practice of Reflexivity. Qualitative Health Research. doi:10.1177/104973202129120052 Examines reflexivity as the researcher's explicit, self-aware analysis of their own role and of how intersubjective elements shape data collection and analysis, and insists its provenance and process be documented. It distinguishes confessional method accounts, examinations of the researcher's reactions and analyses of co-construction.
• Sandelowski, M. (1995). Sample size in qualitative research. Research in Nursing and Health. doi:10.1002/nur.4770180211 Argues that numbers are not unimportant in qualitative sampling, since a sample can be too small to support a claim of redundancy or saturation and too large for deep case-oriented analysis. Adequacy becomes a judgement about the information collected against the intended use.
• Simmons, J. P., Nelson, L. D. and Simonsohn, U. (2011). False-Positive Psychology. Psychological Science. doi:10.1177/0956797611417632 Uses simulations and two real experiments to show that flexible decisions about dependent variables, stopping rules, covariates and reported conditions let a researcher present almost any hypothesis as significant, reaching a 60.7 per cent false-positive rate when four freedoms combine. Six disclosure requirements for authors follow.
• Munafo, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E., Ware, J. J. and Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour. doi:10.1038/s41562-016-0021 Sets out a manifesto for reproducible science with measures at each stage, from methods and reporting standards to evaluation and incentives. The compact statement of practices for a student study, and candid that the supporting evidence is partial.
• Cockburn, A., Dragicevic, P., Besancon, L. and Gutwin, C. (2020). Threats of a replication crisis in empirical computer science. Communications of the ACM. doi:10.1145/3360311 Argues that empirical computer science faces the same replication problems as other experimental fields, caused by incentive structures and publication practices that produce low power and selective reporting, so the audit demand reaches quantitative HCI.
• Xiao, Z., Yuan, X., Liao, Q. V., Abdelghani, R. and Oudeyer, P. (2023). Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. 28th International Conference on Intelligent User Interfaces. doi:10.1145/3581754.3584136 Tests a workflow in which an expert-drafted codebook is given to a large language model for deductive coding, comparing the model's codes with expert coding. It reports fair to substantial agreement for well-specified codes and unreliability where a code needs contextual judgement.