Quality is argued in the terms a tradition accepts, not scored

In a trial, rigour has a figure: an estimate, an interval, and a reader who can ask how far the interval would move under a different assumption. Qualitative work has no equivalent number, so the judgement is made in words and the words belong to a tradition. This ninth of ten units argues that qualitative quality is settled by argument in the terms that tradition accepts, not by a single checklist. The unit before it asked what a design warrants, and this one asks how the warrant is judged.

The founding translation was four criteria, and it has been disputed since

This is not vagueness, but a claim about what criteria are for. Lincoln and Guba (1985) made the founding move. They replaced the canons of internal and external validity, reliability and objectivity with four trustworthiness criteria as their qualitative parallels: credibility, transferability, dependability and confirmability.

Sandelowski (1993) argued that the persistent concern with rigour produces rigidity. The search for validity can threaten validity, and the search for reliability may undermine it. Her worked example is member validation, which looks like quality assurance but assumes a reading is the kind of thing a participant can confirm.

Morse and colleagues (2002) answer from the other side. Reliability and validity remain appropriate, and responsibility for them rests with the investigator during the study rather than with the reader at the end. Their five verification strategies are built into the conduct of inquiry: methodological coherence, sampling sufficiency, collecting and analysing concurrently, thinking theoretically and theory development. Returning results to participants looks like the strongest check available, and they argue it is not a verification strategy at all, since findings have been synthesised across individuals and a participant has no reason to recognise themselves in them. Rigour becomes something a design does rather than something a report displays, and none of these three sources proposes a figure.

No set of criteria is shared, and that is the live dispute

Rolfe (2006) states the disagreement as three positions. Qualitative work can be judged by the same criteria as quantitative research, by a separate set, or by no predetermined criteria at all. He takes the third, questioning the assumption of a unified qualitative paradigm and arguing that the value of qualitative research lies in its capacity to challenge rather than confirm.

Sparkes and Smith (2009) give the dispute its name. Criteriology is the belief that universal criteria exist and can be applied; relativism is the belief that they cannot. They argue for a middle position in which criteria are used but held open to revision, which is the position this unit takes.

Tracy (2010) separates the means of qualitative research from its ends. She proposes eight markers of quality that run from a worthy topic and rich rigour to sincerity, credibility, resonance, ethics and meaningful coherence.

Funders asked for reporting standards instead of criteria

Tong, Sainsbury and Craig (2007) built COREQ by compiling 76 items from 22 existing checklists, removing duplicates and impractical items, and grouping the rest into three domains: the research team and reflexivity, study design, and data analysis and reporting. The published instrument has 32 items, and its method bounds what it can claim, since COREQ consolidates prior opinion rather than testing which items predict anything.

O'Brien and colleagues (2014) made the same move for medicine. They searched PubMed, Web of Science and Google to July 2013, contacted experts and sent a near-final item list to five external reviewers. The resulting SRQR has 21 items. Transparency is a property of a document, not yet of an analysis.

Kempny and colleagues (2026) reviewed 75 studies using a large language model at a substantive analytic stage. Reporting was thin: three-quarters gave no parameter settings and half did not say how the model was deployed, while agreement with human coders ranged from 36 to 99 per cent.

Malterud (2001) shows why the pressure was real. Writing for a clinical audience, she proposes relevance, validity and reflexivity as overall standards for qualitative inquiry. She names reflexivity, transferability and the shared assumptions of interpretation as the challenges medical researchers meet. Her standards are argued rather than scored, and deliberately unlike trial criteria.

A completed checklist can flatter a thin analysis

Barbour (2001) lodged the objection that has not been answered. A checklist can be completed without the analytic work it is meant to evidence, so procedure substitutes for judgement. Her phrase, the tail wagging the dog, locates the risk in the direction of causation: the instrument should follow the quality, not generate it.

She names five devices used this way: purposive sampling, grounded theory, multiple coding, triangulation and respondent validation, none of which confers rigour on its own. A study can declare grounded theory and still derive its coding categories from the questions on its topic guide rather than from what participants said, which is the reverse of what the label promises.

Reporting still matters, but completeness and quality are different properties, and the checklist measures the first. A report can say where the sample came from and how themes were derived while the analysis stays thin, and nothing among the 32 COREQ items or the 21 SRQR items asks whether the interpretation is any good. The 32 COREQ items never ask what language the interview was conducted in, so a report can be complete and still leave unstated whether a participant answered in a second language and who translated the extract.

Smith (2018) reaches past the report in a second direction. He sets out four kinds of generalisability: naturalistic, transferability, analytical and intersectional. He argues that it is a misunderstanding to claim qualitative research lacks generalisability.

Contested question: does standardised reporting produce rigour or displace it?

The disagreement is not whether a report should be complete. It is whether completeness is a route to quality or a substitute for it, and the literature divides into three positions.

• Reporting criteria improve quality, because transparency is a precondition for appraisal and is worth having in its own right (Tong, Sainsbury and Craig, 2007; O'Brien et al., 2014).
• Reporting criteria measure reporting rather than quality, and a completed checklist can flatter a thin analysis (Barbour, 2001; Tracy, 2010).
• The criteria are misconceived, because traditions do not share the assumptions one set would need, so predetermining them settles the wrong question (Rolfe, 2006).

On the standardisation position the instrument is a floor rather than a ceiling, and a floor is worth building. On the sceptical position it displaces the judgement it was meant to support. Sandelowski (1993) had already argued that the pursuit of rigour can rigidify the work, and Van Cleave and Marn (2026) take the objection furthest: transparency and reflexivity are a writerly performance, so a report can display trustworthiness without possessing it.

What would settle it is a comparison the field has not run. Have one group of reviewers rate a set of reports with COREQ or SRQR. A second group, blind to those ratings, judges whether each study's interpretation is warranted by the evidence it presents. If the ratings predict that judgement the checklists reach quality; if they predict completeness but not the judgement they measure the report. A subtractive test would remove the analysis items and ask whether the rating still predicts anything. Until such a test exists the dispute is argued from cases.

The positions are argued, and the checklists consolidate rather than test, since COREQ and SRQR each compiled prior opinion rather than testing it (Tong, Sainsbury and Craig, 2007; O'Brien et al., 2014). Barbour (2001) argues the sceptical case from examples, and Van Cleave and Marn (2026) from the concept of performance, and no study has compared what a completed checklist predicts with an independent judgement. While the dispute is open, a designer should use the checklist to fix reporting and argue the analysis in the tradition's own terms. The strongest objection is that the comparison proposed here may be unrun, because the criterion is not measurable.

Where the argument is heading

The direction of travel is from criteria as properties of a report to criteria as an argued relation between a study and the tradition it belongs to. Wilson (2025) shows the shape of that move when two approaches are combined, asking after their conceptual coherence and methodological integrity rather than a score. Smith (2018) replaces one borrowed criterion with four named kinds a study can claim or decline. The question is whether coherence can be specified well enough to argue about without being scored.

The sources this unit rests on

Each note states what the cited document did and found and what it means for the argument. Records were checked against Crossref, OpenAlex and PubMed on 31 October 2026, and every figure against the record it comes from.

• Lincoln, Y. S. and Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. https://us.sagepub.com/en-us/nam/naturalistic-inquiry/book842 Presents four trustworthiness criteria, credibility, transferability, dependability and confirmability, as the qualitative analogues of internal and external validity, reliability and objectivity. It is the origin of the vocabulary every later source here argues within; the note rests on the publisher record because the book carries no DOI of its own.
• Sandelowski, M. (1993). Rigor or rigor mortis: the problem of rigor in qualitative research revisited. doi:10.1097/00012272-199312000-00002 Argues that the persistent concern with rigour produces rigidity, that the search for validity threatens validity and that the search for reliability may undermine it, with member validation as the worked example. It supplies the unit's case that borrowed rigour can be self-defeating rather than securing.
• Morse, J. M., Barrett, M., Mayan, M., Olson, K. and Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. doi:10.1177/160940690200100202 Argues that reliability and validity remain appropriate for qualitative work and that responsibility for them belongs with the investigator during the study rather than with the reader afterwards. Its five verification strategies are methodological coherence, sampling sufficiency, collecting and analysing concurrently, thinking theoretically and theory development.
• Rolfe, G. (2006). Validity, trustworthiness and rigour: quality and the idea of qualitative research. doi:10.1111/j.1365-2648.2006.03727.x Divides the quality literature into three positions: judging qualitative work by quantitative criteria, by a separate set, or by no predetermined criteria at all, and questions the assumption of a single qualitative paradigm. Rolfe takes the third, arguing that the value of qualitative research lies in its capacity to challenge rather than confirm.
• Sparkes, A. C. and Smith, B. (2009). Judging the quality of qualitative inquiry: criteriology and relativism in action. doi:10.1016/j.psychsport.2009.02.006 Analyses the tension between criteriology, the belief that universal criteria exist, and relativism, the belief that they do not, showing how each position plays out rather than settling it. It argues for a middle position in which criteria are used but held open to revision.
• Tracy, S. J. (2010). Qualitative quality: eight "Big-Tent" criteria for excellent qualitative research. doi:10.1177/1077800410383121 Distinguishes the means of qualitative research from its ends and proposes eight markers of quality: a worthy topic, rich rigour, sincerity, credibility, resonance, significant contribution, ethics and meaningful coherence. It describes the set as a common language that leaves space for dialogue, imagination, growth and improvisation.
• Tong, A., Sainsbury, P. and Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. doi:10.1093/intqhc/mzm042 Compiled 76 items from 22 existing checklists, removed duplicates and items that were impractical to assess, and grouped the rest into three domains: the research team and reflexivity, study design, and data analysis and reporting. The published instrument has 32 items, and its method is consolidation of prior opinion rather than a test of what predicts quality.
• O'Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A. and Cook, D. A. (2014). Standards for reporting qualitative research: a synthesis of recommendations. doi:10.1097/acm.0000000000000388 Searched PubMed, Web of Science and Google to July 2013, contacted experts and sent a near-final item list to five external reviewers, producing the 21-item SRQR. Its aim is a transparent document rather than a warranted interpretation.
• Kempny, C., Frings, J., Rust, P., Meister, S. and Fehring, L. (2026). The use and methodological reporting of large language models in qualitative research: a scoping review. BMC Medical Research Methodology, 26, 137. doi:10.1186/s12874-026-02913-1 Screened five databases and included 75 studies that used a large language model at a substantive analytic stage. It found parameter and deployment details widely unreported, with agreement against human coders from 36 to 99 per cent, and proposes a COREQ extension.
• Malterud, K. (2001). Qualitative research: standards, challenges, and guidelines. doi:10.1016/s0140-6736(01)05627-6 Proposes relevance, validity and reflexivity as overall standards for qualitative inquiry and discusses reflexivity, transferability and the shared assumptions of interpretation as the challenges medical researchers meet. It argues that qualitative and quantitative strategies are complementary rather than incompatible.
• Barbour, R. S. (2001). Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? doi:10.1136/bmj.322.7294.1115 Argues that the growing use of checklists to secure rigour risks the tail wagging the dog, because a checklist can be completed without the analytic work it is meant to evidence. It is the unit's central critique of the standardisation turn.
• Smith, B. (2018). Generalizability in qualitative research: misunderstandings, opportunities and recommendations for the sport and exercise sciences. doi:10.1080/2159676x.2017.1393221 Argues that it is a misunderstanding to claim qualitative research lacks generalisability and that statistical generalisability cannot judge it. It sets out four kinds, naturalistic generalisation, transferability, analytical generalisability and intersectional generalisability, with strategies for making and judging the claim.
• Wilson, J. (2025). Interpretive description and reflexive thematic analysis: exploring conceptual coherence and methodological integrity. doi:10.1177/10497323251378303 Compares interpretive description, an applied qualitative health methodology, with reflexive thematic analysis across their philosophical foundations, use of theory, analytic strategies and indicators of quality. It argues that co-using them requires an account that is methodologically coherent and transparent about researcher subjectivity.
• Van Cleave, J. and Marn, T. (2026). Producing participants: trustworthiness as convenient fiction. doi:10.1177/10778004251383520 Argues that trustworthiness relies on transparency, reflexivity and the believability of the written research and participants, that all such representations are fiction, and that verisimilitude secures belief. It holds that trustworthiness is not a property a study either has or lacks but something a report produces; the record carries a 2025 online-first date with a 2026 issue.