Rigour without a reliability figure: reflexivity and the credibility of an interpretation

A study can be defensible at every procedural stage and still leave a reader unable to say whether its interpretation is warranted. The commoner failure is quieter. A report lists the strategies it used, from triangulation to member checking to a reflexive paragraph among the opening pages. The list stands where the argument ought to be.

This tenth and last unit argues that no checklist and no statistic substitutes for an analyst making their interpretive position, decisions and counter-cases visible. The credibility of a claim rests on what a reader can see the analyst do with the evidence that could count against it. The nine units before it each took one analytic step. This one asks what makes the whole account credible.

Rigour has to be fitted to the project, not imported

Sandelowski (1986) names four factors that complicate any debate about the scientific merits of qualitative work. They are the variety of methods, the blurred boundary between the two families, the habit of judging such research by conventional criteria, and the artistic features of inquiry. Judging it by the canons of quantitative research is the obstacle, so rigour has to be built for the project. Malterud (2001) makes the same move from inside medicine, treating the two strategies as complementary. She proposes relevance, validity and reflexivity as overall standards for qualitative inquiry.

The trustworthiness settlement moved rigour from the inquiry to the report

Lincoln and Guba (1985) supplied the vocabulary the field still uses: credibility, transferability, dependability and confirmability as the analogues of internal and external validity, reliability and objectivity. Morse and colleagues (2002) describe what it did in practice. The rejection of reliability and validity in the 1980s shifted responsibility for rigour from the investigator's actions during the research to the reader who judges the finished account. Quality became something a report displays rather than something an inquiry does.

When the criteria harden into a checklist

A displayed property can be displayed without being present. Barbour (2001) made the case in the BMJ. The uncritical adoption of technical fixes, among them purposive sampling, grounded theory, multiple coding, triangulation and respondent validation, does not confer rigour, and overzealous use is counterproductive. She draws the five from what she meets most often as a reviewer of journal articles and grant submissions. She asks of each whether it is woven into a reasoned design or merely listed. Morse (2015) takes the critique to the criteria themselves. The trustworthiness strategies of the 1980s remain in use with only minor changes, have not been critically examined, and have an unmeasured impact on completed research. Varpio and colleagues (2017) name four terms that carry the problem: thematic emergence, triangulation, saturation and member checking. They show that each is often used without reflection on its roots or implications.

Reflexivity is work on the analysis, not a confession

Finlay (2002) drew the map. She asks how researchers should reflexively evaluate the ways intersubjective elements transform their research, and describes the work as negotiating a swamp of ambiguity and multiple trails. Her five variants are introspection, intersubjective reflection, mutual collaboration, social critique and discursive deconstruction. They are competing accounts of what reflexivity is for. A report that gestures at reflexivity without naming the variant, or what it changed, has said nothing checkable. Johnson, Adkins and Chauvin (2020) show the field agreeing on its importance. They describe reflexivity as a researcher's insight into their own biases and decisions, and call it critical to rigour.

Verification during the inquiry and methodological integrity are two exits from the checklist

Morse and colleagues (2002) do not accept evaluation after the fact as a substitute for verification during it. Reliability and validity remain the right concepts, they argue, and the investigator should reclaim them through verification built into the inquiry. Their five elements are methodological coherence, sampling sufficiency, a dynamic relationship between sampling, data collection and analysis, thinking theoretically, and theory development. Being self-correcting, they can stop a study before a threat to its conclusions hardens. Levitt and colleagues (2017) propose a different standard rather than a longer list, putting methodological integrity in place of reliability. They define it through fidelity to the subject matter and utility in achieving research goals. Smith and McGannon (2018) reach a compatible conclusion, judging member checking, inter-rater reliability and universal criteria ineffective for verification. They propose a relativist, context-sensitive approach instead. Their case against member checking is concrete. A participant may not follow the analysis, or may treat the researcher as the expert and agree with a reading they do not hold. The studies they review seldom report a disagreement, which is not the same as there being none. Their alternative, member reflection, asks what the account opens up rather than whether it is confirmed.

Quality divides into means and ends, and no single test covers both

Tracy (2010) distinguishes the means of qualitative research from its ends, and proposes eight criteria: a worthy topic, rich rigour, sincerity, credibility, resonance, significant contribution, ethics and meaningful coherence. Sincerity can be met only by disclosing what the analyst did and why, while credibility and resonance are properties a reader has to feel. Yardley (2000) answers the question of what criteria suit a qualitative analysis with four open-ended principles: sensitivity to context, commitment and rigour, transparency and coherence, and impact and importance. As principles rather than thresholds, they call for judgement in the case at hand.

Contested question: do the trustworthiness criteria do the work they promise?

Three positions are live, and the evidence behind all three is conceptual argument.

The first holds that criteria are the right instrument and need only be used well. Lincoln and Guba (1985) introduced the four criteria in a book, so that qualitative work could be judged on its own terms, and the vocabulary is now the field's common language. Tracy (2010) and Yardley (2000) extend it in conceptual articles, and Johnson, Adkins and Chauvin (2020) show in a review of indicators how widely the criteria circulate. On this view the failure lies in the application, not in the standard.

The second holds that the criteria have become a replacement checklist whose effects have never been tested. Morse (2015) states the gap plainly in a critical analysis: how well the strategies achieve their intended goal has not been examined. Barbour (2001) supplies the mechanism from review experience rather than from data, since a listed fix can be reported without being understood. Varpio and colleagues (2017) show four terms whose meanings drifted once expected.

The third holds that the standard itself is the problem. Levitt and colleagues (2017) put methodological integrity in the place of fixed procedures in a set of recommendations. Smith and McGannon (2018) reject universal criteria for a relativist, context-sensitive approach, and their case against member checking rests on what the studies they review fail to report: a disagreement, which a participant in later life may withhold out of deference. Morse and colleagues (2002) belong here too, from a different direction, wanting reliability and validity moved from the reader's judgement back to the investigator's conduct.

What would settle it is empirical, and Morse (2015) has marked the missing study. Take completed reports, score them for the criteria they claim, then measure independently what a reader can inspect, whether the interpretation survives the counter-cases in its own data set, and whether the findings are usable. Strong agreement would favour the first position. Weak agreement would mean the list largely measures itself. The strongest counter-argument to this unit's own claim is that disclosure of position does not license a claim. A report can be frank about the analyst and still leave the interpretation uncheckable, so visibility is necessary and not sufficient.

What to do while the dispute is open

Report the counter-cases your own data set resisted, and say what would have to appear in the data to overturn the reading. Name the variant of reflexivity you practise, and what it changed. Do not offer member checking as confirmation. Ask instead what the account opens up.

Where the argument is heading

The direction of travel is away from the audit and towards the account. Levitt and colleagues (2017) describe the shift in their own terms, from standardised and decontextualised procedures towards the methodological bases of trustworthiness as they function within projects. Finlay (2002) gives the analyst routes through the swamp rather than one correct posture. Smith and McGannon (2018) also address editors, not only authors. For interaction research the consequence is concrete: a claim about an interface is used to justify a design. A study of a system in use can show, in its own report, which reading of the record was resisted, and what would overturn it.

The sources this unit rests on

• Sandelowski, M. (1986). The problem of rigor in qualitative research. Advances in Nursing Science 8(3), 27-37. doi:10.1097/00012272-198604000-00005 Sandelowski argues that qualitative research can be as rigorous as it is relevant, and that judging it by conventional scientific criteria is the obstacle. She names four complicating factors, among them the variety of methods and the blurred boundary between the two families, and asks for rigour fitted to the project.
• Malterud, K. (2001). Qualitative research: Standards, challenges, and guidelines. The Lancet 358(9280), 483-488. doi:10.1016/s0140-6736(01)05627-6 Malterud treats qualitative and quantitative strategies as complementary and proposes relevance, validity and reflexivity as overall standards for qualitative inquiry. It argues for those standards inside a quantitative discipline.
• Lincoln, Y. S. and Guba, E. G. (1985). Naturalistic Inquiry. Beverly Hills: Sage. https://www.sagepub.com/shop/buy-a-book/naturalistic-inquiry-1-842 The book sets out naturalistic inquiry as an alternative paradigm and proposes credibility, transferability, dependability and confirmability as four analogues of internal and external validity, reliability and objectivity. It is the origin of the vocabulary the unit argues about.
• Morse, J. M., Barrett, M., Mayan, M., Olson, K. and Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods 1(2), 13-22. doi:10.1177/160940690200100202 Morse and colleagues describe the shift in responsibility from the investigator's conduct to the reader's evaluation of the finished account, and argue that reliability and validity remain the right concepts. Their verification strategies operate during the study rather than after, giving the unit its conduct-based alternative.
• Barbour, R. S. (2001). Checklists for improving rigour in qualitative research: A case of the tail wagging the dog? BMJ 322(7294), 1115-1117. doi:10.1136/bmj.322.7294.1115 Barbour argues that reducing qualitative research to a list of technical procedures, among them purposive sampling, multiple coding, triangulation and respondent validation, is over-prescriptive and results in the tail wagging the dog. No fix confers rigour on its own, the unit's clearest case of a quality device reported instead of earned.
• Morse, J. M. (2015). Critical analysis of strategies for determining rigor in qualitative inquiry. Qualitative Health Research 25(9), 1212-1222. doi:10.1177/1049732315588501 Morse argues that the trustworthiness criteria introduced in the 1980s remain in use with only minor changes, that the strategies behind them have never been critically examined, and that their impact on completed research is unknown. She recommends returning to rigour, reliability, validity and generalisability, and placing each strategy where it belongs. It states the empirical problem at the centre of the unit's contested question.
• Varpio, L., Ajjawi, R., Monrouxe, L. V., O'Brien, B. C. and Rees, C. E. (2017). Shedding the cobra effect: Problematising thematic emergence, triangulation, saturation and member checking. Medical Education 51(1), 40-50. doi:10.1111/medu.13124 Varpio and colleagues examine four terms, thematic emergence, triangulation, saturation and member checking, and show that each has roots, definitions and implications that users often do not reflect on. They argue the terms became expected in health professions education, and that uncritical use has misrepresented qualitative work. It shows expectations, not analysis, driving what a report claims.
• Finlay, L. (2002). Negotiating the swamp: The opportunity and challenge of reflexivity in research practice. Qualitative Research 2(2), 209-230. doi:10.1177/146879410200200205 Finlay asks how researchers should reflexively evaluate the ways intersubjective elements transform their research, and describes the work as negotiating a swamp of ambiguity and multiple trails. Her five maps of reflexive practice, from introspection to discursive deconstruction, show competing rationales inside the one word, so the unit insists that a reflexive statement name the variant it practises.
• Johnson, J. L., Adkins, D. and Chauvin, S. (2020). A review of the quality indicators of rigor in qualitative research. American Journal of Pharmaceutical Education 84(1), article 7120. doi:10.5688/ajpe7120 Johnson, Adkins and Chauvin review the quality indicators the field uses across designing, conducting and reporting qualitative research. They describe reflexivity as a researcher's insight into their own biases and rationale for decisions as a study progresses, and call it critical to rigour. It shows where the field itself places the analyst.
• Levitt, H. M., Motulsky, S. L., Wertz, F. J., Morrow, S. L. and Ponterotto, J. G. (2017). Recommendations for designing and reviewing qualitative research in psychology: Promoting methodological integrity. Qualitative Psychology 4(1), 2-22. doi:10.1037/qup0000082 Levitt and colleagues propose methodological integrity in place of reliability, defined through fidelity to the subject matter and utility in achieving research goals. They aim to move review away from standardised and decontextualised procedures towards the methodological bases of trustworthiness as they function within a project. It is the unit's constructive answer to the checklist.
• Smith, B. and McGannon, K. R. (2018). Developing rigor in qualitative research: Problems and opportunities within sport and exercise psychology. International Review of Sport and Exercise Psychology 11(1), 101-121. doi:10.1080/1750984x.2017.1317357 Smith and McGannon review member checking, inter-rater reliability and universal criteria, and conclude that each is ineffective for verification, trustworthiness or reliability. They propose a relativist, context-sensitive approach, and direct part of the argument at editors as well as authors. It gives the unit its sharpest statement of the rejectionist position.
• Tracy, S. J. (2010). Qualitative quality: Eight "big-tent" criteria for excellent qualitative research. Qualitative Inquiry 16(10), 837-851. doi:10.1177/1077800410383121 Tracy proposes eight criteria for excellent qualitative research, among them a worthy topic, rich rigour, sincerity, credibility and meaningful coherence. Her model separates the means of research from its ends It supplies the strongest version of the criteria position.
• Yardley, L. (2000). Dilemmas in qualitative health research. Psychology and Health 15(2), 215-228. doi:10.1080/08870440008400302 Yardley asks what criteria are appropriate for the validity of a qualitative analysis and answers with four open-ended principles: sensitivity to context, commitment and rigour, transparency and coherence, and impact and importance. The principles matter because they are explicitly not thresholds.