Mixed methods paradigms, appraisal and the quality of inference

Year 4 / Module 26 Advanced qualitative and mixed-methods research / Unit 8 / Mixed methods paradigms, appraisal and the quality of inference

Mixed methods paradigms, appraisal and the quality of inference

HCI4102 · Unit 8 of 10

The value of mixing rests on the quality of the meta-inference it yields, and the incompatibility thesis that quantitative and qualitative paradigms cannot be combined remains a live dispute that appraisal tools only partly settle.

Mixing is worth what its meta-inference is worth

This eighth of ten units argues that the value of mixing rests on the quality of the conclusion the study licenses, the meta-inference. The unit before it settled how integration is designed, and this one asks what the combination warrants.

A paradigm is a set of assumptions, and a meta-inference is a claim in its own right

A mixed methods study is not valued for holding two datasets. It is valued for producing a third thing, a conclusion that neither a qualitative nor a quantitative study reaches alone. Whether that third thing exists is an evidential question. The older dispute, whether the two paradigms can be combined at all, remains live.

Two terms need fixing. A paradigm is a set of assumptions about what there is to know and how it may be known, and the quantitative and qualitative traditions are ordinarily said to carry different ones (Smith, 1983; Sale, Lohfeld and Brazil, 2002). A meta-inference is the conclusion a study invites once both strands have been analysed. It is a claim in its own right rather than a summary of the other two. The place where the strands meet is the point of integration (Schoonenboom and Johnson, 2017).

The incompatibility dispute is three positions rather than two

Purists hold the paradigms irreconcilable. Pragmatists hold that methods are tools. Smith (1983) put the disagreement on three questions: the investigator's relationship to what is investigated, facts and values, and the goal of investigation. On that reading the dispute concerns assumptions about the phenomena under study, not whether a questionnaire may sit beside an interview.

Howe (1988) attacked the incompatibility thesis directly, arguing from a pragmatic perspective that no incompatibility exists at the level of practice or of epistemology. Sale, Lohfeld and Brazil (2002) reached a narrower and harder conclusion. The two paradigms hold different views of reality, so they do not study the same phenomena. The methods therefore cannot be combined for triangulation or cross-validation, only for complementary purposes. That is a partial incompatibility claim with an argument attached.

Giddings (2006) supplies the sharpest critique from the other direction. She argues that the promise of the best of both worlds takes on a certainty that serves economic restraint and evidence-based practice. The qualitative and quantitative labels, she adds, reinforce a binary that marginalises diversity within each tradition. Giddings and Grant (2006) add a distinction that helps. The most commonly used mixed methods designs rest on positivist assumptions, while the combination of methods can be used within any paradigm.

Pragmatism is the field's default warrant, and it carries more than it can bear

The usual answer is pragmatism: choose the combination that answers the question, and Biesta (2010) examines how much that answers rather than asserting it. He reviews the claim that pragmatism can provide the philosophical foundation for mixed methods research, separates stronger and weaker versions, and identifies seven levels at which ideas about mixing occur. Reconstructing Dewey's theory of knowing, he concludes that pragmatism helps dissolve certain dualisms rather than settle the paradigm question.

Timans, Wouters and Heilbron (2019) press the same doubt from sociology, analysing the rise of mixed methods research from a field-theoretical perspective and engaging critically with the assumptions behind its conception of using multiple methods. Their target is the framework's assumptions rather than its technique. The warning is that a design vocabulary can travel faster than the reasoning behind it.

The point of integration is a design decision, and the inference is its deliverable

A design has to be built with the quality of that inference in view. Schoonenboom and Johnson (2017) set out seven design dimensions, among them purpose, theoretical drive, timing, point of integration and complexity, with ten secondary dimensions. They place validity as a concern running through the design, rather than a property checked at the end.

Younas, Fabregues, Munce and Creswell (2025) give the term a typology. Reviewing published studies, they separate global meta-inferences, drawn for populations beyond the sample, from specific ones, and name five sub-types under both: relational, predictive, causal, comparative and elaborative.

Their two cases show the dimensions at work. Roth's study of the gender wage gap in Wall Street securities firms asked closed questions inside in-depth interviews, so the design was dependent as well as simultaneous. McMahon's study of rape myths among college athletes ran a survey first and focus groups afterwards, and let the two strands disagree: the survey found low acceptance while the groups showed subtle versions of it, and the disagreement was the outcome the design was built to produce. Both samples are also narrow: brokers and college athletes are literate respondents who are easy to reach and have time to spare, so the meta-inference they support is warranted for people like them.

Creamer (2018) goes further, making integration constitutive of the method. Her text follows integration through design, analysis and reporting, and devotes a section to judging published work.

Contested question: can an instrument settle the quality of the inference?

The live disagreement is not whether quality matters, but whether a single instrument can decide it. On the instrument position, quality is inspectable against stated criteria. Hong and colleagues (2018) present version 2018 of the Mixed Methods Appraisal Tool, which appraises qualitative, quantitative and mixed methods studies against five core criteria for each of five design categories. Pace and colleagues (2012) tested the pilot instrument on 32 participatory research evaluations with two reviewers, at about 14 minutes per study. Agreement ran from moderate to almost perfect across the criteria, and was substantial on the overall score. Fabregues and colleagues (2024) argue that quality frameworks check whether integration is present rather than whether it was rigorous. They offer a framework for the second question.

On the sceptical position, the criteria are not neutral and the object is not stable. O'Cathain (2010) reviews the conceptualisations of quality in this field and opens with the admission that no accepted criteria for judging a mixed methods study exist. Her framework is built from a critical review and tested on one study. Giddings (2006) supplies one reason a checklist may flatter a study: if the combination covers for a continuing positivism, an instrument built inside that assumption certifies it. The audit evidence is uncomfortable for both sides. Fabregues and colleagues (2020) screened eight palliative care journals, included 159 mixed methods studies, and found that authors usually did not describe the design type and gave little detail on integration. 57.9 per cent used a convergent design and 82.4 per cent named complementarity.

What would settle it is a test of the instruments against the inferences, not another framework. Reviewers who did not conduct a study can rate a set of reports with a named instrument. They then judge, blind to those ratings, whether each report's meta-inference depends on both strands. Correspondence would show that appraisal reaches the product, not only the procedure. A second study can ask whether reviews built on highly rated studies reached better supported conclusions.

The instrument position has one reliability study, a pilot applied to 32 evaluations by two reviewers (Pace et al., 2012), and a widely used instrument (Hong et al., 2018). The sceptical position has a critical review tested on a single study (O'Cathain, 2010) and two arguments from theory (Giddings, 2006; Fabregues et al., 2024). The strongest empirical finding cuts across both: of 159 studies drawn from eight journals, authors rarely described the design type. While the dispute is open, a designer should read an appraisal score as a claim about the report and test the meta-inference separately, since a report can be complete and still rest on a combination that adds nothing.

Where the argument is heading

The direction of travel is from presence to rigour, and from a checklist applied to a report towards a warrant for the inference. Schoonenboom and Johnson (2017) place the point of integration among the design dimensions. Creamer (2018) folds the evaluation of published work into the method; Fabregues and colleagues (2024) move the target from whether integration happened to how well it was done. The instruments built since answer O'Cathain's (2010) request for agreed criteria rather than settling it. The question now is whether a reader can tell, from the report, how much of the conclusion rests on the combination.

The sources this unit rests on

Each note states what the cited document did and found and what it means for the argument. Records were checked against Crossref and OpenAlex on 31 October 2026, and every figure against the record it comes from.

Smith, J. K. (1983). Quantitative versus qualitative research: an attempt to clarify the issue. doi:10.3102/0013189x012003006 Sets the two traditions against three questions, on the investigator's relation to what is investigated, on facts and values, and on the goal of investigation. It locates the disagreement in those assumptions rather than in technique, which is where this unit starts.
Howe, K. R. (1988). Against the quantitative-qualitative incompatibility thesis. doi:10.3102/0013189x017008010 Answers the claim that a preference for what works ignores the incompatibility of positivist and interpretivist paradigms, arguing pragmatically that no incompatibility exists at the level of practice or of epistemology. It is the unit's strongest counter-position.
Sale, J. E. M., Lohfeld, L. H. and Brazil, K. (2002). Revisiting the quantitative-qualitative debate. doi:10.1023/a:1014301607592 Reviews the arguments for and against mixing and concludes that the paradigms hold different views of reality, so the methods cannot be combined for cross-validation or triangulation, only for complementary purposes. It shows the incompatibility claim alive in argued and narrow form.
Giddings, L. S. (2006). Mixed-methods research. doi:10.1177/1744987106064635 Argues that the promise of the best of both worlds takes on a certainty that serves economic restraint and evidence-based practice, and that the labels reinforce a binary that marginalises diversity within each tradition. Its charge, that mixed methods covers for a continuing positivism, is the unit's reason to doubt that appraisal instruments are neutral.
Giddings, L. S. and Grant, B. M. (2006). Mixed methods research for the novice researcher. doi:10.5172/conu.2006.23.1.3 Overviews the varieties of mixed methods design and situates them in a paradigmatic framework, finding that the most commonly used designs rest on positivist assumptions while the combination can be used within any paradigm. The unit uses it to separate a paradigm constraint from a design convention.
Biesta, G. (2010). Pragmatism and the philosophical foundations of mixed methods research. doi:10.4135/9781506335193.n4 Reviews the claim that pragmatism can supply the philosophical foundation for mixing, separating stronger and weaker versions and naming seven levels at which mixing ideas occur, to show where mixing is unproblematic and where it is not. Reconstructing Dewey's theory of knowing, it concludes that pragmatism dissolves certain dualisms rather than settling the question.
Creamer, E. G. (2018). An introduction to fully integrated mixed methods research. doi:10.4135/9781071802823 Makes integration constitutive of a mixed methods project, following it through design, analysis and reporting, and devoting a section to evaluating the quality of published mixed methods research. It is the unit's evidence that the field now folds appraisal into the method; the note rests on the publisher's record.
Schoonenboom, J. and Johnson, R. B. (2017). How to construct a mixed methods research design. doi:10.1007/s11577-017-0454-1 Sets out seven major design dimensions, among them purpose, theoretical drive, timing, point of integration and complexity, with ten secondary dimensions and two case studies, and keeps validity in view throughout. The point where the strands meet is therefore a decision rather than a discovery.
Younas, A., Fabregues, S., Munce, S. and Creswell, J. W. (2025). Framework for types of metainferences in mixed methods research. BMC Medical Research Methodology, 25, 18. doi:10.1186/s12874-025-02475-8 Reviews the mixed methods literature and proposes two broad types of meta-inference, global and specific, with five sub-types: relational, predictive, causal, comparative and elaborative. It illustrates each from published studies.
Timans, R., Wouters, P. and Heilbron, J. (2019). Mixed methods research: what it is and what it could be. doi:10.1007/s11186-019-09345-5 Analyses the rise of mixed methods research from a field-theoretical perspective, engages critically with the assumptions behind its conception of using multiple methods, and closes with an alternative view of method use. Its target is the assumptions behind the framework, not its technique.
O'Cathain, A. (2010). Assessing the quality of mixed methods research. doi:10.4135/9781506335193.n21 Reviews existing conceptualisations of quality, builds a comprehensive framework from a critical review of the literature, tests it on a mixed methods study and records the remaining challenges. It opens by stating that no accepted criteria for judging a mixed methods study exist, which is the gap the later instruments answer.
Hong, Q. N., Fabregues, S., Bartlett, G. and colleagues (2018). The Mixed Methods Appraisal Tool (MMAT) version 2018. doi:10.3233/efi-180221 Presents the 2018 revision, which assesses qualitative, quantitative and mixed methods studies against five core criteria for each of five design categories, and reports that it rested on usefulness testing and a Delphi study. Integration is one criterion among five rather than the object of the instrument.
Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J. and Seller, R. (2012). Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool. doi:10.1016/j.ijnurstu.2011.07.002 Had two independent reviewers appraise 32 participatory research evaluations with the pilot instrument at about 14 minutes per study, with agreement from moderate to almost perfect on the criteria. It concludes that reliability is promising but needs development, which bounds what such a tool can bear.
Fabregues, S., Hong, Q. N., Escalante-Barrios, E. L., Guetterman, T. C., Meneses, J. and Fetters, M. D. (2020). A methodological review of mixed methods research in palliative and end-of-life care. doi:10.3390/ijerph17113853 Screened eight specialist journals and included 159 mixed methods studies, finding that authors usually did not describe the design type and gave little detail on integration. 57.9 per cent used a convergent design and 82.4 per cent named complementarity, which the unit reads as an audit against the frameworks.
Fabregues, S., Younas, A., Escalante-Barrios, E. L., Molina-Azorin, J. F. and Vazquez-Miraz, P. (2024). Toward a framework for appraising the quality of integration in mixed methods research. doi:10.1177/15586898241257555 Argues that most quality frameworks check whether integration is present rather than whether it was rigorous, and presents a Mixed Methods Integration Quality Framework built from a methodological review. It is the unit's most recent test of whether appraisal reaches the inference.

Check your understanding & apply it · 4 questions, 1 essay

1A methods section says the strands will be integrated through triangulation because pragmatism licenses combining whatever works. Which diagnosis does the unit support?

2A doctoral candidate names triangulation as the purpose for combining a survey and interviews. Applying Sale, Lohfeld and Brazil (2002), what does the unit predict about that purpose?

3A reviewer rates a mixed methods report as high quality with an appraisal instrument and concludes that its meta-inference is well warranted. Which judgement does the unit support?

4A journal editor wants evidence about reporting practice in her field and considers Fabregues and colleagues' (2020) review of eight palliative care journals. Which judgement matches what the unit reports?

5Essay A doctoral candidate proposes to survey 400 clinicians about a new decision-support tool and then interview 20 of them, and writes in the methods section that the two strands will be "integrated through triangulation" because "pragmatism licenses combining whatever works". Advise them. State what the paradigm dispute obliges them to address and who holds each position, say what the appraisal literature would let them claim about the quality of their study and what it would not, name what has to be reported for a reader to judge their meta-inference, and set out the evidence that would settle whether their conclusion is genuinely mixed rather than two studies in one report. Cite the sources you use at each step.

The candidate's methods section is doing two different jobs badly. The appeal to pragmatism is a warrant that has to be argued rather than invoked: Biesta (2010) separates stronger and weaker versions of the claim that pragmatism can ground mixed methods research, identifies seven levels at which ideas about mixing occur, and concludes that Dewey's pragmatism dissolves certain dualisms rather than settling the paradigm question. The candidate therefore owes an account of why a survey and an interview schedule are commensurable here, and of what each is taken to be measuring. The paradigm dispute is not settled, and three positions bear on their design. Howe (1988) argues from a pragmatic perspective that no incompatibility exists at the level of practice or of epistemology, which would licence their plan if it is right. Sale, Lohfeld and Brazil (2002) argue that the two paradigms hold different views of reality and therefore study different phenomena, so the methods cannot be combined for cross-validation or triangulation and may be combined only for complementary purposes. Triangulation is precisely the purpose the candidate has named, so this position, not the caricature of purism, is the one their design has to answer. Giddings (2006) adds the reason to be careful about instruments, arguing that the promise of the best of both worlds takes on a certainty that serves economic restraint and evidence-based practice, and that mixed methods covers for a continuing positivism. On appraisal, the candidate may claim that a tool exists and can be applied: Hong and colleagues (2018) present version 2018 of the Mixed Methods Appraisal Tool, with five core criteria for each of five design categories, and Pace and colleagues (2012) found that two reviewers appraising 32 participatory research evaluations with the pilot version agreed from moderate to almost perfect on its criteria at about 14 minutes per study, while concluding that development was still needed. What the candidate may not claim is that quality is thereby settled. O'Cathain (2010) states that no accepted criteria for judging a mixed methods study exist, and Fabregues and colleagues (2024) argue that frameworks check whether integration is present rather than whether it was rigorous. The reporting burden follows: Fabregues and colleagues (2020) screened eight palliative care journals, included 159 mixed methods studies, and found that authors usually did not describe the design type and gave little detail on integration, which is the failure the candidate must avoid. To let a reader judge the meta-inference, the report has to name the point of integration as a design decision, following Schoonenboom and Johnson (2017), who set out seven design dimensions including point of integration and keep validity in view throughout, and it has to show how the interview findings change what the survey count means, as Creamer (2018) requires when she makes integration constitutive of the method. Finally, the candidate should state what would settle whether the conclusion is mixed: reviewers who did not run the study would rate the report with a named instrument and then judge, blind to those ratings, whether the conclusion depends on both strands; an inference that survives the removal of either strand was mixed, and one that does not was two studies in one report.