Positioning the literature: review method and the appraisal of evidence

A literature review is examined twice in a project proposal: for what it says about the field and for how it was assembled. This third of ten units argues that a review earns its place by making the contested question visible and by saying which studies are strong, rather than by listing who wrote what. The two cannot be separated, because the method a review adopts fixes what it can claim. A systematic, a scoping and a narrative review make three different claims about knowledge, and they are judged against different standards. A proposal that borrows the authority of one while doing the work of another invites the panel to notice the mismatch. The choice between the three types is first a choice about what the review can claim.

The argument has three steps. The review is a method with a declared purpose, which Snyder (2019) and Rowe (2014) defend. The vocabulary of types is descriptive rather than a ranking, which Grant and Booth (2009) and Munn and colleagues (2018) show when read together. And once the method is chosen, the substance of the review is the judgement it passes on the studies it cites. Webster and Watson (2002) and Baumeister and Leary (1997) place that judgement at its centre.

A review is a method with a claim attached

Snyder (2019) argues that the literature review is a research method in its own right rather than a preamble to empirical work. She separates three broad approaches: systematic, semi-systematic and integrative. A systematic review synthesises and compares evidence, and it needs a narrow question. A semi-systematic review, usually called a narrative review, maps a field whose concepts have been defined differently across disciplines, where a full systematic search is impossible. An integrative review assesses and critiques a body of work to build a new framework. Her instruction is that the question and purpose determine the strategy.

Rowe (2014) supplies the negative definition. He defines a literature review as work that synthesises past knowledge on a topic, identifies important biases and knowledge gaps in it, and proposes corresponding future research directions. He also grades the requirement. A review aiming at explanation must be highly systematic, whereas one aiming at understanding need not be, provided its coverage of the topic is good.

The vocabulary of review types is a description, not a ranking

Grant and Booth (2009) examined the vocabulary of review and synthesis against a framework they call SALSA, for search, appraisal, synthesis and analysis. They described fourteen review types. Few of the types have a prescribed and explicit method, and many are not mutually exclusive, so the label secures nothing in itself.

Munn and colleagues (2018) turn the typology into a decision rule. Scoping is the right choice where the purpose is to identify knowledge gaps, to scope a body of literature, to clarify a concept or to investigate research conduct. They add that scoping still requires rigorous and transparent methods, so the broader type is a different standard rather than a lower one.

Mapping and estimating are different claims

Arksey and O'Malley (2005) set out the first framework for scoping studies. Their list of purposes is already a statement about claim: to examine the extent and range of research activity, to judge whether a full systematic review is warranted, to summarise and disseminate findings, and to identify gaps. The framework came out of their own review of services for carers of people with mental health problems. Its stages are therefore the ones that particular review needed, rather than a neutral template. Levac and colleagues (2010) refined that framework from three of their own scoping studies. They required a clear link between purpose and question, a balance of feasibility against breadth, an iterative team approach to selecting and analysing studies, and consultation with stakeholders as a required stage rather than an optional one.

Reporting makes a review auditable

Page and colleagues (2021) report the PRISMA 2020 statement, which replaces the 2009 guideline and presents a twenty-seven item checklist. The items make a review's conduct visible, so a reader can decide whether to believe it. Tricco and colleagues (2018) extend the same discipline to mapping. Their scoping extension was developed by a twenty-four member expert panel and two research leads, and it contains twenty essential reporting items and two optional ones.

vom Brocke and colleagues (2009) put the search itself on the record. They examined review articles published in ten major information systems journals. They found that many did not thoroughly document how the literature had been searched. They argued that the search process must be described comprehensibly, so that a reader can judge the exhaustiveness of the review and other researchers can reuse its results. The record should state any language restriction, since a search of English-language databases cannot show what it left out.

A method built in medicine, adapted in computing

Brereton and colleagues (2006) report their experience of applying the systematic literature review process within software engineering. They find the basic process appropriate and single out preparing and validating a review protocol in advance as especially valuable. They also record where adaptation is needed: software engineering indexing databases are inadequate, and abstracts are of such poor quality that a study's relevance usually cannot be judged from its abstract alone.

Appraisal is the substance of the review, not an appendix to it

Webster and Watson (2002) draw the distinction on which the thesis turns. One review is author-centric and marches through the literature one source at a time. The other is concept-centric and organised around ideas through a concept matrix. Baumeister and Leary (1997) show that appraisal is not optional even in the least formal type. A narrative review can address broader and more abstract questions than an empirical report, can support a null result, and can use methodological diversity. A review that merely describes a series of studies has not accomplished enough to warrant publication. They also set out what a review is entitled to conclude. The list has four entries: that a hypothesis is correct, that it is the best current guess without being established, that it is false, or that the evidence permits no conclusion. A review that does not identify the biases in the literature it covers has not done what the form requires.

Contested question: is the systematic review the gold standard, or is the hierarchy spurious?

The first position holds that the systematic form is superior by method. As Greenhalgh, Thorne and Malterud (2018) describe it, systematic reviews are assumed to be better. They are assumed to have a focused question, to be methodologically explicit and to be less subject to bias. They trace that assumption to a comparison of review types across five leading biomedical journals. The proportion of reviews that were systematic ran from none in the New England Journal of Medicine to 11 per cent in the Lancet. It reached 72 per cent in the Annals of Internal Medicine. That makes the ranking a fact about editorial practice as well as about method.

The second position, which Greenhalgh and colleagues hold, is that the hierarchy is spurious. Conventional systematic reviews address narrowly focused questions, and their key contribution is summarising data. Narrative reviews provide interpretation and critique, and their key contribution is deepening understanding. On this account, a proposal that reaches for the systematic label to look rigorous, when its question is conceptual, has chosen the wrong instrument.

The third position is the qualified middle. Rowe (2014) requires systematicity for a review that aims at explanation but not for one that aims at understanding. Baumeister and Leary (1997) defend the narrative form where the goal is theory building. What would settle the dispute is a study the literature does not contain. It would review one question in both forms, and measure coverage, reproducibility across two independent teams, and whether each review answers the question it set. The thesis survives without that study, because it asks only that the type be declared and matched to the question.

Read by kind, both sides reason from the properties of the two forms and from counts of what journals publish, and neither supplies a head-to-head study of one question in both. Snyder (2019) and Rowe (2014) argue from the demands of the method, Grant and Booth (2009) argue from an analysis of fourteen types, and Greenhalgh, Thorne and Malterud (2018) argue from the two forms' different contributions. While the dispute is open, a student should declare the review type in the method, state the question that type can answer, and log the search that produced the studies. The strongest objection is that the unit's rule reverses the order Snyder defends: on her account the question and purpose determine the strategy, so a proposal that declares a type first may have fixed the wrong instrument.

Where the argument is heading

The direction of travel is from a one-off synthesis towards a maintained process with a published protocol and an update cycle. Iannizzi and colleagues (2023) surveyed the guidance for living systematic reviews, which are kept current by continual surveillance. They found guidance in seventeen of the twenty-one included papers for conducting a review, in six for reporting and in only two for appraising its quality. Their findings were used to prepare a PRISMA extension for living reviews. Tricco and colleagues (2018) gave scoping reviews a reporting checklist, and vom Brocke and colleagues (2009) made the search log part of the evidence. Munn and colleagues (2018) predicted that scoping reviews would increasingly precede systematic ones.

The sources this unit rests on

Each note states what the work did, what it found and how it bears on the argument. Every identifier was resolved on 31 October 2026.

• Snyder, H. (2019). Literature review as a research methodology: An overview and guidelines. doi:10.1016/j.jbusres.2019.07.039 Argues the review is a research method and separates three approaches, systematic, semi-systematic and integrative, each matched to a purpose. The organising source for the claim that the type follows the question.
• Rowe, F. (2014). What literature review is not: diversity, boundaries and recommendations. doi:10.1057/ejis.2014.7 Defines a review as synthesis, the identification of biases and gaps, and the proposal of future directions, and grades how systematic it must be by its goal. Supplies the definition that stops a reading summary being called a review.
• Grant, M. J. and Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. doi:10.1111/j.1471-1842.2009.00848.x Analysed the vocabulary of review and synthesis against the SALSA framework, describing fourteen types. Found that few have a prescribed method and that many overlap, so a label secures nothing in itself.
• Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A. and Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors. doi:10.1186/s12874-018-0611-x Sets out the indications for a scoping review, to identify gaps, scope a body of literature, clarify a concept or investigate research conduct, and notes that scoping can precede a systematic review. Gives the unit its decision rule.
• Arksey, H. and O'Malley, L. (2005). Scoping studies: towards a methodological framework. doi:10.1080/1364557032000119616 Set out the first framework for scoping studies and listed their purposes, including testing whether a full systematic review is warranted, with consultation optional. The origin point for the claim that mapping answers a different question.
• Levac, D., Colquhoun, H. and O'Brien, K. K. (2010). Scoping studies: advancing the methodology. doi:10.1186/1748-5908-5-69 Refined that framework from three of their own scoping studies, requiring a clear purpose-question link, a balance of feasibility against breadth and consultation as a required stage. Shows that scoping still needs a purpose and a stopping rule.
• Page, M. J., McKenzie, J. E., Bossuyt, P. M. and colleagues (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. doi:10.1136/bmj.n71 Replaces the 2009 guideline with a twenty-seven item checklist, an expanded checklist and an abstract checklist. The unit's reporting standard for making a review auditable.
• Tricco, A. C., Lillie, E., Zarin, W. and colleagues (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. doi:10.7326/m18-0850 Reports a scoping review reporting guideline developed by a twenty-four member panel and two research leads, with twenty essential and two optional items. Shows that scoping can be reported as carefully as systematic review.
• vom Brocke, J., Simons, A., Niehaves, B., Riemer, K., Plattfaut, R. and Cleven, A. (2009). Reconstructing the giant: on the importance of rigour in documenting the literature search process. https://aisel.aisnet.org/ecis2009/161/ Examined review articles in ten information systems journals and found that many did not document their search process, then proposed guidelines for documenting it. Carries the claim that the search log is part of the evidence.
• Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M. and Khalil, M. (2006). Lessons from applying the systematic literature review process within the software engineering domain. doi:10.1016/j.jss.2006.07.009 Applied the systematic literature review process in software engineering. Found the basic process appropriate and protocol preparation valuable, but indexing databases inadequate and abstracts too poor to judge relevance from. Carries the transfer-and-adapt claim.
• Webster, J. and Watson, R. T. (2002). Editorial in MIS Quarterly, 26(2), on writing a literature review. doi:10.2307/4132319 Argues that an effective review creates a conceptual foundation for theory rather than summarising a field, and recommends a concept-centric review presented through a concept matrix. The source of the distinction between listing sources and positioning them.
• Baumeister, R. F. and Leary, M. R. (1997). Writing Narrative Literature Reviews. doi:10.1037//1089-2680.1.3.311 Argue that narrative reviews can address broader questions, support null results and use methodological diversity, and that a review merely describing a series of studies warrants no publication. The defence of the narrative form and the demand for critique.
• Greenhalgh, T., Thorne, S. and Malterud, K. (2018). Time to challenge the spurious hierarchy of systematic over narrative reviews? doi:10.1111/eci.12931 Argues that systematic reviews address narrow questions and summarise data, while narrative reviews interpret and critique to deepen understanding, so the two are complementary rather than ranked. The contested half of the unit's thesis.
• Iannizzi, C., Akl, E. A., Anslinger, E. and colleagues (2023). Methods and guidance on conducting, reporting, publishing, and appraising living systematic reviews: a scoping review. doi:10.1186/s13643-023-02396-x Mapped guidance for living systematic reviews and found it thinnest for reporting and appraisal, then used the findings to prepare a PRISMA extension. Carries the claim that the review is becoming a maintained process.