A systematic review is a method, not a reading list

The earlier units treated the single study as the unit of evidence and asked what a reader can recover from it. This one takes the next step, the synthesis, where many studies become one claim about a field. This unit argues that a synthesis is only as trustworthy as the protocol that produced it. The reporting standards now governing systematic reviews make the search, the screening and the appraisal auditable. The method transfers imperfectly into human-computer interaction, however, where the evidence is often qualitative.

What makes a synthesis a method rather than a reading

A reading list is assembled. A systematic review is conducted, and the conducting leaves a trail. The question, the databases, the search strings, the inclusion criteria and the appraisal of each study are fixed and reported, so that a second person could repeat the exercise. Moher and colleagues published the first PRISMA statement, with a checklist and a four-phase flow diagram. It runs from the records the search identified, through screening and eligibility, to the studies included (Moher et al., 2009). The diagram makes the author say how many records the search returned and how many fell away at each stage, which is where a selective synthesis becomes visible. A search string in English, run over a handful of indexed databases, also sets the edge of the literature: work published in another language, or in a journal the database does not hold, is outside the count. Page and colleagues updated the standard as PRISMA 2020, a twenty seven item checklist with an expanded checklist, an abstract checklist and revised flow diagrams (Page et al., 2021a). A companion explanation and elaboration paper gives the reason for each item (Page et al., 2021b).

The question chooses the review type, and the type chooses the search

Sutton and colleagues catalogued review types from published typologies and found forty eight in seven families, then searched for the retrieval guidance each type imposes. Guidance is becoming more specific, they found, but much of it remains generic. Many types carry no explicit requirement for how evidence is to be identified (Sutton et al., 2019).

Munn and colleagues give the decision rule for the commonest fork. A scoping review suits identifying knowledge gaps, mapping a body of literature, clarifying concepts or examining research conduct. A systematic review suits a narrow question that evidence can settle (Munn et al., 2018). Their concern is mislabelling in both directions, and the label is the reader's only guide to what the paper can claim.

Scoping reviews show that the standards are written, not found

Arksey and O'Malley set out the founding framework for scoping studies, with stages from identifying the question through searching, selecting and charting to summarising. They argue that a consultation exercise would make the results more useful to practitioners (Arksey and O'Malley, 2005). Levac, Colquhoun and O'Brien refined the framework from three scoping studies, linking purpose to question, balancing feasibility against comprehensiveness, and making stakeholder consultation a required stage (Levac et al., 2010). Tricco and colleagues then wrote the reporting standard for the genre, a PRISMA extension from a panel of twenty four experts that contains twenty essential items and two optional ones (Tricco et al., 2018).

Pooling studies into one number adds assumptions of its own

Meta-analysis is the statistical core of synthesis and a research design with assumptions of its own. Gurevitch and colleagues describe it as the quantitative synthesis that established evidence-based practice, and record that its implementation has drawn criticism (Gurevitch et al., 2018). Pooling requires a decision about which effect to extract from each study, how much the studies differ, and whether those differences make the pooled number meaningless. The standard textbook teaches the method on three small datasets: one of continuous outcomes, one of binary outcomes and one of correlations. The continuous-data example reduces every study to a single standardised mean difference before anything is combined. The model, the estimate of how much the studies differ and the judgement about whether pooling is defensible all rest on that reduced set. Higgins and Thompson proposed statistics, including I-squared, that do not depend on the number of studies or the effect metric. They define I-squared as the proportion of total variation in study estimates due to heterogeneity rather than chance. It, or the related statistic H, should be reported in preference to a test for heterogeneity (Higgins and Thompson, 2002). Pigott and Polanin set out what a methodologist requires of a high-quality meta-analysis. The requirements are the review methods behind it, explicit description of the distribution of effect sizes, attention to dependence among effects from one study, and reporting transparent enough to reproduce (Pigott and Polanin, 2020).

A reporting standard is not a method

PRISMA is a reporting guideline rather than a method for conducting a review, as its own authors say (Page et al., 2021a). A review can therefore satisfy every item and remain weak. It can report its search honestly and still have asked a question the search could not answer, or pool studies measuring different things. Reporting standards raise the floor without raising the ceiling.

In HCI the method establishes what was built, not what it showed

The two HCI syntheses in this unit's list show the reach and the limit at once. Hertzum meta-analysed 105 studies reporting the System Usability Scale alongside workload, task time and error rate for pairs of systems. He found the questionnaire correlated strongly with self-reported workload, while its relation to performance was weak, so the two are related but somewhat independent. The higher-scoring system imposed higher workload in only ten per cent of the studies. It produced the poorer task time in twenty four per cent, and the poorer error rate in twenty three per cent (Hertzum, 2026). That limits what a usability questionnaire licenses a team to claim.

Shi's synthesis of interaction design in spherical video-based virtual reality for learning meta-analysed twenty five articles from 2016 to 2025. It extracted five interaction types, among them object manipulation and test and voice scaffolds. It reports that most implementations still deliver content passively (Shi, 2026). A meta-analysis of twenty five studies establishes what the field has built, not yet what it has shown.

Contested question: trustworthy instrument, or mass-produced genre?

The literature agrees that a protocol makes a synthesis auditable and disagrees about whether the genre as practised produces trustworthy evidence. Ioannidis holds that it does not. From PubMed surveys he reports that between 1986 and 2015 the database tagged 266,782 items as systematic reviews and 58,611 as meta-analyses. Annual output from 1991 to 2014 grew by 2,728 and 2,635 per cent against 153 per cent for all indexed items. He concludes that possibly the large majority are unnecessary, misleading or conflicted (Ioannidis, 2016).

The standards camp, the authors of the guidance this unit rests on, offers the opposite diagnosis. Page, Tricco, Munn and Pigott and Polanin do not dispute that weak reviews exist. They hold that those are failures of conduct against a specification that now exists. A review which fixes its question, reports its counts and reports heterogeneity when it pools is therefore evidence (Page et al., 2021a; Tricco et al., 2018; Munn et al., 2018; Pigott and Polanin, 2020).

Three things would settle it, and none is another editorial. First, an audit of registered protocols, checking whether the search was designed after the question (Sutton et al., 2019; Munn et al., 2018). Second, a count, at publication, of reviews duplicating an existing one, the redundancy Ioannidis describes rather than infers (Ioannidis, 2016). Third, evidence that a synthesis changed a decision, which no paper in this list reports.

The strongest counter-argument to this unit's thesis also comes from Ioannidis. If mass production of weak reviews is driven by publication incentives, and is as visible in medicine, the home of the method, as anywhere, then the problem is the genre rather than the transfer. HCI reviews are no worse than reviews elsewhere, and a heterogeneous, often qualitative evidence base may even guard against the over-pooling he documents. The reply is that both can be true. The audit is worth having because the incentive to publish reviews faster than evidence accumulates belongs to no single field.

The two camps rest on different objects. The evidence of Ioannidis is a count of items in a bibliographic database, and he infers quality from volume and growth, which is an inference from the record rather than a measurement of reviews. The standards camp offers specifications and the syntheses that follow them, of which a meta-analysis of 105 studies is the largest. Neither side has audited a sample of published reviews against its own protocol. While the dispute is open, a designer should read the search and the count at each stage before the pooled number, and ask whether the question was a map or a test.

Where the argument is heading

PRISMA 2020 replaced a single statement with a suite of checklists and revised flow diagrams, reflecting advances in how studies are identified, selected and synthesised (Page et al., 2021a). PRISMA-ScR does the same work for reviews whose question is a map (Tricco et al., 2018). What lags is retrieval, where Sutton and colleagues found guidance for many review types still generic, without explicit requirements for identifying evidence (Sutton et al., 2019). Report the protocol, report the counts, report heterogeneity when pooling, and say plainly when the question was a map rather than a test.

The sources this unit rests on

Each note states what the cited document did and found. Every DOI was resolved and its record read on 1 November 2026.

• Ioannidis, J. P. A. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. doi:10.1111/1468-0009.12210 Reports 266,782 items tagged as systematic reviews and 58,611 as meta-analyses between 1986 and 2015, with annual growth of 2,728 and 2,635 per cent against 153 per cent for indexed items, and concludes that possibly the large majority are unnecessary, misleading or conflicted.
• Page, M. J., McKenzie, J. E., Bossuyt, P. M., et al. (2021a). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. doi:10.1136/bmj.n71 Presents PRISMA 2020, which replaces the 2009 statement, with a twenty seven item checklist, an expanded checklist, an abstract checklist and revised flow diagrams. Its authors call it a reporting guideline rather than a method.
• Page, M. J., Moher, D., Bossuyt, P. M., et al. (2021b). PRISMA 2020 explanation and elaboration. doi:10.1136/bmj.n160 Explains each PRISMA 2020 item with the reason for it and an example from a published review, so that a reader can judge whether a review is trustworthy.
• Moher, D., Liberati, A., Tetzlaff, J., et al. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. doi:10.1371/journal.pmed.1000097 Introduces a reporting checklist and a four-phase flow diagram counting records from identification through screening and eligibility to inclusion. It is the origin of the count-at-each-stage discipline this unit describes.
• Tricco, A. C., Lillie, E., Zarin, W., et al. (2018). PRISMA extension for scoping reviews (PRISMA-ScR). doi:10.7326/M18-0850 Gives scoping reviews their own reporting standard, built by twenty four experts and two research leads, with twenty essential items and two optional ones. It legitimises a review whose question is a map.
• Munn, Z., Peters, M. D. J., Stern, C., et al. (2018). Systematic review or scoping review? Guidance for authors. doi:10.1186/s12874-018-0611-x Sets out when a scoping review suits identifying gaps, mapping a literature, clarifying concepts or investigating conduct, and when a narrow systematic review suits instead. Both demand transparent methods, and the labels are often misapplied.
• Arksey, H. and O'Malley, L. (2005). Scoping studies: towards a methodological framework. doi:10.1080/1364557032000119616 Sets out the founding scoping framework, from identifying the question through searching, selecting and charting to summarising, and argues that a consultation exercise makes the results more useful to practitioners.
• Levac, D., Colquhoun, H. and O'Brien, K. K. (2010). Scoping studies: advancing the methodology. doi:10.1186/1748-5908-5-69 Refines that framework from three scoping studies, balancing feasibility against comprehensiveness and making stakeholder consultation a required stage. It answers the criticisms the original attracted.
• Sutton, A., Clowes, M., Preston, L., et al. (2019). Meeting the review family: exploring review types and associated information retrieval requirements. doi:10.1111/hir.12276 Identifies forty eight review types in seven families and the retrieval guidance each imposes, finding the guidance generic for many types. The type is chosen before the search.
• Gurevitch, J., Koricheva, J., Nakagawa, S., et al. (2018). Meta-analysis and the science of research synthesis. doi:10.1038/nature25753 Reviews meta-analysis at its fortieth anniversary as the quantitative synthesis that established evidence-based practice and resolved contradictory findings, while recording criticism of its implementation, and treats pooling as a design with assumptions.
• Higgins, J. P. T. and Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. doi:10.1002/sim.1186 Defines I-squared as the proportion of total variation in study estimates due to heterogeneity rather than chance, and argues that it or the related statistic H should be reported instead of a test for heterogeneity.
• Pigott, T. D. and Polanin, J. R. (2020). Methodological guidance paper: high-quality meta-analysis in a systematic review. doi:10.3102/0034654319877153 Sets out what a high-quality meta-analysis requires: the review methods behind it, description of the distribution of effect sizes, and methods and results transparent enough to reproduce.
• Hertzum, M. (2026). System Usability Scale: a meta-analysis of how SUS relates to workload, task time, and error rate. doi:10.1080/10447318.2026.2625260 Meta-analysed 105 studies and found the System Usability Scale strongly correlated with self-reported workload but only weakly with performance: the higher-scoring system imposed higher workload in ten per cent of studies and poorer task time in twenty four per cent.
• Shi, Y. (2026). Beyond passive viewing: a systematic review and meta-analysis of human-computer interaction design modalities in spherical video-based virtual reality learning systems. doi:10.1080/10447318.2025.2562958 Meta-analysed twenty five articles from 2016 to 2025 and extracted five interaction types, among them object manipulation and test and voice scaffolds, while reporting that most implementations still deliver content passively.
• Borenstein, M., Hedges, L. V., Higgins, J. P. T., et al. (2009). Introduction to meta-analysis. Wiley. doi:10.1002/9780470743386 Sets out the theory and practice of meta-analysis with worked computations, teaching the method on datasets for continuous, binary and correlational outcomes. Each study is reduced to one effect size before anything is combined, which is the decision the pooled number rests on.