Venues are communities of judgement, and their judgements are studyable

The previous unit read a paper as a claim and its evidence; this one asks who judges it. A venue is not a container into which findings are dropped. This unit argues that the community decides what a finding is worth, and that named people make that judgement under stated rules. Because the research on peer review has measured how it works, those judgements are studyable. That research shows they are moved by things unrelated to the work: prestige, the visibility of the author's identity, and the incentives around publication. Two versions of the same study can appear in print and only one count.

The venue is a community, and its output can be mapped

Liu and colleagues analysed 3,152 articles and 16,035 keywords from the CHI conference between 1994 and 2013, using co-word analysis to map two decades of the field (Liu et al., 2014). The published record served as data about the community rather than a list of findings. Boudreau and colleagues put a number on how that judgement bends. They ran a grant-proposal competition at a research university and randomised which evaluators saw which proposals, generating 2,130 evaluator-proposal pairs. Evaluators scored proposals lower when they were close to their own expertise and lower again when strongly novel. The authors read the pattern as the bias of a boundedly rational evaluator rather than as noise. The proposals a funder most wants found are exactly the unusual ones.

Kostakos (2015) argues, in a position piece rather than a study, that HCI lacks persistent mainstream themes and rolls from topic to topic without developing any substantially. Read as a hypothesis about what the community rewards, it is what Liu's method could test.

Tennant and colleagues reviewed innovations in peer review and argue the monolithic model is being replaced by options. The community judging a paper must therefore be identified rather than assumed (Tennant et al., 2017).

Reliability: how much judgement sits inside a verdict

Bornmann (2011) surveyed the empirical research and reports low agreement between reviewers and modest predictive power for recommendations. Siler, Lee and Bero (2015) measured it. Across 1,008 manuscripts at three elite medical journals, citation outcomes followed editor and reviewer appraisals only weakly. The finding cuts two ways: acceptance is a judgement rather than a reading of intrinsic quality, and some of what moves a verdict may be noise.

Whether concealment changes the review is separate, and the evidence is thin. Jefferson and colleagues reviewed 28 studies and found no clear-cut evidence that concealing author or reviewer identity changed the quality assessment (Jefferson et al., 2007).

Prestige moves the judgement

Okike and colleagues sent an identical fabricated manuscript, carrying five subtle errors, to 119 reviewers at one orthopaedic journal. They randomised each to see or not see two past presidents of a professional body as its authors (Okike et al., 2016). Reviewers who saw the names recommended acceptance about 87 per cent of the time, against about 68 per cent when they were removed. Both groups found the same number of errors, so the difference is a property of the authors' standing rather than the work.

Tomkins, Zhang and Heavlin tested this at a computer science conference with a 15.6 per cent acceptance rate (Tomkins, Zhang and Heavlin, 2017). Every submission was scored by two reviewers who saw the authors' names and affiliations and two who did not. Once papers were allocated, the single-blind reviewers were significantly more likely to recommend acceptance for work from famous authors, top universities and top companies. The odds multipliers were 1.63, 1.58 and 2.10. Read from the other side, the same odds are a penalty on the author at a less-resourced institution.

Blinding, and the remedy that did not arrive

The obvious remedy is to hide the author's identity, and two randomised trials tested it. van Rooyen and colleagues randomised 527 manuscripts to blinded or revealed authorship and to masked or unmasked review (van Rooyen et al., 1998). Mean quality scores differed little, at 2.82 for masked against 2.96 for unmasked, and recommendations and review time did not differ significantly. Godlee, Gale and Martyn planted eight weaknesses in an accepted paper and allocated 420 reviewers to blinded or unblinded conditions (Godlee, Gale and Martyn, 1998). Of the 221 reports returned (53 per cent), blinding made no significant difference to the weaknesses found.

The two results do not conflict. The prestige studies isolate the effect of author identity on a decision about a known paper. The trials asked a harder question: does removing identity improve the quality of written reviews? Blinding narrows one channel through which standing enters a verdict and leaves the rest of the instrument much as it was (Tomkins, Zhang and Heavlin, 2017).

Incentives, and what the pressure to publish does

A venue does not only judge; it sits inside a reward structure that shapes the work offered to it. Fanelli (2010) measured how often a large random sample of papers by United States based authors supported a tested hypothesis. Positive results were more likely where states produced more academic papers per capita. He reads that as competitive environments increasing bias as well as productivity, though prestige could not be ruled out, so it is an association rather than a mechanism.

Policy can move the judgement the other way. Bravo and colleagues compared 9,220 submissions and 18,525 reviews before and during a pilot in which five journals could publish referee reports (Bravo et al., 2019). Publishing did not significantly compromise willingness to review, recommendations or turnaround times, and only 8.1 per cent of referees revealed their identity.

Contested question: biased judgement, or noisy judgement?

The literature agrees that a venue's verdict is a judgement and disagrees about what imperfection dominates it. One position holds it is systematically biased by features of the authors. Okike and colleagues (2016) give the controlled demonstration for prestige, and Tomkins and colleagues (2017) put odds multipliers on it. The opposing position holds it is mostly unreliable rather than biased. Bornmann (2011) reports low inter-reviewer agreement, and Jefferson and colleagues (2007) find no clear-cut evidence that concealment changes the quality assessment of a report. Lee and colleagues make the strongest version of that challenge, arguing that methodological limits in the bias literature raise questions about the existence and extent of many hypothesised biases (Lee et al., 2013).

The test that separates them is whether a bias survives when quality is held constant. Okike's fabricated manuscript is that test, since identical papers with different author attributions produced roughly a nineteen point gap in acceptance recommendations; Tomkins's four-reviewer design is a second version on real submissions. What would settle the question is a trial at scale, randomising real submissions to blinded and unblinded review and comparing outcomes for papers whose quality is estimated independently. No such trial exists.

The two sides rest on different instruments. The bias position has two controlled comparisons, a randomised trial on a fabricated manuscript and four reviewers on the same real submissions. The noise position rests on a survey of agreement and a review of studies that did not hold quality constant. The controlled designs are the stronger evidence of bias; the agreement figures, of its random component. While the dispute is open, a designer should read a venue's verdict as a judgement, and report the review conditions alongside the finding.

The strongest counter-argument: the evidence is narrow

The strongest objection is that the evidence is narrow and may tell the wrong story. The controlled bias results come from one journal and one conference, and Okike and colleagues say generalisation from a single specialty is unknown (Okike et al., 2016). The two trials of blinding find that removing identity does not improve recorded review quality (van Rooyen et al., 1998; Godlee, Gale and Martyn, 1998). A sceptic can conclude the error is mostly random, and random error is not a community deciding what a finding is worth. The reply is that reliability and bias are different quantities: low agreement does not show the disagreement is unbiased, and where quality is held constant the identity effect is real.

Where the argument is heading

The direction of travel is from treating a venue as a fact of nature to treating it as a designed, auditable process. Ross-Hellauer (2017) found the phrase "open peer review" covers at least 22 distinct configurations of seven traits across a corpus of 122 definitions. The single review model is now a set of named design choices. Bravo and colleagues (2019) show that publishing the reports changes referee behaviour without breaking the process. One venue has gone further. Since January 2023 eLife has published every manuscript it reviews as a reviewed preprint with a public assessment and the reviewers' reports, and no accept or reject decision; a year on it reported more than 6,200 submissions and no significant change in the quality of what it reviewed (Behrens et al., 2024). A venue's judgement is becoming something read off its published policy and record rather than inferred from its name.

The sources this unit rests on

Each note summarises what the cited document did and found, not a quotation, and every record was checked against the publisher's registered metadata on 1 November 2026. Where a published title contains a spelling the house style forbids, the British form is given.

• Tennant, J. P., Dugan, J. M., Graziotin, D., et al. (2017). A multi-disciplinary perspective on emergent and future innovations in peer review. doi:10.12688/f1000research.12037.3 Reviews emergent models of peer review and concludes that the single model is being replaced by options that must be identified rather than assumed.
• Lee, C. J., Sugimoto, C. R., Zhang, G., et al. (2013). Bias in peer review. doi:10.1002/asi.22784 Reviews the empirical literature on bias in peer review and argues that its methodological limits raise questions about the existence and extent of many hypothesised biases.
• Liu, Y., Goncalves, J., Ferreira, D., et al. (2014). CHI 1994-2013: mapping two decades of intellectual progress through co-word analysis. doi:10.1145/2556288.2556969 Analysed 3,152 CHI articles and 16,035 keywords to map the conference's thematic evolution, sorting topics into popular, core and backbone groups.
• Fanelli, D. (2010). Do pressures to publish increase scientists' bias? An empirical support from US states data. doi:10.1371/journal.pone.0010271 Measured how often papers by United States based authors supported a tested hypothesis, finding positive results more likely in states producing more papers per capita. Fanelli reads this as competition increasing bias, and prestige could not be ruled out.
• Jefferson, T., Rudin, M., Brodney Folse, S., et al. (2007). Editorial peer review for improving the quality of reports of biomedical studies. doi:10.1002/14651858.MR000016.pub3 Reviewed 28 studies of editorial peer review and found no clear-cut evidence that concealing author or reviewer identity changed the outcome of the quality assessment process. Standardising interventions showed inconsistent effects.
• van Rooyen, S., Godlee, F., Evans, S., et al. (1998). Effect of blinding and unmasking on the quality of peer review: a randomized trial. doi:10.1001/jama.280.3.234 Randomised 527 manuscripts to blinded or unblinded review, following up 467 (89 per cent). Mean quality scores differed little, with no difference in recommendations or review time.
• Godlee, F., Gale, C. R. and Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports. doi:10.1001/jama.280.3.237 Randomised 420 reviewers of a paper carrying eight planted weaknesses, receiving 221 reports (53 per cent). Blinding made no significant difference to the weaknesses found.
• Tomkins, A., Zhang, M. and Heavlin, W. D. (2017). Reviewer bias in single- versus double-blind peer review. doi:10.1073/pnas.1707323114 Had two single-blind and two double-blind reviewers score every submission to a conference with a 15.6 per cent acceptance rate. Single-blind reviewers favoured famous authors and top institutions, with odds multipliers of 1.63, 1.58 and 2.10.
• Bornmann, L. (2011). Scientific peer review. doi:10.1002/aris.2011.1440450112 Surveys the empirical research on peer review and reports low agreement between reviewers and modest predictive power for recommendations, warning that what moves a verdict may be unreliability rather than bias.
• Okike, K., Hug, K. T., Kocher, M. S., et al. (2016). Single-blind vs double-blind peer review in the setting of author prestige. doi:10.1001/jama.2016.11014 Sent an identical fabricated manuscript with five planted errors to 119 reviewers, randomising whether prestigious names were visible: about 87 per cent recommended acceptance, against about 68 per cent when they were not.
• Kostakos, V. (2015). The big hole in HCI research. doi:10.1145/2729103 Argues, in a position piece rather than a study, that HCI lacks persistent mainstream themes and moves from topic to topic without developing any substantially.
• Bravo, G., Grimaldo, F., Lopez-Inesta, E., et al. (2019). The effect of publishing peer review reports on referee behaviour in five scholarly journals. doi:10.1038/s41467-018-08250-2 Compared 9,220 submissions and 18,525 reviews before and during a pilot in which five journals could publish referee reports. Publishing did not significantly compromise willingness to review, recommendations or turnaround times, and only 8.1 per cent of referees revealed their identity.
• Ross-Hellauer, T. (2017). What is open peer review? A systematic review. doi:10.12688/f1000research.11369.2 Systematically reviewed definitions of open peer review from a corpus of 122 definitions. It found 22 distinct configurations of seven traits, so openness is a family of design choices rather than one practice.
• Boudreau, K. J., Guinan, E. C., Lakhani, K. R., et al. (2016). Looking across and looking beyond the knowledge frontier: intellectual distance, novelty, and resource allocation in science. Management Science. doi:10.1287/mnsc.2015.2285 Ran a grant-proposal process at a research university and randomised evaluator and proposal assignments to generate 2,130 evaluator-proposal pairs, finding lower scores for proposals close to an evaluator's expertise and for highly novel ones. It reads the pattern as bias against the unfamiliar.
• Siler, K., Lee, K. and Bero, L. (2015). Measuring the effectiveness of scientific gatekeeping. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1418218112 Analysed 1,008 manuscripts at three elite medical journals and found that citation outcomes followed editor and reviewer appraisals, while 12 of the 14 papers that became most cited were rejected without review.
• Behrens, T. E., Dalal, Y., Harper, D. M., et al. (2024). The first year of a new era. eLife. doi:10.7554/elife.96413 The editors' account of eLife's first year without accept or reject decisions, in which every reviewed manuscript is published as a reviewed preprint with a public assessment: more than 6,200 submissions and no significant change in assessed quality. A self-report by the journal rather than an independent evaluation.