Intercoder agreement: what a reliability figure can and cannot show

Unit 2 ended with a test for a codebook: another person should be able to follow it and reach the analyst's coding. This third of ten units follows that test as it is converted into a number, and the number is then asked to do work it cannot.

An agreement statistic, the argument goes, measures the shared conventions and training of coders more than it measures the truth of an interpretation. Those conventions include how coders cut the text into units. Read as a description of the instrument, the figure is a diagnostic. Read as the criterion of a qualitative claim, it certifies what it was never built to see. The three units that follow turn to the named methods.

The figure measures the coders, not the text

Cohen (1960) introduced the coefficient from which most of this literature descends. It is an index of agreement between two coders on nominal categories, and it corrects observed agreement for the agreement expected by chance. The correction answers a real objection. Two coders who lean on one dominant category agree often without applying the same rule. The correction also fixes the object of measurement: what is compared is neither the text nor the phenomenon. It is the application of categories by particular people. Those people need not both be human. Bijker and colleagues (2024) had ChatGPT build coding schemes from 537 forum posts and apply each in repeated conversations, then computed Fleiss' kappa for agreement across those runs: 0.72 to 0.82 for the inductive schemes and 0.58 to 0.73 for the deductive ones. The coders a coefficient compares can now be repeated runs of one model, so a high figure records the stability of a prompt rather than a shared training. The authors state their own limit: they assessed reliability, not validity. Hayes and Krippendorff (2007) argue that conclusions from data generated by trained observers can be trusted only after their reliability has been demonstrated. They propose Krippendorff's alpha as the standard, general across numbers of observers, measurement levels, sample sizes and missing data. Krippendorff (2019) places unitising, sampling and recording among the components that precede the constructs a study reports, and reliability and validity among the techniques that follow. Agreement is a property of the path from text to construct, not of the interpretation the construct carries.

The coefficient is a choice, and the choice is substantive

Were the figure a natural quantity, analysts holding one data set and one codebook would report one value. They do not. Popping (2010) surveys the views of agreement and argues that research should use a single one: mean pairwise agreement. Hayes and Krippendorff (2007) make the same move for a different coefficient. They argue that proliferating proposals leave investigators unsure which to choose, and propose alpha as the standard.

Zapf and colleagues (2016) test that presupposition in a large simulation of the two measures and their confidence intervals, with a case study alongside. Point estimates did not differ. The measures parted company in other conditions. Alpha was stable when data were missing completely at random, where a complete-case approach for Fleiss' kappa was biased. The asymptotic interval for Fleiss' kappa fell short of its nominal coverage, while bootstrap intervals for both came close to the theoretical value. Their recommendation follows: use either measure with a bootstrap interval for complete nominal data. Use alpha when data are missing or the scale is finer. Two analysts with one data set and two defensible commitments can publish different numbers, and neither has erred.

Segmentation decides part of the figure before meaning is in question

Kurasaki (2000) observes that the popular method for establishing reliability presents predetermined text segments to coders. That risks altering meanings by lifting text out of context, and it pre-empts the judgement about how much text is codable. Her procedure ran the other way, with coders working on free-flowing text and choosing the segment length. Twenty third-generation Japanese American men and women were interviewed. Reliability was above .80 for fifteen of the seventeen themes, with an average agreement of .90. Coders select their own segments. The analysis then samples lines at random. It compares the codes given within five lines. A difference of a few words then does not register as a disagreement about the category.

Two analysts can hold identical definitions and boundaries. Their agreement still depends on whether they cut a passage into the same units. Krippendorff (2019) treats unitising as a component settled before recording begins. A project that reports a coefficient but leaves the unit definition and segmentation rule unstated reports a number whose largest input is invisible.

Training and codebook revision are what the number moves with

The strongest case for keeping an agreement figure is that it detects a defective instrument. Hruschka and colleagues (2004) examined coding across three HIV-related studies for the CDC, the US public health agency. The populations were in the United States and Zimbabwe. A team initially produces very different codings. Codebook revision and recoding can then establish strong reliability, with most codes reaching a kappa of about 0.8. The figure signals an under-specified code rather than a verdict on an interpretation. The repair belongs to the codebook.

Burla and colleagues (2008) show the mechanism at a smaller scale. Two researchers independently coded ten transcripts from a study of low back pain, and they read the kappa of .67 as satisfactory to solid. Varying agreement rates identified problems in the scheme: low agreement on particular codes showed they had been defined too broadly, and the scheme was clarified before the analysis was completed. O'Connor and Joffe (2020) generalise the benefits: systematicity, communicability, transparency, reflexivity and dialogue within teams, and the ability to persuade audiences of an analysis's trustworthiness.

The bands are a convention, and the reporting is thin

Interpretation of the figure rests on a benchmark that is itself a convention. Landis and Koch (1977) developed generalised kappa-type statistics and supplied the bands the literature cites. They label agreement from poor below zero through slight, fair, moderate and substantial to almost perfect at 0.81 and above. The labels are ordinal descriptions of an index, not requirements a qualitative analysis must meet. Burla and colleagues (2008) read a kappa of .67 as satisfactory to solid. That is a reasoned judgement about a coding scheme, not a band the value falls into.

Reporting practice is the weaker link. Lombard, Snyder-Duch and Bracken (2002) examined how reliability was assessed and reported in the communication literature. Their sample was 200 content-analysis studies published between 1994 and 1998. Researchers often failed to assess or report it, and figures that appeared often rested on percent agreement, an overly liberal index. Only 69 per cent of the 200 studies carried any reliability figure at all. The average report spent four and a half sentences on it.

Contested question: does a reliability figure belong in qualitative research at all?

The disagreement is not about arithmetic. It is about what the arithmetic is for, and the evidence on each side is weaker than the confidence with which each side is stated.

Morse (1997) holds that the pursuit of inter-rater reliability is a myth. Her evidence is a three-page editorial that reports no study. She argues that agreement is bought by simplifying insights, coding and analyses until another person can place the same datum in the same category, and that a consensus reached that way is healthy but dead. Armstrong and colleagues (1997) supply the empirical version, whose scope is a single focus group transcript read by six researchers. The dispersion sat in how each analyst packaged the themes, which is the layer a coefficient cannot see.

Against them stands the case that agreement is a precondition of a shared construct, and its evidence is statistical rather than comparative. Hayes and Krippendorff (2007) argue that conclusions from coded data cannot be trusted until reliability has been demonstrated. Kurasaki (2000) treats intercoder reliability as a proxy for the validity of constructs from open-ended data. Burla and colleagues (2008) show a figure doing diagnostic work on ten transcripts, a check that assumes a second coder on the team. O'Connor and Joffe (2020) concede that some regard the step as unnecessary and defend it as good practice. Zapf and colleagues (2016) do not enter the dispute at all, because their evidence is a large simulation of two measures; it settles which index to use, not whether to use one.

The strongest counter-argument to this unit's own claim is that a shared construct cannot be shown to exist without some demonstration that the coders share it. On that reading the sceptics have asserted a loss they never measured, and the coefficient is the only evidence that two analysts applied one category rather than two.

What would settle it is a comparison the literature does not contain. Hold the codebook constant, vary the freedom to segment, and have the accounts judged on their defensibility by readers who never see the coefficient.

What to do while the dispute is open

Report the figure as a diagnostic of the instrument, never as the criterion of the claim. Give it with the unit definition, the segmentation rule, the measure and an interval, and say what it was used for. Where a reviewer demands a coefficient, supply it and state the reading it cannot support. Keep the free-flowing texts, so the segmentation decision can be re-examined.

Where the argument is heading

The direction of travel is towards fewer bare coefficients. Zapf and colleagues (2016) ask for an interval with the estimate. Popping (2010) asks for one agreed view of agreement, and Hayes and Krippendorff (2007) for one agreed measure. O'Connor and Joffe (2020) move the emphasis from the number to the process, treating assessment as a site of team reflexivity. Krippendorff (2019) keeps unitising among the components settled before recording. Hruschka and colleagues (2004) show the figure as the product of an iterative repair of the codebook. The emerging convention has the coefficient arrive with the unit definition, the segmentation rule, the measure and an interval. It documents the instrument rather than grading the analysis.

The sources this unit rests on

• Armstrong, D., Gosling, A., Weinman, J. and Marteau, T. (1997). Six researchers identified themes in one focus group transcript. Agreement on the basic themes was close, but each packaged them differently, placing the dispersion in the construction of the account.
• Bijker, R., Merkouris, S. S., Dowling, N. A. and Rodda, S. N. (2024). Bijker and colleagues had ChatGPT build coding schemes from 537 forum posts and apply each in repeated conversations, computing Fleiss' kappa across the model's own runs: 0.72 to 0.82 for the inductive schemes and 0.58 to 0.73 for the deductive ones. The coders the figure compares can be repeated runs of one model, and the authors assessed reliability rather than validity.
• Burla, L., Knierim, B., Barth, J., Liewald, K., Duetz, M. and Abel, T. (2008). Two researchers independently coded ten transcripts; the kappa of .67 was read as satisfactory to solid. Varying agreement rates identified codes defined too broadly, and the scheme was clarified. The figure works here as a diagnostic of the codebook.
• Cohen, J. (1960). Cohen sets out the coefficient that corrects observed agreement between two coders on nominal categories for chance agreement. It is the origin of the chance-corrected index the later debate argues over, and most of the statistics discussed here are its descendants or deliberate alternatives.
• Hayes, A. F. and Krippendorff, K. (2007). Hayes and Krippendorff describe the criteria for a good reliability measure and propose Krippendorff's alpha as the standard, general across numbers of observers, sample sizes and missing data. It is the unit's clearest case for a figure.
• Hruschka, D. J., Schwartz, D., St. John, D. C., Picone-Decaro, E., Jenkins, R. A. and Carey, J. W. (2004). Three HIV-related studies for the CDC showed that a team initially produces very different codings, and that codebook revision and recoding brought most codes to a kappa near 0.8. The figure documents an iterative repair of the instrument.
• Krippendorff, K. (2019). The fourth edition sets out the history, principles and design logic of content analysis, placing unitising, sampling and recording among the components that precede analytical constructs and reliability and validity among the evaluative techniques. It frames the figure as belonging to the instrument.
• Kurasaki, K. S. (2000). Kurasaki argues that intercoder reliability can serve as a proxy for the validity of constructs drawn from open-ended data, and shows that presenting predetermined segments to coders risks altering meanings by lifting text out of context. Her free-flowing procedure reached agreement above .80 on fifteen of seventeen themes.
• Landis, J. R. and Koch, G. G. (1977). Landis and Koch develop generalised kappa-type statistics and give the bands the applied literature cites, labelling agreement from poor below zero through slight, fair, moderate and substantial to almost perfect at 0.81 and above. Those bands are a convention, not a standard for an analysis.
• Lombard, M., Snyder-Duch, J. and Bracken, C. C. (2002). Lombard and colleagues examined how reliability was assessed and reported in 200 content-analysis studies published from 1994 to 1998. Many failed to assess or report reliability, and the common resort was percent agreement, an overly liberal index.
• Morse, J. M. (1997). Morse's three-page editorial argues that the pursuit of inter-rater reliability is a myth, since agreement is bought by simplifying the analysis until another person can place the same datum in the same category, and a consensus reached that way is healthy but dead. The note rests on the title and later quotations of it.
• O'Connor, C. and Joffe, H. (2020). O'Connor and Joffe review the arguments for and against intercoder reliability. They register the charge that it is inappropriate or unnecessary, and list its benefits as systematicity, communicability, transparency, reflexivity, dialogue within teams and trustworthiness.
• Popping, R. (2010). Popping surveys the views of agreement and argues for using only one, mean pairwise agreement. The choice of index is a substantive decision, so two analysts can defend different figures for one data set.
• Zapf, A., Castell, S., Morawietz, L. and Karch, A. (2016). Zapf and colleagues ran a large simulation of Fleiss' kappa and Krippendorff's alpha. Point estimates did not differ, but alpha was stable under missing data where a complete-case Fleiss' kappa was biased, and the asymptotic interval for Fleiss' kappa had low coverage.