Sampling and who counts as a participant

A sample is an argument about who a finding is for. Who was recruited, and who refused, shapes a study's conclusions more than how many were recruited. A convenience sample fails in a direction rather than at random. Size governs precision. The frame governs validity, and the two cannot be traded for one another. A thousand responses drawn from the wrong list are not a noisier version of a good sample. They estimate the wrong population, and a second sample would err in the same direction.

User research inherits this problem rather than inventing it, and this second of the ten units asks who may answer the question the first one posed. The weakness is knowable in advance. A study can state who was in the frame, who returned, and how far the two differ on the measure that matters.

A sample is a claim about who the finding covers

A study that leaves its population unnamed lets its sample stand for humanity. Henrich, Heine and Norenzayan (2010) reviewed the comparative database across the behavioural sciences. They found substantial variability in experimental results between populations. Participants from Western, educated, industrialised, rich and democratic societies were among the least representative groups one could choose for generalising about humans. Their claim is not that a single result is false. It is that the warrant for calling one universal is missing.

Arnett (2008) shows the narrowness belongs to a literature rather than to one study. Americans make up under five per cent of the world's population. Her analysis of six premier APA journals finds contributors, samples and editorial leadership predominantly American, reported as though they described people in general. A design team that recruits its own users repeats the move on a smaller scale. The repair is procedural. A study should name the population before recruiting, and say who fell outside it.

Nonresponse fails in a direction, and size cannot repair it

The 1936 Literary Digest poll is the standard case because size and error sit together. Lusinchi (2012) reanalysed it. About ten million ballot cards went out from subscription, telephone and car-registration lists. Nearly 2.4 million came back, a response rate of roughly 24 per cent. The poll forecast a Landon victory. Roosevelt took 62.5 per cent of the two-party vote against a forecast 42.9 per cent. Lusinchi's central finding is that a majority of telephone and automobile owners, 53.9 per cent, in fact favoured Roosevelt. Poll respondents and nonrespondents backed opposite candidates. The bulk of the 19.6-point error was therefore nonresponse bias rather than frame bias.

Groves (2006) supplies the mechanism. He reviews the evidence on household surveys and finds the connection between nonresponse rates and nonresponse bias weak. Bias is governed by the gap between respondents and nonrespondents on the variable being estimated. Professional standards still press for high response rates. Who returns is a second sample drawn from the first by self-selection. The difference between the two groups decides the damage.

Bethlehem (2010) reviews the two faults that make web surveys unreliable. Under-coverage leaves groups with less access out of the frame, such as households with no reliable connection. Self-selection makes recruitment depend on who chooses to answer. He concludes that correction by adjustment weighting and reference surveys is imperfect, and that self-selection leads to unreliable survey outcomes. Scale buys precision inside the wrong distribution.

What a non-probability sample must supply first

The task force report of the American Association for Public Opinion Research is the field's own standard. Baker and colleagues (2013) conclude that no single framework covers non-probability sampling. Inference from such a sample is possible only while the model is stated and its departures tested. Sample matching is the most promising route for surveys. They add that selection bias risks a sample distribution differing from the population's by enough to mislead an inference. Elliott and Valliant (2017) give the formal version. Inference from a non-probability sample requires either quasi-randomisation, estimating pseudo-inclusion probabilities from covariates, or a superpopulation model. The burden rises with the strength of the claim.

A platform is a population, and one that moves

Paolacci, Chandler and Ipeirotis (2010) opened the practice this unit has to assess. They reported demographic data on the Turk workforce and compared effect magnitudes with traditional subject pools. That workforce is reachable only through a device, a connection and English, so the frame is set before recruitment begins. The effects reproduced, and they judged the platform viable. What followed showed the population is not stable. Kennedy and colleagues (2020) analysed 38 studies and found fraud by virtual private servers had increased recently. Fraudulent respondents supply particularly low-quality data and can weaken treatment effects. The usual defences had lost power.

Douglas, Ewell and Brauer (2023) compared five sources and found quality differences large enough to change what a study can claim. Prolific and CloudResearch outperformed MTurk and a general panel. The cost per high-quality respondent was $1.90 and $2.00 against $4.36 for MTurk. The validity evidence for a panel expires as its workforce changes.

Categorisation also decides who counts

Recruitment decides who is asked. Coding decides what can be said about them. Schlesinger, Edwards and Grinter (2017) reviewed a corpus of 140 CHI papers on gender, ethnicity, race, class and sexuality published between 1982 and 2016. They found that identity was usually analysed one facet at a time. Work on ethnicity and race lagged behind work on gender and socio-economic class. An unrecorded facet travels with a recorded one. A difference that the paper attributes to gender cannot then be separated from the class or the ethnicity its sample never recorded. Treating identity categories as independent variables can invert a comparison. How a sample is described is part of the sampling claim, not a reporting afterthought.

The participant is becoming optional

A newer proposal removes the recruiting problem by removing the participants. Argyle and colleagues (2023) conditioned GPT-3 on thousands of sociodemographic backstories. The resulting samples reproduced the response distributions of the corresponding human subgroups, a property they call algorithmic fidelity. A later benchmark found less. Bisbee and colleagues (2024) prompted ChatGPT to adopt personas and give feeling-thermometer scores for eleven sociopolitical groups. The averages tracked the American National Election Study closely. The samples did not: the synthetic responses varied less, regression coefficients often differed significantly from the survey estimates, and the same prompt returned a different distribution three months later. A matching average is not a matching sample. Agnew and colleagues (2024) surveyed the substitution proposals. They argue that these proposals conflict with three things participation does. Participation represents participants' interests, includes them, and builds understanding through engagement. A simulated participant is therefore a hypothesis rather than evidence. A model has no life to be sampled from, and its training text is itself a convenience sample.

Contested question: can a self-selected sample be repaired?

The literature agrees that self-selection is a fault and disagrees about whether it is fatal. Bethlehem (2010) holds that self-selection corrupts the estimate itself, so adjustment weighting and reference surveys cannot be trusted to restore it. Baker and colleagues (2013) take the other position. Non-probability samples can support inference once the modelling assumptions are explicit. They observe that such samples have often performed well in electoral polling, while the evidence of accuracy elsewhere is less clear. Groves (2006) sits between them. A low response rate is dangerous only in proportion to the difference between respondents and nonrespondents on the variable of interest.

Argument cannot settle the disagreement, because the two positions make different claims about a measurable quantity. What would settle it is a benchmark study. It would take estimates from a self-selected sample and compare each against a known population value on the same measure. It would report failures as well as successes, across several domains rather than election polling alone. The task force report asks for measures of bias and precision for non-probability estimates. Those measures do not yet let a reader judge a single study.

The three positions rest on different kinds of warrant. Bethlehem's warning comes from a review of what web surveys have delivered. Baker and colleagues write as a task force summarising practice. Groves reviews the survey literature on a quantity the other two largely assert. None reports a head-to-head comparison against a known population value. While the dispute is open, state the model rather than trusting the sample. The strongest objection is that a convenience sample with a stated model can beat a small probability sample on the same question.

Where the argument is heading

The direction is towards stating distance rather than deploring it. Muthukrishna and colleagues (2020) apply the fixation index to the World Values Survey. They build a scale of cultural and psychological distance from any chosen population. Their analysis of regional diversity finds the United States relatively homogeneous, so a national sample can conceal the spread inside it. Read beside Agnew and colleagues (2024), the programme is to declare what a sample is a sample of. That declaration is part of the result rather than a limitation paragraph.

The sources this unit rests on

• Henrich, J., Heine, S. J. and Norenzayan, A. (2010). The weirdest people in the world? Behav. Brain Sci. doi:10.1017/s0140525x0999152x Reviews the comparative database across the behavioural sciences and finds substantial variability between populations, WEIRD participants being frequent outliers and among the least representative groups, so who is sampled can invalidate a general claim.
• Arnett, J. J. (2008). The neglected 95%: Why American psychology needs to become less American. American Psychologist. doi:10.1037/0003-066x.63.7.602 Analyses articles in six premier APA journals and finds contributors, samples and editorial leadership predominantly American, while Americans are under five per cent of the world's population, so the narrowness belongs to a literature rather than to one study.
• Lusinchi, D. (2012). "President" Landon and the 1936 Literary Digest Poll. Social Science History 36(1), 23-54. doi:10.1215/01455532-1461650 Reanalyses the poll's nearly 2.4 million returns and argues that telephone and automobile owners mostly favoured Roosevelt, so nonresponse bias rather than frame bias carried most of the error. It separates two faults the unit keeps apart, though its single retrospective source leaves the revision contested.
• Groves, R. M. (2006). Nonresponse Rates and Nonresponse Bias in Household Surveys. Public Opinion Quarterly. doi:10.1093/poq/nfl033 Reviews survey nonresponse and finds rates and bias only weakly connected, because bias depends on how far respondents differ from nonrespondents on the measure of interest, which breaks the link between response rate and bias.
• Bethlehem, J. (2010). Selection Bias in Web Surveys. International Statistical Review. doi:10.1111/j.1751-5823.2010.00112.x Reviews under-coverage and self-selection in web surveys and the corrections available, concluding that under-coverage may ease with time while self-selection still yields unreliable outcomes, so a large web sample fails in a specific direction.
• Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J. and Tourangeau, R. (2013). Summary Report of the AAPOR Task Force on Non-probability Sampling. Journal of Survey Statistics and Methodology 1(2), 90-143. doi:10.1093/jssam/smt008 Reviews when quota, river and panel samples can support inference, concluding that no single framework covers non-probability sampling, that modelling assumptions must be stated, and that sample matching is most promising. It sets the justification a convenience sample must meet.
• Elliott, M. R. and Valliant, R. (2017). Inference for Nonprobability Samples. Statistical Science. doi:10.1214/16-sts598 Sets out the statistical conditions for inference from a nonprobability sample: either quasi-randomisation, with pseudo-inclusion probabilities estimated from covariates, or a superpopulation model. It states formally what convenience sampling would have to supply.
• Paolacci, G., Chandler, J. and Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgm. Decis. Mak. doi:10.1017/s1930297500002205 Reports demographic data on the Mechanical Turk workforce and compares effect magnitudes with traditional subject pools, finding the effects reproduce, which opened a practice the unit must now assess critically.
• Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R. and Winter, N. J. G. (2020). The shape of and solutions to the MTurk quality crisis. Political Science Research and Methods 8(4), 614-629. doi:10.1017/psrm.2020.6 Analyses 38 studies and finds fraud by virtual private servers rising, fraudulent respondents supplying low-quality data that weakens treatment effects, and the standard defences losing power.
• Douglas, B. D., Ewell, P. J. and Brauer, M. (2023). Data quality in online human-subjects research. PLOS ONE. doi:10.1371/journal.pone.0279720 Compares data quality across five sources on attention, honesty and completion, and computes the cost paid per high-quality respondent, from $1.90 on Prolific to $8.17 on Qualtrics, making platform choice an explicit sampling decision.
• Schlesinger, A., Edwards, W. K. and Grinter, R. E. (2017). Intersectional HCI. CHI '17. doi:10.1145/3025453.3025766 Meta-reviews 140 CHI papers on identity published between 1982 and 2016 and finds it usually analysed one facet at a time, with race and ethnicity lagging gender and class, so categorisation changes what a sample can show.
• Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C. and Wingate, D. (2023). Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis. doi:10.1017/pan.2023.2 Shows that GPT-3 conditioned on sociodemographic backstories reproduces the response distributions of the matching human subgroups, a property the authors call algorithmic fidelity. The strongest case for synthetic respondents, and the claim later benchmarked.
• Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B. and Larson, J. M. (2024). Synthetic replacements for human survey data? The perils of large language models. Political Analysis 32(4), 401-416. doi:10.1017/pan.2024.5 Prompted ChatGPT under personas to score eleven sociopolitical groups. The averages matched the American National Election Study, but the synthetic responses varied less, regression coefficients often differed significantly from the survey estimates, and the same prompt returned different distributions three months later.
• Agnew, W., Bergman, A. S., Chien, J., Diaz, M., El-Sayed, S., Pittman, J., Mohamed, S. and McKee, K. R. (2024). The Illusion of Artificial Inclusion. CHI '24. doi:10.1145/3613904.3642703 Scoping review of proposals to replace human participants with model-generated data, grouped by the argument each makes, arguing they conflict with representation, inclusion and understanding.
• Muthukrishna, M., Bell, A. V., Henrich, J., Curtin, C. M., Gedranovich, A., McInerney, J. and Thue, B. (2020). Beyond WEIRD psychology: measuring and mapping scales of cultural and psychological distance. Psychological Science 31(6), 678-701. doi:10.1177/0956797620916782 Applies the fixation index to the World Values Survey to build a distance scale from any chosen population, and finds the United States relatively homogeneous across regions, so a national sample can conceal the spread inside it.