Surveys and questionnaires: writing an item that measures one thing

A questionnaire is easy to mistake for a list of questions. It is better read as a measuring instrument: fixed items, uniform administration, one composite score. This seventh of the ten units asks what people say, and argues that the instrument has a documented error profile. Item wording, scale format and response style are design decisions whose consequences have been measured. Tourangeau, Rips and Rasinski (2000) supply the mechanism that makes the claim more than a preference.

A survey item is not a question with a right answer. It is an instrument reading, and the instrument carries its error with it.

A survey answer is produced, not retrieved

Tourangeau, Rips and Rasinski (2000) describe survey answering as four stages. The respondent comprehends the question, retrieves relevant information, judges what has been retrieved, and reports an answer that may be edited for acceptability. Each stage is a place where the answer can leave the fact it is meant to describe. Their review treats deliberate misreporting as only one distortion among several. The design consequence is exact: asking one thing at a time repairs a stage where two things cannot be retrieved together. A study Schwarz reports shows the size of the effect. Asked what matters most in preparing children for life, 61.5 per cent chose to think for themselves when the option appeared in a list. Only 4.6 per cent gave an answer in that category when the question was left open.

Krosnick (1991) named the second failure. When answering optimally would cost more effort than the respondent will spend, the respondent satisfices, producing an acceptable answer rather than an accurate one. Strategies include choosing the first response that seems reasonable, agreeing with whatever the question asserts, endorsing the status quo, failing to differentiate among objects rated, and answering at random. Satisficing is more likely when the question is hard, which places much of the error in the instrument rather than in the respondent's carelessness.

Krosnick (1999) later found the field's conventions moving under the weight of accumulated experiments: the verbal labels on rating-scale points shift the answers, and respondents read questions through the norms of ordinary conversation, so a question that breaks those norms introduces its own error.

Wording, format and context are measured decisions

Schwarz (1999) reviewed how question wording, format and context shape the answers people give. He located the effect in the processes that generate a response rather than in deliberate distortion. His clearest case, run by Schwarz and colleagues (1985), is a question about television. German respondents were asked how long they watched on a typical day. They were offered one of two six-band scales. One topped out at two and a half hours, the other at four and a half hours. Of those given the low scale, 16.2 per cent reported watching more than two and a half hours a day. Of those given the high scale, 37.5 per cent did. The bands did not measure viewing so much as tell the respondent what counted as a lot of it.

• Wording of an item. The same construct asked two ways yields systematically different answers (Schwarz, 1999).
• Rating-scale labels. The verbal labels attached to scale points shift the answers (Krosnick, 1999).
• Cognitive demand. Harder questions raise satisficing, so the error tracks difficulty (Krosnick, 1991).
• Response set. Acquiescence and social desirability move answers as a style (Nederhof, 1985).
• Reading level. Harder text raises satisficing, so an item above the respondent's reading level collects an acceptable answer rather than an accurate one (Krosnick, 1991).

Response style is a measured property of the instrument

Some error belongs to the person rather than the item, but it is still measurable and still a design choice. Nederhof (1985) separated social desirability into two factors, a self-deceptive tendency and an other-deceptive one, and sorted the remedies into detection and prevention. Scales and item-desirability ratings detect the bias; forced-choice items, the randomised response technique, the bogus pipeline, self-administration and the choice of interviewer attempt to prevent it. No single method coped with both forms under all conditions, and a combination was the best available choice.

Careless responding is the second style, and common enough to change a result. Meade and Craig (2012) compared five ways of finding it, including instructed-response items, consistency indices, multivariate outlier analysis, response time and self-reported diligence. They found two distinct patterns, random and nonrandom, needing different indices to detect. About ten to twelve per cent of undergraduates completing a lengthy survey for course credit were careless responders. The authors recommend collecting identified rather than anonymous responses and placing instructed-response items before collection. A third style appeared after 2023. Zhang, Xu and Alvero (2025) found that 34 per cent of participants recruited online reported using a large language model on open-ended questions. Model-written answers were more homogeneous and more positive than human answers from three pre-ChatGPT studies, especially about social groups. The homogenisation hides the variation the item exists to measure. A questionnaire whose careless cases are neither prevented nor removed has published a measure of unknown quality.

The System Usability Scale is one instrument, not a list

Brooke (1996) introduced the System Usability Scale as a ten-item instrument with a five-point response format. It alternates positive and negative statements and produces a single composite score from 0 to 100. Each odd item contributes its scale position minus one and each even item five minus the scale position; the ten contributions are summed and multiplied by 2.5.

Bangor, Kortum and Miller (2008) analysed nearly ten years of System Usability Scale data gathered across many products. They found the scale robust and versatile. They added an adjective rating that corresponds to a given score, so that a raw number becomes a judgement. Dillman, Smyth and Christian (2014) argue that a survey must be tailored to its population. The sequence of contact, invitation and reminder matters as much as the questions, which places administration inside the measurement.

Contested question: is reliability one number or a model?

Cronbach (1951) derived coefficient alpha as the mean of all the split-half coefficients obtained by dividing a test, presenting it as an estimate of the correlation between two random samples of items. He also warned that a test divisible into distinct subtests should be divided before the formula is used: alpha then estimates the concentration of the first factor rather than the structure of the whole.

Sijtsma (2009) attacks the statistic directly: alpha is unrelated to the internal structure of the test, so using it as a measure of internal consistency is a category error. Because alpha always takes a value, that value cannot be the reliability of the scores under the usual assumptions. A single coefficient conveys little about the accuracy of an individual's score, and he recommends modelling the items instead.

McNeish (2018) accepts the diagnosis and rejects the conclusion: the criticisms are aimed at alpha, not at reliability, and violating alpha's assumptions makes a measure look less reliable than it is. The response is to report an estimator with weaker assumptions, such as omega total, rather than abandon the concept. Sijtsma would replace the coefficient with an item-level model. McNeish would keep reliability and replace only the estimator. What would settle it is a comparison against an external criterion. A study would report alpha, omega and a model-based reliability for scales of known and unknown dimensionality, showing which figure tracks the criterion and how closely.

A second dispute sits inside the treatment of bad data. Meade and Craig (2012) and Curran (2016) treat careless responding as a screening problem, detected with indices and removed before analysis. Krosnick's (1991) account locates the same behaviour in question difficulty. That makes it a design problem, to be reduced by writing easier items and varying the format. A field experiment manipulating item difficulty and measurement context against a verified criterion would show how much design can remove and how much screening must catch.

The evidence here is mostly conceptual, and the two sides answer different questions. Cronbach offers the derivation and its own caution about subscales, Sijtsma argues from the algebra of what alpha estimates, and McNeish from the consequences of violated assumptions. No study in this unit compares the estimators against an external criterion. While the dispute is open, report more than one figure, including an estimator with weaker assumptions such as omega, and state the dimensionality assumed. The strongest objection is McNeish's, that the criticism targets one estimator, so an instrument can still be measured well.

Where the argument is heading

The instrument and its administration are being treated as one object. Dillman, Smyth and Christian (2014) argue a survey must be tailored to its population and delivered through a planned sequence of contacts; Curran (2016) and Meade and Craig (2012) make the detection of careless responding routine rather than exceptional; Krosnick (1999) recorded conventions being overturned as experiments accumulated; and Cronbach's alpha is being displaced by the estimators Sijtsma (2009) and McNeish (2018) argue about. A study that states its wording, its scale, its response-set defences and the fate of its careless cases will be assessable. One that reports a single coefficient will not.

The sources this unit rests on

• Tourangeau, R., Rips, L. J. and Rasinski, K. (2000). The Psychology of Survey Response. Cambridge University Press. doi:10.1017/cbo9780511819322 Sets out survey answering as four stages, comprehension, retrieval, judgement and reporting, and reviews the sources of response error. It treats deliberate misreporting as one distortion among several and supplies the mechanism behind the unit's item-writing rules.
• Schwarz, N., Hippler, H.-J., Deutsch, B. and Strack, F. (1985). Public Opinion Quarterly 49(3), 388-395. doi:10.1086/268936 Ran the range experiment the review reports: German respondents given a scale topping out at two and a half hours reported less television viewing than those given one topping out at four and a half hours. The primary behind the range effect.
• Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology. doi:10.1002/acp.2350050305 Proposes satisficing: a respondent who will not spend the effort an optimal answer needs gives an acceptable one, by agreeing, endorsing the status quo or answering at random. It locates much of the error in question difficulty rather than carelessness.
• Krosnick, J. A. (1999). SURVEY RESEARCH. Annual Review of Psychology. doi:10.1146/annurev.psych.50.1.537 Reviews survey methodology and records conventions being overturned as experiments accumulated, among them the effect of verbal labels on rating-scale points and the role of conversational norms, so design choices carry measured consequences.
• Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist. doi:10.1037/0003-066x.54.2.93 Reviews how wording, format and context shape answers through the processes that generate a response rather than deliberate distortion. The same construct asked two ways yields systematically different answers.
• Zhang, S., Xu, J. and Alvero, A. J. (2025). Generative AI meets open-ended survey responses: research participant use of AI and homogenization. Sociological Methods and Research 54(3), 1197-1242. doi:10.1177/00491241251327130 Surveyed research participants recruited online and found 34 per cent reported using a large language model on open-ended questions. Model answers were more homogeneous and more positive than human answers from three pre-ChatGPT studies, especially about social groups.
• Cronbach, L. J. (1951). Coefficient Alpha and the Internal Structure of Tests. Psychometrika. doi:10.1007/bf02310555 Derives coefficient alpha as the mean of all the split-half coefficients obtained by dividing a test, an estimate of the correlation between two random samples of items. The origin of the reliability coefficient questionnaires are routinely reported with.
• Sijtsma, K. (2009). On the Use, the Misuse, and the Very Limited Usefulness of Cronbach's Alpha. Psychometrika. doi:10.1007/s11336-008-9101-0 Argues that alpha is unrelated to the internal structure of a test and cannot equal the reliability of the scores under the usual assumptions, so its use as an internal-consistency measure is a category error. Model the items directly instead.
• McNeish, D. (2018). Thanks coefficient alpha, we will take it from here. Psychological Methods. doi:10.1037/met0000144 Reviews the case against alpha and argues the criticisms target the estimator rather than reliability, so the field should report alternatives such as omega total. It turns the dispute into a choice of estimator rather than a rejection of reliability.
• Nederhof, A. J. (1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology. doi:10.1002/ejsp.2420150303 Reviews methods for coping with social desirability bias, separating self-deception from other-deception and detection from prevention. It found no single method that excelled under all conditions and recommended a combination of prevention and detection.
• Meade, A. W. and Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods. doi:10.1037/a0028085 Compared five methods for detecting careless survey responses and found two patterns, random and nonrandom, needing different indices. About ten to twelve per cent of undergraduates completing a lengthy survey were identified as careless responders.
• Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology. doi:10.1016/j.jesp.2015.07.006 Reviews and integrates techniques for detecting carelessly invalid responses, including attention checks, long-string analysis, response time and multivariate approaches. It makes data quality an assessed property rather than an assumption.
• Brooke, J. (1996). SUS: a quick and dirty usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester and I. L. McClelland (eds.), Usability Evaluation in Industry, 189-194. London: Taylor and Francis. doi:10.1201/9781498710411-35 Introduces the System Usability Scale as a ten-item questionnaire with a five-point format and a single composite score from 0 to 100. The origin of the field's most widely used scale, and the reason a scale is treated as one instrument rather than a list of questions.
• Bangor, A., Kortum, P. T. and Miller, J. T. (2008). An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction. doi:10.1080/10447310802205776 Analysed nearly ten years of System Usability Scale data across many products, found the scale robust and versatile, and added an adjective rating for a given score, turning a raw number into a normed judgement.
• Dillman, D. A., Smyth, J. D. and Christian, L. M. (2014). Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. 4th edition. Hoboken, NJ: Wiley. https://www.wiley.com/en-us/Internet%2C+Phone%2C+Mail%2C+and+Mixed-Mode+Surveys%3A+The+Tailored+Design+Method%2C+4th+Edition-p-9781118456149 The standard practical text on designing and administering surveys across modes, arguing a survey must be tailored to its population and that the contact and reminder sequence matters as much as the questions. Administration sits inside the measurement.