Studying social computing: trace data, ethics and the limits of measurement

The previous nine units took the literature at face value. The tenth and last asks what kind of evidence its findings are, and the answer is narrower than the field's confidence suggests. Social computing research runs on data collected for other purposes, about people who never agreed to be studied. That buys scale and charges the two things that make a claim about people generalisable: a stated relation between the data and a population, and the consent of the people in it. The methods matured within a decade, while the ethics remained, on the profession's own account, plural and disputed (Vitak, Shilton and Ashktorab, 2016; franzke and colleagues, 2020). The two limits are one problem seen twice: the data exist for reasons unrelated to the research question.

The promise of scale met the claim that data are made, not found

Lazer and colleagues (2009) announced computational social science on the claim that digital traces allowed human behaviour to be studied at a scale and in real time that surveys could not reach. If that promise holds, sampling is answered by size. Eleven years later the same first author, with more collaborators, published the stocktake (Lazer et al., 2020), naming obstacles rather than achievements: restricted access to platform data, weak replication, and the power of a few firms to set the research agenda.

boyd and Crawford (2012) supplied the conceptual claim the unit depends on. Big data is a sociotechnical phenomenon rather than a neutral resource, to be interrogated through their six provocations about knowledge, objectivity, context, ethics and access. The aura of objectivity around a large dataset obscures the choices that produced it: what the platform recorded, what the interface invited, who was present, and therefore what the sampling frame was. A corpus of posts is evidence about posting, and no technique recovers the people who were never recorded.

Scale does not supply the sample, and access decides what can be seen

Tufekci (2014) set out the pitfalls the enthusiasm of 2009 left unstated: representativeness and validity problems routinely ignored, on the ground that platform users are not a sample of anything in particular, and an algorithmic invisibility that covers the practices, subtweeting, mock-retweeting and screenshots, that move consequential content outside the trace. A study of ten million accounts establishes what those accounts did, not what a population did.

Ruths and Pfeffer (2014) give that concern its particulars. Platform populations are not small versions of the public: Instagram appeals especially to adults aged 18 to 29, to women and to urban residents, and Twitter's public sample is drawn from about one of ten servers and covers roughly 1 per cent of tweets. Such biases are seldom corrected, and the servers differ by geography, so two studies can analyse different populations. Size cannot repair a frame chosen before the data arrived.

Bandy (2021) reviewed the audit literature following PRISMA, screening over 500 English articles and extracting 62 algorithm audit studies coded by behaviour, domain, organisation and method. Audits concentrate on black-box studies of search and recommendation systems, and she calls for attention to advertising algorithms, code auditing and under-served organisations, because access is restricted and terms are set by the firms (Lazer et al., 2020).

Public does not mean permissible

Zimmer (2010) examined a Facebook dataset assembled for research and argued that its technical availability settled nothing about its ethical status, since re-identification remained possible. Publicness, he concluded, turns on context and expectation rather than access, which is why scraped data need no automatic permission.

Hudson and Bruckman (2004) tested that default in live IRC chatrooms. Hostility was significantly higher where the study announced itself than in the silent control, and they were removed from 63.3 per cent of those chatrooms against 29 per cent of the control. When occupants could opt in, only 4 of 766 did.

Fiesler and Proferes (2018) reproduced the result on Twitter, where the public-data default is strongest. Few users in their survey knew public tweets could be used by researchers, and most thought they should not be used without consent. Their attitudes varied with who conducted the research and what it was about.

The ethics framework was built for a different study, and practice holds no standard

Metcalf and Crawford (2016) argued that data science is discontinuous with the regulatory tradition it inherits, since the distinction between research and practice cannot be exported cleanly from biomedical research. The proposed revisions to the Common Rule would have excluded data-science methods, particularly uses of public datasets, so the people whose traces are analysed can sit outside the framework built to protect research subjects. Zook and colleagues (2017) offered the practitioner's answer: ten rules that begin by acknowledging that data are people and can do harm and end with knowing when to break the rules.

Vitak, Shilton and Ashktorab (2016) surveyed 263 researchers who use online data. They found uncertainty about consent, privacy and whether the Belmont framework applies, with disagreement that did not map onto discipline, method or workplace.

Fiesler, Zimmer, Proferes, Gilbert and Jones (2024) reviewed 134 papers that used Reddit data and discussed ethics. They found ethically motivated methodological choices, such as disguising sources, that the guidelines do not require, alongside recurring concerns about privacy and vulnerable populations.

Bruckman's (2002) guidelines permitted analysis without consent only where material is officially and publicly archived, needs no password, violates no site policy and is not highly sensitive. Seventeen years later, pluralism was the official position: franzke and colleagues (2020), under Association of Internet Researchers membership approval on 6 October 2019, state the reason directly.

The issues raised by Internet research are ethical problems precisely because they evoke more than one ethically defensible response to a specific dilemma or problem. Ambiguity, uncertainty, and disagreement are inevitable.

Contested question: should internet research ethics converge on a rule, or accept pluralism?

Convergence holds that the field needs shared norms. It rests on the researchers Vitak and colleagues (2016) surveyed, on the ten rules of Zook and colleagues (2017), and on a regulatory default that leaves public datasets outside review (Metcalf and Crawford, 2016).

Pluralism holds that the answer depends on the context of the data and the people in it. Bruckman (2002), franzke and colleagues (2020) and the contextual findings of Fiesler and Proferes (2018) all sit here: a rule that fits one setting misdescribes another.

A third position cuts across both: the obligation attaches at collection, so later anonymisation cannot repair data gathered without consent (Zimmer, 2010; Metcalf and Crawford, 2016).

What would settle it is not a further statement of principle, because the sides disagree about what settling means: coherence against local defensibility. Comparative research could test whether one rule produces outcomes participants across jurisdictions accept. Evidence of practice now exists (Fiesler et al., 2024), though it does not settle which rule to adopt.

Rate the evidence on each side. Convergence rests on a survey of 263 researchers and on ten rules, both belief rather than outcome (Vitak et al., 2016; Zook et al., 2017). Pluralism rests on a professional guideline and on surveys of user expectation (franzke et al., 2020; Fiesler and Proferes, 2018). The collection-time position argues from the properties of the data (Zimmer, 2010; Metcalf and Crawford, 2016), and a review of what researchers did with Reddit data supplies evidence of practice (Fiesler et al., 2024). While the dispute is open, record consent, anonymisation and access beside the data.

Where the argument is heading

Lazer and colleagues (2020) argue that progress needs infrastructure and norms, not only methods: data access, research ethics, incentives and a better fit between the university and the field's requirements. The audit literature points the same way from below, since what can be established about a platform is bounded by what can be observed without cooperation (Bandy, 2021). The method reaches the platform, not the reader on a metred connection or a shared phone.

Bail (2024) takes the next step. Generative AI can improve online experiments, agent-based models and automated content analysis, but training-data bias can damage the research, and accuracy, reproducibility and interpretability remain unsolved. Read in this unit's terms, the prospect is double-edged: a synthetic respondent removes the representativeness problem and the consent problem by removing the human being, giving a faster supply of populations to a field whose failure mode is a population that is not a sample of anything. Bail asks whether that is a solution or the limit the field has spent fifteen years naming.

The sources this unit rests on

Each note states what the document did and found. Identifiers were resolved on 2 November 2026.

• Bail, C. A. (2024). Can generative AI improve social science? doi:10.1073/pnas.2314021121 Assesses generative AI as a tool for online experiments, agent-based models and automated content analysis, and warns that training-data bias can damage the research while accuracy and reproducibility remain unsolved. It carries the unit's forward-looking claim about simulated respondents.
• Bandy, J. (2021). A systematic literature review of algorithm audits. doi:10.1145/3449148 A PRISMA review of over 500 English articles that yielded 62 algorithm audit studies, coded by behaviour, domain, organisation and method. It finds audits concentrated on black-box studies of search and recommendation systems, with gaps around advertising and code auditing, which grounds the claim that access shapes which platform behaviours become findings.
• boyd, D. and Crawford, K. (2012). Critical questions for big data. doi:10.1080/1369118x.2012.678878 Six provocations arguing that big data is a sociotechnical phenomenon rather than a neutral resource, and that its claimed objectivity obscures the choices that produced it. It grounds the claim that data are made, not found.
• Bruckman, A. (2002). Ethical guidelines for research online. Georgia Institute of Technology. https://www.cc.gatech.edu/~asb/ethics/ Rules of thumb, dated April 2002, permitting analysis without consent only where material is officially and publicly archived, needs no password, violates no site policy and is not highly sensitive. It dates the context-dependent position the field still holds.
• Fiesler, C. and Proferes, N. (2018). "Participant" perceptions of Twitter research ethics. doi:10.1177/2056305118763366 An exploratory survey of Twitter users in which few had known that public tweets could be used by researchers, and the majority thought tweets should not be used without consent. The attitudes varied with who conducts the research and what it is about, so expectation is a function of the study.
• franzke, a. s., Bechmann, A., Zimmer, M., Ess, C. and the Association of Internet Researchers (2020). Internet research: ethical guidelines 3.0. https://aoir.org/reports/ethics3.pdf Approved by the AoIR membership on 6 October 2019. It sets out ethical pluralism, arguing that internet research evokes more than one defensible response so that the best instrument is guidelines rather than recipes, and treats data ownership and storage as live questions.
• Hudson, J. M. and Bruckman, A. S. (2004). "Go away": participant objections to being studied and the ethics of chatroom research. doi:10.1080/01972240490423030 Entered IRC chatrooms in four conditions. Hostility was significantly higher in the conditions where the study announced itself, removal rates were 63.3 per cent in the message conditions against 29 per cent in the control, and only 4 of 766 occupants opted in. It shows that technical publicness does not travel with participants' understanding.
• Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabasi, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., Jebara, T., King, G., Macy, M., Roy, D. and Van Alstyne, M. (2009). Computational social science. doi:10.1126/science.1167742 Announced a field that would use data collected at scale to reveal patterns of individual and group behaviour. It is the baseline promise the unit reads the later obstacles against.
• Lazer, D. M. J., Pentland, A., Watts, D. J., Aral, S., Athey, S., Contractor, N., Freelon, D., Gonzalez-Bailon, S., King, G., Margetts, H., Nelson, A., Salganik, M., Strohmaier, M., Vespignani, A. and Wagner, C. (2020). Computational social science: obstacles and opportunities. doi:10.1126/science.aaz8170 The decade-later stocktake: data sharing, research ethics and incentives must improve, and the field's requirements sit awkwardly with the university's organisation. It converts the promise into obstacles.
• Metcalf, J. and Crawford, K. (2016). Where are human subjects in big data research? The emerging ethics divide. doi:10.1177/2053951716650211 Argues that data science is discontinuous with biomedical research ethics, since the research and practice distinction cannot be cleanly exported, and that the proposed revisions to the Common Rule would largely exclude data-science methods, especially uses of public datasets.
• Tufekci, Z. (2014). Big questions for social media big data: representativeness, validity and other methodological pitfalls. https://arxiv.org/abs/1403.7400 (ICWSM 2014). Sets out representativeness and validity problems that are routinely ignored, and names algorithmic invisibility, field effects and the risks of importing methods from other fields. The deposited preprint carries the ICWSM citation; the AAAI page blocks bots.
• Vitak, J., Shilton, K. and Ashktorab, Z. (2016). Beyond the Belmont principles: ethical challenges, practices and beliefs in the online data research community. doi:10.1145/2818048.2820078 Qualitative and quantitative responses from a survey of 263 researchers who use online data. It finds uncertainty about consent and privacy and about whether the Belmont framework applies, with disagreement uncorrelated with discipline or method.
• Zimmer, M. (2010). "But the data is already public": on the ethics of research in Facebook. doi:10.1007/s10676-010-9227-5 Examines a Facebook dataset assembled for research and argues that technical availability does not settle ethical status, because re-identification remains possible. It works through consent, expectations of privacy, anonymisation before release and the limits of board expertise.
• Zook, M., Barocas, S., boyd, D., Crawford, K., Keller, E., Gangadharan, S. P., Goodman, A., Hollander, R., Koenig, B., Metcalf, J., Narayanan, A., Nelson, A. and Pasquale, F. (2017). Ten simple rules for responsible big data research. doi:10.1371/journal.pcbi.1005399 Ten rules from a National Science Foundation funded council, covering harm, privacy as more than a binary value, re-identification, data sharing and auditability. The authors state that the variety of data and methods belies a one-size-fits-all checklist.
• Ruths, D. and Pfeffer, J. (2014). Social media for large studies of behaviour. doi:10.1126/science.346.6213.1063 Argues that large social media studies need higher methodological standards, showing biased platform populations and that Twitter's public sample covers about 1 per cent of tweets. It gives the sampling failure its particulars. (The published title uses the American spelling.)
• Fiesler, C., Zimmer, M., Proferes, N., Gilbert, S. and Jones, N. (2024). Remember the human: a systematic review of ethical considerations in Reddit research. doi:10.1145/3633070 Reviewed 134 papers using Reddit data, finding ethically motivated methodological choices, such as disguising sources, that guidelines do not require.