The platform as laboratory: A/B tests, telemetry and censoring

A platform assigns variants to a million users at random and returns a number. The assignment is clean and the sample is enormous, and the number can still be wrong. This ninth of ten units follows that distrust onto a platform, and argues that the online controlled experiment is the laboratory experiment at a scale where the manipulation has stopped being the weak point. The metric is a proxy chosen by the organisation that will act on it, the units influence one another, and the observations that end early are censored rather than failed.

The laboratory did not get better, it got bigger

Kohavi and colleagues describe what scale looks like. At Microsoft's Bing more than 200 controlled experiments ran concurrently on any given day, each with millions of users (Kohavi et al., 2013). Users are randomised to variants, an outcome is measured, and the difference is attributed to the variant, as in a laboratory study with two conditions. What changed is that the outcome is recorded by the same system that serves the treatment, so the measurement is a decision of the platform.

The metric, not the manipulation, is where the check has to go

Kohavi, Tang and Xu set out the machinery that a platform needs before its numbers can be trusted. The list is dominated by measurement rather than randomisation: key metrics and an overall evaluation criterion, trustworthiness checks for violated assumptions, and pitfalls such as carry-over effects. One such pitfall is what their field calls Twyman's law, that an interesting figure is usually wrong (Kohavi, Tang and Xu, 2020). The check belongs on the metric because the effects are small: many ideas move key metrics by around one per cent, and they are not well estimated beforehand (Kohavi et al., 2013). A team defines engagement as videos watched per session, ships automatic playback, and sees videos per session rise along with sessions ending within seconds. The metric never records the user on a metered connection, for whom the extra playback costs money. The metric moved with the proxy and against the construct.

Sensitivity is engineered, and that lowers the threshold

The platform engineers the sensitivity that its traffic gives it. Deng and colleagues introduced CUPED, which uses pre-experiment data as a covariate and on Bing reduced variance by about 50 per cent, the same power with half the users or half the duration (Deng et al., 2013). An artefact smaller than any practical gain becomes visible, and the organisation acts on what it once called noise. Kohavi and colleagues alert experimenters to statistical interactions between concurrent experiments (Kohavi et al., 2013). Google's answer to that overlap is infrastructure rather than analysis: Tang and colleagues allocate many concurrent experiments to layers so that they run on the same users at once, which does not make those users independent (Tang et al., 2010).

Independence is the assumption the platform breaks

Eckles, Karrer and Ugander state the assumption plainly. Familiar designs assume that one unit's outcome does not depend on others' treatment assignments, and when interference exists, causal estimates are biased; graph cluster randomisation and neighbour-treatment analyses reduce that bias in simulations (Eckles et al., 2016). Saveski and colleagues make the assumption measurable, running a completely randomised and a cluster-based randomised experiment at the same time and comparing the estimates. Deployed on LinkedIn, two experiments showed network effects and bias in standard A/B testing (Saveski et al., 2017).

Censoring: what a finished session cannot tell you

Kaplan and Meier derived the product-limit estimator for exactly this case. It estimates how long something survives when the event of interest is missed for some units. They state that the lifetime is assumed independent of the loss time, and that in practice this assumption deserves careful scrutiny (Kaplan and Meier, 1958). A retention analysis observes users for a fixed window, and users still active at the end are censored rather than failed. A dashboard that divides abandonments by everyone who started has assumed the window is long enough and that the censored users resemble the observed ones. A user on an older device or a poor connection may be censored for reasons the metric cannot see, as may a user whose interface is delivered through a screen reader. Singer and Willett made the discrete-time version usable for behavioural researchers, fitting a hazard model by ordinary logistic regression on data for 3,941 special educators, to answer both whether and when an event occurs (Singer and Willett, 1993). Treating a censored observation as a failure biases a survival estimate downwards, and heavy users are usually observed for longer, so the dependence Kaplan and Meier warn about is not hypothetical in interaction data.

The metric is fixed before the variant is built

Register the overall evaluation criterion and the expected metrics before the variant is built, and use the trustworthiness checks rather than one headline figure (Kohavi, Tang and Xu, 2020). Where the dashboard is watched while the test runs, use inference that stays valid under continuous monitoring (Johari et al., 2017). Test for interference rather than assuming it away (Saveski et al., 2017), and report abandonment or retention with a survival estimator and an explicit statement about censoring (Kaplan and Meier, 1958; Singer and Willett, 1993).

Contested question: is the platform a laboratory, or an instrument it built for itself?

One position is that the machinery makes platform experiments trustworthy evidence. Kohavi, Tang and Xu argue that randomised assignment with an overall evaluation criterion, trustworthiness checks and alerts yields decisions that beat the highest paid person's opinion (Kohavi, Tang and Xu, 2020). On this account the platform is a laboratory that happens to be large.

A second position accepts the experiment's inferential superiority and doubts what it can claim. Milli, Belli and Hardt assessed observational estimates from user self-selection against randomised ground truth in four large-scale within-study comparisons on Twitter. Every estimator performed poorly at recovering the randomised estimate, and all but one took the opposite sign (Milli, Belli and Hardt, 2022). Self-selection replaces the experiment as platforms give users more agency, and it fails, leaving the platform to define both the treatment and the only credible estimate of its effect.

A related doubt concerns the studies, not the platform's own tests. Boegershausen and colleagues reviewed 133 published studies that used the A/B tools of Facebook or Google Ads. Those tools do not assign ads to consumers at random, because the platform's targeting algorithms decide who sees which creative. Most of the 133 were still presented as randomised experiments and most findings described as causal, a practice whose prevalence had not fallen (Boegershausen et al., 2025). That is the platform study a student is likeliest to read.

A third position points outside the platform. Grosz and colleagues argue that natural experiments, including instrumental-variable and regression-discontinuity designs, sometimes establish random or as-if random assignment when randomisation is unethical or unfeasible. Psychology has under-used them, though such studies are hard to find and rest on assumptions that are difficult to validate empirically (Grosz et al., 2024).

What would settle the disagreement is a measurement of the disagreement: a within-study comparison of the kind Milli and colleagues ran, extended from treatment assignment to metric definition, and a preregistered interference test of the kind Saveski and colleagues deployed (Milli, Belli and Hardt, 2022; Saveski et al., 2017).

The evidence divides by kind. The first position rests on industrial practice and a practitioner monograph. The second rests on four within-study comparisons against randomised ground truth, and none recovered the randomised estimate. The third is a methodological review rather than a measurement.

The strongest objection is that scepticism about the platform leaves no alternative of comparable strength: the randomised estimate stays the best available, the observational substitutes fail, and natural experiments are rare.

What to do while the dispute is open

While the dispute is open, name who chose the metric and record that choice first.

Where the argument is heading

The direction of travel is toward the platform as an instrument for generalisable knowledge, not a machine for per-feature comparisons. Bakshy, Eckles and Bernstein describe PlanOut, a language that separates experimental design from application code (Bakshy et al., 2014). Always valid inference was shipped by a commercial testing vendor, and interference detection was deployed at LinkedIn, so assumptions once argued about in papers are now measured in production. Fabijan and colleagues add the institutional constraint, a company moving through technical, organisational and business phases towards continuous experimentation (Fabijan et al., 2017), so the quality of platform evidence will be decided by how organisations define and defend their metrics.

The sources this unit rests on

• Boegershausen, J., Cornil, Y., Yi, S. and Hardisty, D. J. (2025). On the persistent mischaracterization of Google and Facebook A/B tests: How to conduct and report online platform studies. International Journal of Research in Marketing. doi:10.1016/j.ijresmar.2024.12.004 They reviewed 133 published platform studies and found most presented as randomised experiments and most findings described as causal, though the platforms' targeting algorithms confound the creative; the prevalence had not fallen.
• Bakshy, E., Eckles, D. and Bernstein, M. S. (2014). Designing and deploying online field experiments. Proceedings of the 23rd international conference on World Wide Web. doi:10.1145/2566486.2567967 They introduce PlanOut, a language separating experimental design from application code, and a namespaced management system for parallel experiments, with two field experiments on Facebook. It shows the platform answering questions beyond one feature comparison.
• Deng, A., Xu, Y., Kohavi, R., et al. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. Proceedings of the sixth ACM international conference on Web search and data mining. doi:10.1145/2433396.2433413 They propose CUPED, using pre-experiment data as a covariate, and report variance reductions of about 50 per cent on Bing, the same power with half the users or half the duration. It shows a platform engineering its sensitivity.
• Eckles, D., Karrer, B. and Ugander, J. (2016). Design and Analysis of Experiments in Networks: Reducing Bias from Interference. Journal of Causal Inference. doi:10.1515/jci-2015-0021 They show that standard designs assume no interference, that estimates are biased when it exists, and that graph cluster randomisation and neighbour-treatment analyses reduce bias under broad interference. It states the assumption the platform test most often violates.
• Fabijan, A., Dmitriev, P., Olsson, H. H., et al. (2017). The Evolution of Continuous Experimentation in Software Product Development. ACM/IEEE International Conference on Software Engineering: Companion Proceedings. doi:10.1109/ICSE.2017.76 From a case study of a large software-intensive company they present an Experimentation Evolution Model with technical, organisational and business phases. It supports the claim that the binding constraint on better platform evidence is institutional.
• Grosz, M. P., Ayaita, A., Arslan, R. C., et al. (2024). Natural Experiments: Missed Opportunities for Causal Inference in Psychology. Advances in Methods and Practices in Psychological Science. doi:10.1177/25152459231218610 They describe standard natural experiments, instrumental-variable designs and regression-discontinuity designs, arguing that psychology has under-used them while noting they are hard to find and rest on assumptions difficult to validate. It is the third position in the contested question.
• Johari, R., Koomen, P., Pekelis, L., et al. (2017). Peeking at A/B Tests. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. doi:10.1145/3097983.3097992 They address the unreliability of traditional inference when users monitor a test continuously and provide always valid p values and confidence intervals robust to it; the method was deployed by Optimizely. It names the sampling-plan dependence where it bites hardest.
• Kaplan, E. L. and Meier, P. (1958). Nonparametric Estimation from Incomplete Observations. Journal of the American Statistical Association. doi:10.1080/01621459.1958.10501452 They derive the product-limit estimator of a survival function when the event is not observed for some units because of a loss, and state that the lifetime is assumed independent of the loss time and that in practice this assumption deserves careful scrutiny. It is the source of the unit's censoring argument and of its central caveat.
• Kohavi, R., Deng, A., Frasca, B., et al. (2013). Online controlled experiments at large scale. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. doi:10.1145/2487575.2488217 They report more than 200 concurrent experiments a day at Bing, revenue where one per cent exceeds ten million dollars a year, a high occurrence of false positives, and alerts for interactions between experiments. It is the unit's evidence for what platform scale is.
• Kohavi, R., Tang, D. and Xu, Y. N. (2020). Trustworthy Online Controlled Experiments. Cambridge University Press. doi:10.1017/9781108653985 Contains key metrics and an overall evaluation criterion, trustworthiness checks for violated assumptions, and pitfalls such as carry-over effects and Twyman's law, in companies each running more than 20,000 experiments a year. It is the institutional position the contested section examines first.
• Milli, S., Belli, L. and Hardt, M. (2022). Causal Inference Struggles with Agency on Online Platforms. ACM Conference on Fairness, Accountability, and Transparency. doi:10.1145/3531146.3533103 In four within-study comparisons on Twitter they assessed observational estimates from user self-selection against randomised ground truth; every estimator performed poorly and all but one took the opposite sign. It is the second position in the contested question.
• Saveski, M., Pouget-Abadie, J., Saint-Jacques, G., et al. (2017). Detecting Network Effects. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. doi:10.1145/3097983.3098192 They test the stable unit treatment value assumption by running a completely randomised and a cluster-based randomised experiment simultaneously and comparing estimates, and deployed the design on LinkedIn, where two experiments showed network effects and bias in standard A/B testing. It turns interference from an assumption into a measurement.
• Singer, J. D. and Willett, J. B. (1993). It's about Time: Using Discrete-Time Survival Analysis to Study Duration and the Timing of Events. Journal of Educational Statistics. doi:10.2307/1165085 They derive maximum likelihood estimators for a discrete-time hazard model, fit it with standard logistic regression software, and demonstrate it on longitudinal data for 3,941 special educators to answer whether and when an event occurs. It makes censoring-aware analysis usable for HCI questions about abandonment.
• Tang, D., Agarwal, A., O'Brien, D., et al. (2010). Overlapping experiment infrastructure. Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. doi:10.1145/1835804.1835810 They describe Google's overlapping experiment infrastructure, which runs many concurrent experiments on the same users by allocating them to layers, and the tools and processes the scale requires.