Source inconsistency: course marks these options correct but attaches unrelated EASE feedback. Correctness marks follow course answer key; feedback is retained as a source anomaly.
Study notes · CADIO / NYU School of Global Public Health
Module 4: “Some Conceptual and Technical Aspects of the Factorial Experiment” Authors: Linda M. Collins and Kate Guastaferro (School of Global Public Health, New York University) Format: 5 lessons · 5 videos (72 min) · 5 slide decks (PDF) · 5 knowledge checks · 5-item quiz Set reading: Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 4) Access: free via d3center / CADIO share links, or Coursera
The module's argument in one paragraph.
Module 4 supplies the statistical machinery that makes the factorial experiment, introduced in Module 3 as the workhorse of the optimization phase, actually produce interpretable answers. It runs a single running example (a 2^3 experiment with motivational interviewing, peer mentoring and text message support, outcome = adherence) through every technical question. Lesson 1 sets up the regression model for a factorial experiment and the two coding schemes, dummy (0, 1) and effect (−1, 1); both give the same omnibus F, but they define main effects and interactions differently, and this course uses effect coding exclusively because it makes each regression weight interpretable as half the corresponding effect as the course defines it. Lesson 2 teaches how to read those effects off predicted means, plot the interaction, look for parallel lines, and distinguishes synergistic from antagonistic interactions, up to the three-way case, while disposing of two worries: in an effect-coded balanced factorial, main effects and interactions are uncorrelated, and power for any regression coefficient is identical in a balanced 2^k, so interactions are not inherently underpowered. Lesson 3 is powering 101: Type I and Type II error rates, central vs noncentral distributions, the three levers (N, effect size, alpha), Cohen's d and standardized-regression-coefficient benchmarks, and the practical recipe of powering to the smallest effect size you care about. Lesson 4 reframes the choice of alpha as a matter of research philosophy: the conclusion-priority perspective (peer review as “science court”, alpha ≤.05, power ≥.7,.8) governs the evaluation phase, while the decision-priority perspective, tied to the resource management principle, governs the optimization phase, where raising alpha to buy power with a fixed N is a legitimate strategic trade-off. Lesson 5 handles nesting: cluster structures create contamination risk and, under between-cluster randomization, inflate the needed N by the design effect D = 1 + (n − 1)·ICCX·ICCY, with two practical questions, enough power, and enough clusters per condition.
Lesson 1: The Regression Model and Coding
The lesson sets up the statistical engine of the factorial experiment: the regression model used to analyze it, and the two ways of numerically coding factor levels that go into that model.
Why the factorial experiment: “understanding the factorial design is necessary for understanding the fractional factorial design, the SMART, and the micro-randomized trial.”Narration, Lesson 1
Factorial experiments estimate two types of effects.
Main effect of Factor A: “The main effect of Factor A is defined as the effect of Factor A averaged across all levels of all other factors.”Narration, Lesson 1
Interaction between Factor A and Factor B (each with two levels): “1/2 the effect of Factor A at level 1 of Factor B minus the effect of Factor A at level 2 of Factor B averaged across all levels of all other factors.”Narration, Lesson 1
The conceptual test for interaction: “If the effect of Factor A is the same at each level of Factor B, then there is no interaction. If the effect of Factor A varies depending on what level of Factor B you're looking at, then Factor A and Factor B interact.”Narration, Lesson 1
ANOVA: “Data from a factorial experiment are typically analyzed via analysis of variance, usually referred to by the acronym ANOVA.” The course's perspective is to perform ANOVA via regression.Narration, Lesson 1
The running example: a 2^3 experiment (MI, PEER, TEXT)
The three candidate components used throughout the module are motivational interviewing (MI), peer mentoring (PEER) and text message support (TEXT); for illustration they are treated as the only candidate components. The design table below gives all eight experimental conditions, each with an observed mean on the outcome. The course example uses n = 40 per condition.
Condition
MI
PEER
TEXT
Outcome mean
1
No
No
No
Ȳ₁
2
No
No
Yes
Ȳ₂
3
No
Yes
No
Ȳ₃
4
No
Yes
Yes
Ȳ₄
5
Yes
No
No
Ȳ₅
6
Yes
No
Yes
Ȳ₆
7
Yes
Yes
No
Ȳ₇
8
Yes
Yes
Yes
Ȳ₈
From the slides: in this example the outcome is a measure of adherence to the treatment regimen (“adhere”), and a larger value is better.
Effects in a full ANOVA model for the 2^3 experiment: three main effects (MI, PEER, TEXT); three 2-way interactions (MI×PEER, MI×TEXT, PEER×TEXT); one 3-way interaction (MI×PEER×TEXT). All can be estimated via multiple regression.
“Whenever you perform a regression, you are trying to predict an outcome variable y. This prediction is called Y hat. Y hat is always a linear combination of the predictor variables and regression weights.” There is a Ŷ for each experimental condition; e.g. Ŷ₃ is the predicted Y when MI = No, PEER = Yes and TEXT = No. The βs are used “in a general sense, not to indicate standardized regression weights,” and correspond to estimates of the intercept, main effects and interactions. The X's are the numeric codes for the factor levels, which is where coding enters.
Dummy coding vs effect coding
Dummy coding: “With dummy coding, you create x variables by assigning codes of 0 or 1.” For the main effects of the 2^3, MI has a 0 wherever MI = No and a 1 wherever MI = Yes; the same approach codes PEER and TEXT.Narration + slides, Lesson 1
Effect coding: “with effect coding, there's a -1 every place MI is set to No and a +1 every place MI is set to Yes. In other words, for the main effects, the only difference between effect coding and dummy coding is that where there is a 0 in dummy coding, there's a -1 in effect coding.”Narration, Lesson 1
Interaction codes are obtained by multiplying the main-effect codes. For example, the MI×PEER interaction codes come from multiplying the MI codes by the PEER codes: in condition 1, (−1)(−1) = +1; in condition 3, (−1)(1) = −1. For an individual in condition 3 (MI = No, PEER = Yes, TEXT = No), the entries are −1, 1, −1 respectively. (Statistical software normally does this coding for you; if doing it by hand, include the intercept code, which is 1 for all participants, why the intercept is sometimes called the “constant.”)
Condition
MI (X₁)
PEER (X₂)
TEXT (X₃)
MI×PEER (X₄)
MI×TEXT (X₅)
PEER×TEXT (X₆)
MI×PEER×TEXT (X₇)
1
−1
−1
−1
+1
+1
+1
−1
2
−1
−1
+1
+1
−1
−1
+1
3
−1
+1
−1
−1
+1
−1
+1
4
−1
+1
+1
−1
−1
+1
−1
5
+1
−1
−1
−1
−1
+1
+1
6
+1
−1
+1
−1
+1
−1
−1
7
+1
+1
−1
+1
−1
−1
−1
8
+1
+1
+1
+1
+1
+1
+1
Same omnibus F, different effects: “Dummy coding and effect coding will produce exactly the same omnibus, that is, overall, F, but the main effect and interaction effect estimates, and the hypothesis tests associated with those estimates, are usually different.”Narration, Lesson 1
Course convention: “In this course, we use only effect coding.” In your own data analysis, be sure you know which coding is being used. See the Kugler, Dziak and Trail chapter in Collins & Kugler (2018).Narration + slides, Lesson 1
Interpreting the regression weights
β₁ and the main effect: “Through tedious but not difficult algebra (which I'm not going to go through here) you can show for a 2^k experiment that when effect coding is used, beta sub one can be interpreted as 1/2 of the main effect of MI according to the definition of the main effect we've been using.” The analogous interpretation applies to β₂ and β₃.Narration, Lesson 1
β₄ and the interaction: “you can show that for a 2^k experiment, when effect coding is used, beta sub four corresponds to 1/2 of the interaction, according to the definition of interaction we've been using.” Caveat: for higher-order interactions (three or more factors), which interpretation holds depends on the definition of interaction; this course always defines the interaction in terms of the regression weight (see Collins 2018, sections 3.14 and 4.7).Narration, Lesson 1
Optimization and the “expected outcome”: “Optimization of a multicomponent intervention is the process of identifying an intervention that provides the best expected outcome obtainable within key constraints imposed by the need for affordability, scalability, and/or efficiency.” The “expected outcome” in this definition refers to the Ŷs, “The y hats are estimates of the expected outcome associated with each experimental condition.”Narration, Lesson 1
Knowledge check
MULTIPLE RESPONSE: Which of the following statements about dummy coding and effect coding are correct? Select all that apply.
☑ Dummy coding and effect coding are both ways of numerically representing factor levels in a regression model.
☑ Dummy coding and effect coding produce the same value for the omnibus (overall) F statistic.
☐ Dummy coding and effect coding always produce main effect and interaction effect estimates that can be interpreted in the same way.
☐ Whether the data analyst uses dummy coding or effect coding, the same main effects and interaction effects will always be statistically significant.
Feedback when incorrect: Dummy coding and effect coding always produce main effect and interaction effect estimates that can be interpreted in the same way.This is incorrect. Dummy coding and effect coding usually produce different effect estimates.Whether the data analyst uses dummy coding or effect coding, the same main effects and interaction effects will always be statistically significant.This is incorrect. Different main effects and interaction effects may be statistically significant depending on whether dummy coding or effect coding is used.
Lesson 2: Interpretation of Interaction Effects
The lesson teaches how to read main effects and interactions off the predicted means, with the running 2^3 example (outcome = adhere, larger is better).
Main effect of MI in the 2^3: “The main effect of MI is the mean of conditions 5 through 8, which are shaded darker in this table, minus the mean of conditions 1 through 4, shaded lighter.”Narration, Lesson 2
What effect coding means (and does not): “when effect coding is used, the main effect of MI is not the effect of MI with all the other factors set to 'no'. If you are accustomed to dummy coding, you might have to keep reminding yourself of this.” To estimate the effect of MI with everything else set to No, you would need a different experiment, only conditions 1 and 5.Narration, Lesson 2
Scaling: “you can show that beta sub one equals the main effect of MI divided by 2. That 2 in the denominator is simply a scaling constant that does not affect hypothesis testing.” The same holds for β₄ = (MI×PEER interaction)/2.Narration, Lesson 2
Interaction, conceptually: “if the effect of Factor A is the same at each level of Factor B, then Factor A and Factor B do not interact. If the effect of Factor A is different at each level of Factor B, in other words, if you need to know which level of Factor B you're talking about before you can understand the effect of Factor A, then Factor A and Factor B do interact.”Narration, Lesson 2
Two worries about interactions and the responses
Worry 1: interactions contaminate main effects / make them uninterpretable. Response: interactions should always be considered when interpreting main effects, but “in an effect-coded ANOVA performed on data from a balanced factorial experiment, main effects and interactions are uncorrelated,” which helps separate them. And do not respond by designing interactions out of reach (individual experiments, comparative treatment experiments): “interactions do not go away just because an investigator chooses not to estimate them” (Collins, Dziak & Li, 2009).
Worry 2: there is rarely sufficient power to detect interactions. Response:
Equal power in a balanced 2^k: “Within a balanced 2^k factorial experiment, statistical power for detection of Beta sub j is identical irrespective of whether Beta sub j corresponds to a main effect or an interaction of any order.” This is shown in Cohen (1988) and explained in Chapter 4 of Collins (2018). “This is another reason why the 2^k is such an efficient experimental design.”Narration, Lesson 2
Interactions tell the rest of the story of Ŷ: “You can ask yourself, does a linear combination of the main effects only, as depicted here, tell you what you need to know about the outcome variable? If the answer is yes, a linear combination of the main effects is sufficient, then you won't need to consider the interactions in your decision-making about which components should be included in the optimized intervention. Or, do you need to include one or more of the interaction terms to complete the story?”Narration, Lesson 2
How to interpret interactions: plot them
The practice: “It's always a good idea to plot an interaction. You need to obtain predicted means, that is, means based on the Y-hats, in order to make these plots, and you will often need these to be marginal means, that is means collapsed across one or more factors.”Narration, Lesson 2
The plots below are predicted marginal means for the running example, outcome “adhere” on the y-axis; when PEER is on the x-axis, the two MI = no means are joined by a dashed line and the two MI = yes means by a solid line. “In general, when the lines of a plot like this are parallel, it indicates there is no interaction.”
Main effect of MI, no main effect of PEER, no interaction. Parallel horizontal lines: the effect of MI is the same at each level of PEER.Main effects of both MI and PEER, no interaction. Parallel lines: the effect of each factor is the same at both levels of the other.Synergistic MI×PEER interaction (plus both main effects): with both factors set to yes the outcome is much better than the main effects alone would predict.Synergistic interaction with a crossover: main effect of MI, no main effect of PEER (averages ≈ 14 both levels), but the effect of PEER reverses depending on MI.Antagonistic interaction (main effect of MI, none of PEER): with both factors set to yes, outcome is worse than expected from MI = yes alone.Antagonistic interaction with main effects of both factors: MI + PEER still produces the best outcome, just not as favorable as the main effects alone would predict.
Two types of 2-way interactions: “In a synergistic interaction, the combined effect of two or more factors is more favorable than would be expected based solely on the main effects.” “In an antagonistic interaction, the combined effect of two or more factors is less favorable than would be expected based solely on the main effects.”Narration, Lesson 2
Decision relevance: “an antagonistic interaction between two components does not necessarily mean you should not include both components in the intervention”, the MI + PEER combination can still be the best available outcome (as in the last plot).
Three-way interactions
Definition: “A three-way interaction means that a two-way interaction differs across a third factor.” One way to depict it graphically is a plot for each level of the third factor.Narration, Lesson 2
Synergistic three-way interaction: when TEXT = no the MI×PEER interaction is the crossover pattern; when TEXT = yes it disappears (dashed line flat). Outcome at all-three-yes is best of all, so the three-way interaction is synergistic.Antagonistic three-way interaction: when TEXT = no, MI×PEER is the mild antagonistic pattern; when TEXT = yes, PEER stops helping when MI = yes (solid line flat). Adhere dips at all-three-yes even though both main effects are positive.
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☐ All else being equal, in a balanced 2k experiment there is less power for detection of a regression coefficient corresponding to an interaction than to a regression coefficient corresponding to a main effect.
☑ In a synergistic interaction, the combined effect of two or more factors is more favorable than would be expected based solely on the main effects.
☑ The main effect of Factor A is defined as Factor A’s effect averaged over all the levels of all the other factors in the experiment.
☐ The interpretation of main effects is always the same whether dummy coding or effect coding is used.
Feedback when incorrect: All else being equal, in a balanced 2k experiment there is less power for detection of a regression coefficient corresponding to an interaction than to a regression coefficient corresponding to a main effect.This is incorrect. All else being equal, in a balanced 2k experiment power for detection of a regression coefficient is the same whether it corresponds to a main effect or an interaction effect.”The interpretation of main effects is always the same whether dummy coding or effect coding is used.This is incorrect. Main and interaction effects are typically interpreted differently depending on the numerical coding used in the analysis. In this course we use effect (-1, 1) coding exclusively.”
Lesson 3: Understanding the Basics of Powering a Factorial Experiment
The lesson reviews the logic of hypothesis testing and then shows what power is, what moves it, and how to power a factorial experiment in practice.
Hypothesis testing in a nutshell
You cannot directly observe the parameters μ₁ and μ₂; you estimate them with the sample means Ȳ₁ and Ȳ₂. The null hypothesis is H₀: μ₁ − μ₂ = 0. Even if H₀ is true you would not expect Ȳ₁ − Ȳ₂ = 0 every time, so you need a test statistic.
The test statistic: “the observed t equals the difference between the observed means divided by the standard deviation of Y divided by n.” The distribution of t when the null hypothesis is true is known; this is what enables classical hypothesis testing. The question asked is: assuming H₀ true, what is the probability of observing a t at least as large as t observed? That probability is the p-value. If it is low (by convention ≤ 5% of the time), decide “the null hypothesis is false”; if not low, “cannot reject the null hypothesis.”Narration, Lesson 3
Type I error and α: “A Type I error occurs when you reject the null hypothesis when it is true, in other words, when you mistakenly conclude an effect exists when it does not.” α denotes the Type I error rate, “the probability of rejecting the null hypothesis, given that the null hypothesis is true.”Narration, Lesson 3
Type II error and β: “A Type II error occurs when you fail to reject the null hypothesis when the null hypothesis is false, in other words, when you mistakenly overlook an effect.” β denotes the Type II error rate, “the probability of failing to reject the null hypothesis, given that the null hypothesis is false.” (β also denotes a regression coefficient; context tells which is meant.)Narration, Lesson 3
Power: “power is the probability of rejecting the null hypothesis conditional on the null hypothesis being false.” Power = 1 − β.Narration, Lesson 3
Central and noncentral distributions. The reference distribution for hypothesis testing is the distribution of the test statistic when H₀ is true, the central distribution; there is only one per test statistic. There are infinitely many alternative hypotheses (μ₁ − μ₂ could equal any number), and the distribution of the test statistic under a particular Hₐ is called a noncentral distribution. “Power analysis requires specifying the alternative hypothesis, usually via an expression of anticipated effect size. This is why you need to specify an effect size to conduct a power analysis.”
Schematic after the lesson's figure: the black curve is the central t distribution under H₀; the red curve is the noncentral distribution under a particular alternative (μ₁ − μ₂ = 2). Values at or beyond the boundary (a t a little larger than 2) lead to rejecting H₀; the yellow-shaded area under the alternative distribution is the probability of rejecting H₀ when H₀ is false: statistical power.
Considerations affecting power
Consideration
Direction (all else equal)
Why
Sample size N
larger N → more power
smaller standard errors, more precise estimates, less noise
Effect size
larger effect size → more power
less overlap between the distributions under H₀ and Hₐ
Type I error rate α
larger α → more power
moving the rejection boundary left captures a greater portion of the alternative distribution
Cluster structure (nesting)
matters only under between-cluster random assignment
addressed in Lesson 5
Adding the blue dashed alternative (μ₁ − μ₂ = 3): more of that distribution lies beyond the boundary, so the power region is larger.Increasing the Type I error rate moves the rejection boundary left; a greater portion of the alternative distribution leads to rejection, so power increases. This is a trade-off (Lesson 4).
Expressing effect size
Cohen's d: “Cohen's d is equal to the difference between the two means divided by the standard deviation. Essentially, this is a standardized difference between two means.”Narration, Lesson 3
Cohen's benchmarks for d: “He designated a d ranging between 0.2 and 0.3 as small, a d of about 0.5 as medium and any d of 0.8 or greater as large.”Narration, Lesson 3
Standardized regression coefficient (β*). For the regression of Y on X₁, X₂, …, with uncorrelated predictors,
β*₁ = √( r²X₁Y / (1 − r²X₁Y) )
Cohen's benchmarks for β*: “A beta star of 0.1 is considered small, a beta star of 0.25 is considered medium and a beta star of 0.4 or greater is considered large.” (Different from the d benchmarks.)Narration, Lesson 3
How to power factorial experiments
Two entry points: choose α, level of power and expected effect size, then establish the necessary N; or, when the maximum attainable N is fixed by budget, specify α, power and N and use the power analysis to determine the minimum detectable effect.
The 2^k equal-power property: “for a 2^k experiment, with equal Ns, with effect coding, the same regression weight is associated with the same power irrespective of which effect the regression weight represents.” So powering for main effects powers interactions too.
Decide which effects matter, usually the main effects, then set an effect size for each. The effect size can be the actual anticipated size or “the minimum effect that you care about.”
The “audition” metaphor: “You can consider the optimization trial as the way of 'auditioning' each component for a role in your intervention. From this perspective, to be selected for the intervention, each component has to demonstrate an effect of at least some minimum size.”
Power to the smallest: “Identify the smallest effect size and go from there. This is the only effect size you need. If you power the factorial experiment so that you have adequate power for this effect size, you will have adequate power to detect any main effects or interactions that size or larger.”
Covariate hint: “adding a covariate such as a pretest will often reduce sample size requirements. It makes sense to do this only if the correlation between the pretest and the outcome is substantial, above about a correlation of 0.5. If the correlation is small, this approach can, under some circumstances, reduce power, so be careful.”
Knowledge check
MULTIPLE CHOICE: Which ONE of the following statements best describes the relation between Type I error rate and power?
☐ All else being equal, increasing the Type I error rate will decrease power.
☐ Increasing the Type I error rate has no effect on power.
☑ All else being equal, increasing the Type I error rate will increase power.
Lesson 4: Two Distinct Perspectives on Research: Conclusion-Priority and Decision-Priority
The lesson contrasts the research philosophy most scientists are trained in with an alternative that fits intervention optimization, and shows when each belongs.
The trained approach: conclusion-priority: plan the study to conclude whether an effect exists: assess likely effect size, determine N for sufficient power (say ≥ .8) at the usual α = .05, conduct the study only if resources allow; significant results → reject H₀ and conclude the effect exists; non-significant results → “you don't know whether or not the effect exists.” “We will call this the conclusion- priority perspective.”Narration, Lesson 4
The alternative: decision-priority: optimizing an intervention requires a decision about which components make up the optimized intervention, within given resources. Per the resource management principle, the job is to make the best use of resources to get the largest amount of, and highest-quality, information to enable that decision. “From this perspective, saying We cannot decide, the information is not good enough is not an option. Every component examined in the optimization trial has to be either included in the optimized intervention or not. We will call this the decision-priority perspective.”Narration, Lesson 4
How the two perspectives view Type I and Type II error rates
Peer review as “science court”: “In the conclusion- priority perspective, the investigator is planning for results to be judged in peer review. Think of peer review as 'science court' where your work is going to be judged.” The rules: maximum acceptable α =.05; minimum acceptable power in the.7,.8 range, so β in the.2,.3 range is acceptable. “Notice that from this perspective, the acceptable Type II error rate is routinely 4-6 times the acceptable Type I error rate.”Narration, Lesson 4
The conclusion-priority investigator's stance: “I would much rather overlook an effect that exists, than mistakenly conclude an effect exists when it doesn't. To me, making a Type I error is 4-6 times more painful or costly than a Type II error.”Narration, Lesson 4
The decision-priority stance: “To an investigator optimizing an intervention, overlooking an effective component may be at least as painful or costly as mistakenly concluding that a component is effective.” This suggests a different balance of Type I and Type II error rates could be considered.Narration, Lesson 4
The trade-off, named: “You are strategically accepting a higher Type I error rate, so that you can reduce the Type II error rate and increase power.”Narration, Lesson 4
The worked example
Suppose you want power ≥ .8 and your resources allow at most 100 participants. “You conduct a power analysis using your anticipated effect size and alpha of 0.05, and find you will need a sample size of 126.” A conclusion-priority investigator would stop here. A decision-priority investigator, guided by the resource management principle, considers raising α: at α =.10 the required N drops to 98, and the trial can be accomplished within resources, accepting a higher probability of a Type I error.
Link to the resource management principle: “You've probably noticed that the decision-priority approach and the resource management principle are closely related.” The overall idea is that you need to use resources strategically, “you need to manage resources so as to enable you to make the best decisions. This sometimes means managing the Type I and Type II error rates.”Narration, Lesson 4
Which perspective when: “In the optimization phase of MOST, typically, an investigator is working from the decision-priority perspective. This is particularly true when planning the optimization trial. In the evaluation phase of MOST, when the objective is establishing whether or not, the intervention package has a statistically and clinically significant effect, typically the investigator is working from the conclusion-priority perspective.”Narration, Lesson 4
Summary of the two perspectives
Conclusion-priority
Decision-priority
OK to conclude “we don't know”
Necessary to make a decision (include or not)
Hypothesis-testing is the “master,” with a clear set of rules
Hypothesis-testing is one tool for decision-making, applied in any reasonable and transparent manner
α ≤ .05 dictated by “science court”; the investigator has no say
α selected based on the resource management principle
By convention, Type II error rate is many times larger than Type I error rate
Balance between Type I and Type II error rates determined by the investigator's own preferences
Usually suitable for the evaluation phase of MOST
Usually suitable for the optimization phase of MOST
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☐ When working from the decision-priority perspective, it is never a good idea to increase alpha (the Type I error rate) beyond .05.
☑ The decision-priority perspective and the resource management principle are closely related.
☑ The decision-priority perspective is usually most helpful in the optimization phase of MOST, and the conclusion-priority perspective is usually most helpful in the evaluation phase of MOST.
Feedback when incorrect: When working from the decision-priority perspective, it is never a good idea to increase alpha (the Type I error rate) beyond .05.This is incorrect. When working from the decision-priority perspective, the investigator carefully balances Type I and Type II error rates with the resource management principle in mind.
Lesson 5: Recognizing and Dealing with a Cluster Structure
The lesson explains what a cluster structure is, why it matters, and how it changes sample size requirements, depending on the type of random assignment.
Cluster structure (nesting): “A cluster structure occurs whenever participants are members of clusters, such as schools, neighborhoods, medical practices, and the like.” “Sometimes participants within clusters tend to be more alike or are expected to become more alike than participants in the overall target population.”Narration, Lesson 5
Two kinds of clusters: pre-existing (clinics or medical practices; schools or classrooms; families; faith congregations; communities) and experiment-induced. “Experiment-induced clusters are formed in the course of experimentation. For example, suppose one candidate component is a treatment that is delivered in a group setting. You would expect participants to become more similar over the course of treatment because of the group interaction.”
Intraclass correlation: “The degree of within-cluster similarity in the outcome Y is expressed in the intraclass correlation, which we will call ICC sub y.”Narration, Lesson 5
Why clustering matters: two reasons
Reason 1: contamination. “Contamination occurs when treatment, 'bleeds' across experimental conditions. Contamination tends to reduce treatment effects.” Examples: a married couple in a smoking cessation experiment, one assigned to nicotine lozenges and the other to control, with lozenges shared across the couple; or a group-delivered component in an optimization trial where participants discuss which components they were assigned and share resources. If there could be substantial contamination, it may not make sense for individuals within a cluster to be in different experimental conditions.
Reason 2: the intervention may target the cluster level. A school-based drug abuse prevention program delivered in the classroom targets individual children's behavior and social norms about drug use at the classroom and school levels, “part of the purpose of the intervention may be to change these classroom and school level norms.”
Two types of random assignment
Within-cluster randomization (= individual randomization): “an individual in a given cluster may be assigned to any experimental condition.”Narration, Lesson 5
Between-cluster randomization (= cluster randomization): “entire clusters are assigned to experimental conditions, so all the individuals in a cluster are in the same experimental condition.”Narration, Lesson 5
Effect on power: “If you use within-cluster random assignment, the cluster structure has little effect on power. If you use between-cluster random assignment, a cluster structure can have a huge effect on power.”Narration, Lesson 5
The design effect D
D is a multiplier: “D is a multiplier, if you would need a particular sample size N without a cluster structure, you will need D times N with a cluster structure.” Example: if a standard power analysis gives N = 400 and D = 1.2, you need 400 × 1.2 = 480 participants.Narration, Lesson 5
D = 1 + (n − 1) · ICCX · ICCY
where ICCX is the intraclass correlation for the X (predictor) variables, “the codes representing main effects and interactions”ICCY is the intraclass correlation for the Y (outcome) variable, and n is the number per cluster (in reality it varies across clusters).
Within-cluster randomization: if all participants have the same assignment probabilities, ICCX = 0, everything to the right of the plus sign drops out, and D = 1. “One way to see why the ICC sub x equals 0 is to note that when individual random assignment is done properly, knowing a participant's cluster membership reveals nothing about what condition they've been assigned to.” How to ensure equal assignment probabilities: “Be sure to replicate the entire experiment within each cluster.” Example: a 2^4 experiment in 5 clinics, implement the entire 2^4 in each clinic, so a participant in any clinic has a 1 in 16 chance of any condition. (Note: this conclusion concerns power only; there may be other considerations, e.g. variability in effects across clusters.)
Between-cluster randomization: cluster membership determines experimental condition, so ICCX = 1 and the expression simplifies to D = 1 + (n − 1) · ICCY. All else equal, D is larger when cluster size n is larger and when ICCY is larger.
Two considerations when between-cluster randomization is necessary
1. Will I have enough power? “The issue here is essentially the same as it would be for an RCT and is explored beautifully in Murray's classic book on cluster randomized trials.”
2. Given enough power, will I have enough clusters to assign to experimental conditions? You could have enough participants but not enough clusters to populate all conditions if the clusters are large. “It's a good idea to assign at least two clusters per condition. Technically, you could assign just one, but then if a cluster drops out, it leaves you with an empty experimental condition.” If clusters are scarce, “you may want to consider a fractional factorial design, which would have the same sample size requirements but would involve fewer experimental conditions” (Dziak et al., 2012).
Open area: for experiment-induced clustering in factorial optimization trials see Nahum-Shani & Dziak (2018) and Nahum-Shani et al. (2018); for clustering in SMARTs see NeCamp et al. (2017).
Knowledge check
MULTIPLE RESPONSE: Which of the following are reasons to give serious consideration to cluster randomization when planning an optimization trial? Select all that apply.
☑ If individuals within a cluster are in different experimental conditions, there could be contamination. Original course feedback (source anomaly, not endorsed):This is incorrect. EASE does not involve maximizing all four of these attributes of an intervention simultaneously.
☑ The intervention being optimized is designed to have effects at both the individual level and the cluster level. Original course feedback (source anomaly, not endorsed):This is incorrect. EASE does not involve balancing efficiency against affordability and scalability.
☐ Cluster randomization will increase statistical power.
☑ One or more components being examined are delivered in a group setting.
Feedback when incorrect: Cluster randomization will increase statistical power.This is incorrect. Cluster randomization generally decreases statistical power.
Module quiz (self-test)
Reveal the 5 quiz items with answers
MULTIPLE RESPONSE: Which of the following statements about dummy coding and effect coding are correct? Select all that apply.
☑ Dummy coding and effect coding are both ways of numerically representing factor levels in a regression model.
☑ Dummy coding and effect coding produce the same value for the omnibus (overall) F statistic.
☐ Dummy coding and effect coding always produce main effect and interaction effect estimates that can be interpreted in the same way.
☐ Whether the data analyst uses dummy coding or effect coding, the same main effects and interaction effects will always be statistically significant.
Feedback when incorrect: Dummy coding and effect coding always produce main effect and interaction effect estimates that can be interpreted in the same way.This is incorrect. Dummy coding and effect coding usually produce different effect estimates.Whether the data analyst uses dummy coding or effect coding, the same main effects and interaction effects will always be statistically significant.This is incorrect. Different main effects and interaction effects may be statistically significant depending on whether dummy coding or effect coding is used.
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☐ All else being equal, in a balanced 2k experiment there is less power for detection of a regression coefficient corresponding to an interaction than to a regression coefficient corresponding to a main effect.
☑ In a synergistic interaction, the combined effect of two or more factors is more favorable than would be expected based solely on the main effects.
☑ The main effect of Factor A is defined as Factor A’s effect averaged over all the levels of all the other factors in the experiment.
☐ The interpretation of main effects is always the same whether dummy coding or effect coding is used.
Feedback when incorrect: All else being equal, in a balanced 2k experiment there is less power for detection of a regression coefficient corresponding to an interaction than to a regression coefficient corresponding to a main effect.This is incorrect. All else being equal, in a balanced 2k experiment power for detection of a regression coefficient is the same whether it corresponds to a main effect or an interaction effect.The interpretation of main effects is always the same whether dummy coding or effect coding is used.This is incorrect. Main and interaction effects are typically interpreted differently depending on the numerical coding used in the analysis. In this course we use effect (-1, 1) coding exclusively.”
MULTIPLE CHOICE: Which ONE of the following statements best describes the relation between Type I error rate and power?
☐ All else being equal, increasing the Type I error rate will decrease power.
☐ Increasing the Type I error rate has no effect on power.
☑ All else being equal, increasing the Type I error rate will increase power.
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☐ When working from the decision-priority perspective, it is never a good idea to increase alpha (the Type I error rate) beyond .05.
☑ The decision-priority perspective and the resource management principle are closely related.
☑ The decision-priority perspective is usually most helpful in the optimization phase of MOST, and the conclusion-priority perspective is usually most helpful in the evaluation phase of MOST.
Feedback when incorrect: When working from the decision-priority perspective, it is never a good idea to increase alpha (the Type I error rate) beyond .05.This is incorrect. When working from the decision-priority perspective, the investigator carefully balances Type I and Type II error rates with the resource management principle in mind.
MULTIPLE RESPONSE: Which of the following are reasons to give serious consideration to cluster randomization when planning an optimization trial? Select all that apply.
☑ If individuals within a cluster are in different experimental conditions, there could be contamination. Original course feedback (source anomaly, not endorsed):This is incorrect. EASE does not involve maximizing all four of these attributes of an intervention simultaneously.
☑ The intervention being optimized is designed to have effects at both the individual level and the cluster level. Original course feedback (source anomaly, not endorsed):This is incorrect. EASE does not involve balancing efficiency against affordability and scalability.
☐ Cluster randomization will increase statistical power.
☑ One or more components being examined are delivered in a group setting.
Feedback when incorrect: Cluster randomization will increase statistical power.This is incorrect. Cluster randomization generally decreases statistical power.
Synthesis: the module's machinery in one place
The agent's reading: what this module does for the MOST story
Module 3 argued that the factorial experiment is the efficient workhorse of the optimization phase; Module 4 shows what the numbers from one actually mean, and the running 2^3 example is doing real work: every lesson returns to the same eight conditions, so the coding tables, the plots, and the design effect formula all connect to a single mental model rather than a sequence of disconnected techniques.
Coding is an interpretive choice, not a mechanical one. The same data analyzed under dummy vs effect coding gives the same omnibus F but differently defined effects. Effect coding's main effect is “averaged across all levels of all other factors”, the effect in the whole experiment, not in a reference condition. That matches the optimization question exactly: a component that will be deployed alongside other components should be judged by its average performance across the other components' levels, and that is precisely what the factorial estimate is. The course's insistence on effect coding is the statistical echo of the resource management principle.
Interactions are affordable and interpretable. The equal-power property of balanced 2^k designs is the load-bearing fact: powering for main effects powers interactions at the same level, and effect coding keeps the estimates uncorrelated. The plot-reading rules (parallel lines = no interaction; tipped lines = synergistic/antagonistic) translate directly into component-retention decisions, with the reminder that an antagonistic interaction does not automatically disqualify a component, since the combination can still be the best available outcome.
Powering is a decision, not a ritual. The “audition” framing, power to the smallest effect you care about, makes the power analysis serve the selection decision, and the Lesson 4 contrast makes the alpha choice a deliberate, transparent trade-off: the optimization phase can reasonably accept a higher Type I error rate to buy power within fixed resources, while the evaluation phase must satisfy “science court.” This maps onto the distinction between formative/optimization work and confirmatory evaluation in any intervention research program.
Nesting changes the arithmetic. The design effect D = 1 + (n − 1)·ICCX·ICCY is the reminder that with between-cluster randomization, sample size is not N but effective N/Dand that contamination and cluster-level targets can push an optimization trial toward cluster randomization whether or not the statistics prefer it. The two practical questions (enough power; enough clusters per condition) and the fractional-factorial escape hatch are the field-level translation of the same resource management logic.
Glossary
ANOVA (analysis of variance)
the standard analysis for data from a factorial experiment; in this course, performed via regression.
Main effect of Factor A
the effect of Factor A averaged across all levels of all other factors.
Interaction (2-way)
half the difference between the effect of Factor A at level 1 of Factor B and its effect at level 2 of Factor B, averaged across all other factors; conceptually, whether the effect of A depends on the level of B. This course defines the interaction as the corresponding regression coefficient.
3-way (higher-order) interaction
an interaction involving three or more factors; a three-way interaction means a two-way interaction differs across a third factor.
Dummy coding
representing factor levels numerically with 0 and 1.
Effect coding
representing factor levels numerically with −1 and +1 (for a 2^k experiment); the standard in this course.
Omnibus F
the overall F statistic testing the model as a whole; identical under dummy and effect coding.
Regression weight (β)
a coefficient in the regression model; each β (except the intercept) corresponds to a main effect or interaction estimate; under effect coding, β = effect/2.
Ŷ (Y-hat)
the predicted outcome for an experimental condition; an estimate of the expected outcome associated with that condition.
Predicted / marginal mean
a mean based on the Ŷs; marginal means are collapsed across one or more factors.
Synergistic interaction
combined effect of two or more factors more favorable than expected from the main effects alone.
Antagonistic interaction
combined effect of two or more factors less favorable than expected from the main effects alone.
Type I error / α
rejecting H₀ when it is true; α is its rate (probability of rejecting H₀ given H₀ true).
Type II error / β
failing to reject H₀ when it is false; β is its rate (probability of failing to reject H₀ given H₀ false).
Statistical power
the probability of rejecting H₀ given that H₀ is false; equals 1 − β.
Central / noncentral distribution
the distribution of the test statistic under H₀ (central) vs under a particular alternative hypothesis (noncentral); power compares them.
Cohen's d
standardized difference between two means: (Ȳ₁ − Ȳ₂)/s; benchmarks ≈.2,.3 small,.5 medium, ≥.8 large.
Standardized regression coefficient (β*)
effect size for a regression coefficient; benchmarks .1 small, .25 medium, ≥ .4 large.
Conclusion-priority perspective
research planned to draw generalizable conclusions under peer-review rules (α ≤.05, power ≥.7,.8); governs the evaluation phase.
Decision-priority perspective
research planned to support component-inclusion decisions under resource constraints; may adjust α; governs the optimization phase and is closely related to the resource management principle.
Resource management principle
the investigator must make the best and most efficient use of available resources when obtaining scientific information.
Cluster structure (nesting)
participants nested within clusters (schools, clinics, families, etc.) who tend to be more alike than the target population.
Intraclass correlation (ICC)
degree of within-cluster similarity in a variable (X predictors or Y outcome).
Contamination
treatment “bleeding” across experimental conditions; tends to reduce treatment effects.
Within-cluster (individual) randomization
individuals in a cluster may be assigned to any condition; makes ICCX = 0 and D = 1.
Between-cluster (cluster) randomization
entire clusters assigned to conditions; makes ICCX = 1 and D = 1 + (n − 1)·ICCY.
Design effect D
multiplier on N required by clustering: D = 1 + (n − 1)·ICCX·ICCY.
Fractional factorial design
a design using fewer experimental conditions than the full factorial while keeping the same sample size requirements; useful when clusters are scarce.
Sources
Lesson 1: The Regression Model and Coding (14:23): video · slide deck (PDF)
Lesson 2: Interpretation of Interaction Effects (19:22): video · slide deck (PDF)
Lesson 3: Understanding the Basics of Powering a Factorial Experiment (16:12): video · slide deck (PDF)
Lesson 4: Two Distinct Perspectives on Research: Conclusion-Priority and Decision-Priority (11:05): video · slide deck (PDF)
Lesson 5: Recognizing and Dealing with a Cluster Structure (11:17): video · slide deck (PDF)
References cited in the lessons:
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer.
Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202 to 224.
Dziak, J. J., Nahum-Shani, I., & Collins, L. M. (2012). Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations. Psychological Methods, 17, 153 to 175.
Kugler, K. C., Dziak, J. J., & Trail, J. (2018). Coding and interpretation of effects in analysis of data from a factorial experiment. In L. M. Collins & K. C. Kugler (Eds.), Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer.
Murray, D. M. (1998). Design and analysis of group-randomized trials. Oxford: Oxford University Press.
Nahum-Shani, I., & Dziak, J. J. (2018). Multilevel factorial designs in intervention development. In L. M. Collins & K. C. Kugler (Eds.), The multiphase optimization strategy (MOST): Advanced topics. New York: Springer.
Nahum-Shani, I., Dziak, J. J., & Collins, L. M. (2018). Multilevel factorial designs with experiment-induced clustering. Psychological Methods, 23, 458 to 479.
NeCamp, T., Kilbourne, A., & Almirall, D. (2017). Comparing cluster-level dynamic treatment regimens using sequential, multiple assignment, randomized trials: Regression estimation and sample size considerations. Statistical Methods in Medical Research, 26, 1572 to 1589.
Study notes prepared from the course's narrated videos, slide decks, and in-course questions. Quoted definitions are verbatim from the narration unless marked “Slides.” The synthesis section is the notes author's reading, not course content.