← MOST study notes index · CADIO / NYU School of Global Public Health
MOST from a Methodological Perspective: Module 3
Study notes · CADIO / NYU School of Global Public Health
Module 3: “Some Conceptual and Technical Aspects of the Factorial Experiment”
Authors: Linda M. Collins and Kate Guastaferro (School of Global Public Health, New York University)
Format: 6 lessons (Lesson 1, set reading, Lessons 2 to 5) · 5 videos (approx. 49 min) · 5 slide decks (PDF) · 5 knowledge checks
Set reading: Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 3, plus the Kugler, Dziak & Trail chapter listed under the set reading lesson)
Access: free via CADIO share links, or Coursera
Source anomalies found in Module 3 (flagged, not silently corrected). (1) In the Lesson 2 knowledge check the course marks both “True” and “False” as correct for a single multiple-choice statement; both checkboxes are reproduced exactly as in course.json and the item is flagged, not endorsed. (2) The YouTube caption transcripts contain transcription artefacts (for example “Corsera” for Coursera, “perition” for per-condition, “boneroni” for Bonferroni, “fstistic” for F statistic, “Cougler” for Kugler, “Fiser” for Fisher, “pier” for peer, “Zcore” for z-score, “at least8” for at least .8, “23x3” for 2^3 x 3, “perle” for per-level, “evenumbered/oddnumbered” for even-numbered/odd-numbered, “74” for .74); these are preserved inside narration quotes and noted where they occur. (3) Slide and caption renderings of the second author's surname vary (“Guastafello”, “Guastafferro”, “Guapero”); the standard spelling used in the course text is Guastaferro.
The module's argument in one paragraph.
Module 2 established the factorial experiment as the workhorse optimisation trial design; this module supplies the technical machinery needed to design, analyse and interpret one. The argument runs in four steps. First, coding matters: a factorial ANOVA is a regression model, and the choice between effect coding (−1, 1) and dummy coding (0, 1) changes what the coefficients mean, their standard errors, and which hypothesis tests are significant, even though the omnibus F is identical; effect coding is therefore used throughout this course (Lessons 1 to 2). Second, power in a factorial experiment does not depend on the per-condition sample size but on the N per level of each factor, because every main effect is an average over half the conditions; this is why a 2^4 with n = 22 per condition (N = 352) can be well powered, and why adding factors often requires no additional participants at all (Lesson 3). Third, the same logic has a limit: adding levels to a factor splits the per-level N, roughly halves the per-comparison N (from 176 to about 120 for High vs No in the 2 x 2 x 3 x 2 example), and restoring power costs at least 50% more participants, so dose questions beyond two levels should be handled strategically (Lesson 4). Fourth, experimental control in a factorial experiment is distributed: each factor has its own control level, different factors can use different types of control (for example an attention placebo, diet-information texts, a nature video), and nearly every participant is in the control level of some factor (Lesson 5). The through-line is efficiency: the factorial experiment buys the ability to estimate interactions and to power every effect with the whole sample, and the coding, balance and control machinery in this module is what makes that arithmetic work.
Lesson 1: Comparison of Effect Coding and Dummy Coding, Part 1
The lesson introduces the two coding schemes used for factorial ANOVA and shows why the choice matters. The running example is the 2^3 factorial experiment on the HIV viral-load intervention with three candidate components, motivational interviewing (MI), peer mentoring (PEER) and text message support (TEXT), each at two levels.
Coding matters: “Coding matters in interpretation of effects in a statistical analysis.” “coding refers to how the levels of a factor are represented numerically in data analysis.” The course uses effect coding: “In this course, we are using effect coding. In effect coding, as applied in a 2 to the k factorial experiment, the codes are minus1 and one.” The alternative is dummy (reference category) coding: “you may be more accustomed to so-called dummy coding or reference category coding in which the codes are zero and one.”Narration, Lesson 1; deck slides 4 to 6
Same overall fit, different estimates: “In a factorial ANOVA, the estimates of main effects and interactions are generally not the same across the two types of coding. Although the overall fit is the same that is the overall often called omnibus fs are identical.” The omnibus F test does not distinguish the two schemes; the individual effect estimates do.Narration, Lesson 1; deck slides 5 to 6
Interpretation of the estimates: “when effect coding is used, the main effect of a factor is interpreted as that factor's effect on average across all combinations of levels of all of the other factors.” Dummy-coded estimates should not be read that way: “we prefer not to use the terms main effect and interaction when dummy coding is used because dummy coding generally does not produce estimates that correspond to the main effect and interaction effect definitions that Fiser gave us” (caption artefact: “Fiser” for Fisher). “Instead, when dummy coding is used, we prefer to use the terms first order effects, second order effects, etc.”Narration, Lesson 1; deck slide 6
Point 1 about effect coding: uncorrelated estimates. “In a properly conducted factorial experiment, effect coding produces main effect and interaction estimates that are all uncorrelated if the experimental conditions have equal sample sizes.” “If the sample sizes vary across conditions, the correlations among effects will be modest unless there is a lot of variability.” “By contrast, with dummy coding, effect estimates are usually highly correlated.”Narration, Lesson 1; deck slides 7 to 8
Consequence of uncorrelated estimates: “with effect coding in a perfectly balanced experiment, removing an effect from the regression model has no impact on estimates of the remaining effects.” “The hypothesis tests will change because variance is being moved into the error term and the degrees of freedom change but the effect estimates will remain identical.” By contrast, “with dummy coding in any experiment, removing an effect from the model usually changes the estimates of the remaining effects.”Narration, Lesson 1; deck slide 8
Point 2 about effect coding: identical standard errors. “In a perfectly balanced 2 to the k factorial experiment, the standard errors are identical for all effect estimates. This applies to regression coefficients corresponding to main effects and interactions up to the kway interaction” (caption artefact: “kway” for k-way). “By contrast, with dummy coding, the standard errors usually increase as the order of the effect increases.”Narration, Lesson 1; deck slide 9
The regression model and where the X's come from: “It's the regression model for a 2 to the 3 experiment when effect coding is used.” The b's are the effect estimates and “The X's come from the effect coding process.” In the effect-code table, “where the MI factor is set to the no level in the experiment the effect code is minus1. Similarly, where the MI factor is set to yes in the experiment, the effect code is plus one.”Narration, Lesson 1; deck slides 11 to 14
Notation for dummy coding: “We are going to use the letter A to refer to regression coefficients when dummy coding is used.” “We are going to use the letter Z to refer to dummy-coded effect vectors. Please note that in this context, the letter Z does not refer to a standardized variable. That is, it does not refer to a Zcore” (caption artefact: “Zcore” for z-score).Narration, Lesson 1; deck slide 15
The dummy-coded regression model: “It looks very similar to the regression model when effect coding is used. There are still seven effects to be estimated plus the intercept.” In the dummy-coded model, “a sub 1, a sub 2 and a sub3 correspond to first order effects. a sub4, a sub5 and a sub6 correspond to second order effects and a sub7 corresponds to the third order effect.”Narration, Lesson 1; deck slide 16
You may be thinking: first-order effects are nearly identical: “why are you saying effect coding and dummy coding are so different? Where you would use a zero in dummy coding, you use a minus one in effect coding. That doesn't seem very different.” “In fact, it isn't very different for coding the first order effects.” “If there are no higher order effects or interactions in the model, effect and dummy coding produce identical results. The difference emerges when interactions or higher order effects are added to the model.”Narration, Lesson 1; deck slide 20
Interaction codes are products of main effect codes: “The effect codes for the MI by peer interaction are obtained by multiplying the codes for the main effect of MI by the codes for the main effect of pier” (caption artefact: “pier” for peer). The same rule holds for dummy coding: “in dummy coding, the higher order effect vectors are obtained by multiplying the corresponding first order effect vectors.”Narration, Lesson 1; deck slides 21 to 28
Why the estimates differ: “If effect codes and dummy codes were simply linear transformations of each other, that would be evident because the overall pattern would be the same in these two panels.” “It is evident that the patterns are not identical. There are a lot more zeros in the dummy codes than there are minus ones in the effect codes.” “This is due to the multiplication of the vectors of zeros and ones in dummy coding as opposed to the multiplication of vectors of minus1's and positive ones in effect coding.”Narration, Lesson 1; deck slides 29 to 31
Figure 1: Effect-coded and dummy-coded interaction values for the 2^3 example (redrawn)
| Cell | MI×PEER (effect) | MI×TEXT (effect) | PEER×TEXT (effect) | MI×PEER×TEXT (effect) | MI×PEER (dummy) | MI×TEXT (dummy) | PEER×TEXT (dummy) | MI×PEER×TEXT (dummy) |
|---|
| 1 | +1 | +1 | +1 | −1 | 0 | 0 | 0 | 0 |
| 2 | +1 | −1 | +1 | −1 | 0 | 0 | 0 | 0 |
| 3 | −1 | +1 | −1 | +1 | 0 | 0 | 0 | 0 |
| 4 | −1 | −1 | +1 | −1 | 0 | 0 | 1 | 0 |
| 5 | −1 | −1 | +1 | +1 | 0 | 1 | 0 | 0 |
| 6 | −1 | +1 | −1 | −1 | 0 | 1 | 0 | 0 |
| 7 | +1 | −1 | −1 | −1 | 1 | 0 | 0 | 0 |
| 8 | +1 | +1 | +1 | +1 | 1 | 1 | 1 | 1 |
Redrawn from Lesson 1 deck slides 29 to 31 (“Comparison of effect coding (−1) and dummy coding (0)”). Effect-coded interactions are products of −1/+1 main effect codes; dummy-coded higher-order effects are products of 0/1 first-order codes, so they contain far more zeros. This pattern difference is why the two codings estimate different quantities.
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? There may be more than one.
- ☐ Dummy coding and effect coding produce exactly the same results in factorial analysis of variance.
- ☑ In effect-coded factorial analysis of variance of data from a perfectly balanced factorial experiment, the standard errors are identical for all effect estimates.
- ☐ In effect-coded factorial analysis of variance of data from a perfectly balanced factorial experiment, the effect estimates are often highly correlated.
- ☑ In both effect coding and dummy coding, the codes for interactions are obtained by multiplying main effect codes.
Feedback when incorrect (source text, retained as provided): Answer A. is incorrect. Dummy coding and effect coding always produce the same overall F but can produce different effect estimates.
Answer C. is incorrect. In effect-coded factorial analysis of variance of data from a perfectly balanced factorial experiment, all main effect and interaction estimates are uncorrelated.
Reading for All of Module 3
This lesson has no video or slide deck; it lists the set reading for the module.
- Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 3 in the course textbook; a free PDF may be available through university libraries.)
- Kugler, K. C., Dziak, J. J., & Trail, J. (2018). Coding and interpretation of effects in analysis of data from a factorial experiment. In L. M. Collins & K. C. Kugler (Eds.), Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer. (This chapter is the deep-dive reference for the coding material in Lessons 1 to 2.)
Lesson 2: Comparison of Effect Coding and Dummy Coding, Part 2
The lesson shows how the choice of coding changes the interpretation of the intercept, main effects and interactions in a regression analysis of the same data, and explains why the course uses only effect coding.
The intercept under the two codings: “when effect coding is used, the intercept equals the grand mean.” Under dummy coding, “Here the intercept is the mean when all factor levels are set to the level that has been coded zero.” In the example, “that would be the mean when the MI, peer, and text factors are all set to no.”Narration, Lesson 2; deck slides 6 to 7
Main effect coefficient b1 versus a1: “B sub one that is the regression coefficient corresponding to the main effect of MI is 1/2 of the effect of MI averaged across all levels of the other factors.” By contrast, the dummy-coded first-order coefficient “is the effect of MI when peer and text are set to the level coded zero. In other words, the effect of MI in the specific circumstances where peer and text are both set to no.”Narration, Lesson 2; deck slides 8 to 9
Interaction coefficient b4 versus a4: “When effect coding is used, B sub4 represents 1/2 of the MI by peer interaction. This is the difference between the effect of MI when pier equals yes and the effect of MI when pier equals no averaged across the levels of text” (caption artefact: “pier” for peer). Under dummy coding, “a sub4 represents the difference between the effect of MI when pier equals yes and the effect of MI when pier equals no when text is set to no.”Narration, Lesson 2; deck slides 10 to 11
Different results from the same data: “in general, data analysis using these two approaches will yield different results, which makes sense given that effect coding and dummy coding represent different definitions of effects.” The same data set analysed both ways is shown in Figure 2.Narration, Lesson 2; deck slide 12
Standard errors in the balanced example: “When effect coding is used, the standard errors are identical for all of the effect estimates. When dummy coding is used, the standard errors are identical for effects of a particular order.” “the standard error is 74 for all of the first order effects, but they are different for effects of different orders” (caption artefact: “74” for .74). “You can see the standard error is larger for the second order effects and larger still for the third order effect.”Narration, Lesson 2; deck slides 12 to 13
Hypothesis tests differ: “The pattern of results is different across the two types of coding. The main effect and first order effects of MI are both significant. The main effect of peer is non-significant but the first order effect is not only significant it has a negative effect.” An effect can change from non-significant to significant, and from positive to negative, purely through the coding choice.Narration, Lesson 2; deck slide 14
Why this matters in practice: “Statistical software documentation does not always make it clear whether effect coding or dummy coding is used.” “When you use software to power an optimization trial, make sure you know whether the effects you are powering for are effect coded or dummy coded. It makes a difference.”Narration, Lesson 2; deck slide 15
To sum up: codes are needed: “it's necessary to use numeric codes to define the effects to be estimated. There are various approaches to this. Two of the most common ones are dummy coding, which uses zeros and ones, and effect coding, which for a 2 to the k experiment uses minus1's and ones.”Narration, Lesson 2; deck slide 16
To sum up: same F, different effects: “All else being equal, effect coding and dummy coding produce the same overall or omnibus fstistic” (caption artefact: “fstistic” for F statistic). “However, these two approaches define the effects differently. This means that hypothesis tests will be different and importantly the effects must be interpreted differently.”Narration, Lesson 2; deck slide 17
Why the course prefers effect coding: “Those of us working in intervention optimization tend to prefer effect coding because we prefer the interpretability, robustness, and generalizability of effects averaged across the levels of the other factors.” “we see the main effect as more relevant to building optimized interventions than the first order effect.” “We also like it that effect estimates are uncorrelated if the experiment is perfectly balanced and only moderately correlated otherwise.” Therefore, “in this course, we use only effect coding.”Narration, Lesson 2; deck slide 18
Practical advice and the deep-dive reference: “In your power analysis and data analysis, be sure you know which approach to coding is being used.” “If you are interested in a deep dive about the comparison of effect coding and dummy coding, see the Cougler, Zak, and Trail chapter in the Collins and Cougler book” (caption artefacts: “Cougler” for Kugler, “Zak” for Dziak; the reference is Kugler, Dziak & Trail, 2018, in Collins & Kugler, eds.).Narration, Lesson 2; deck slides 19, 22
Figure 2: Same data analysed with effect coding and dummy coding (redrawn)
| Term | df | Effect coding b | Effect coding SE | Effect coding p | Dummy coding b | Dummy coding SE | Dummy coding p |
|---|
| Intercept | 1 | −0.17 | 0.19 | 0.36 | −3.63 | 0.53 | <0.01 |
| MI | 1 | 6.52 | 0.19 | <0.01 | 6.46 | 0.74 | <0.01 |
| PEER | 1 | 0.12 | 0.19 | 0.53 | −4.89 | 0.74 | <0.01 |
| TEXT | 1 | −0.21 | 0.19 | 0.27 | −0.09 | 0.74 | 0.91 |
| MI×PEER | 1 | 3.14 | 0.19 | <0.01 | 11.51 | 1.05 | <0.01 |
| MI×TEXT | 1 | 0.41 | 0.19 | 0.04 | 0.59 | 1.05 | 0.58 |
| PEER×TEXT | 1 | −0.32 | 0.19 | 0.10 | −2.31 | 1.05 | 0.04 |
| MI×PEER×TEXT | 1 | 0.26 | 0.19 | 0.17 | 2.09 | 1.49 | 0.17 |
Redrawn from Lesson 2 deck slide 12 (“Same data analyzed using effect coding and dummy coding”). Data from a perfectly balanced experiment. Under effect coding every standard error is 0.19; under dummy coding the standard errors grow with the order of the effect (0.74 first order, 1.05 second order, 1.49 third order). Note PEER: non-significant positive under effect coding, significant negative under dummy coding.
Knowledge check
MULTIPLE CHOICE: Is the following statement true or false? Effects produced by factorial analysis of variance using dummy coding generally must be interpreted differently from effects produced by factorial analysis of variance using effect coding.
Lesson 3: More About the Economy of the 2^k Factorial Experiment
The lesson explains why a factorial experiment can be well powered with small per-condition sample sizes and why adding factors often does not require extra participants. It begins with a brief review of statistical power and of how ERCTs are powered.
Review: Type I error, Type II error, power: “The definition of a type one error is rejecting the null hypothesis given that it is true. In other words, mistakenly concluding an effect exists when it does not.” “The definition of a type two error is failing to reject the null hypothesis given that it is false. In other words, mistakenly overlooking an effect.” “Statistical power is 1 minus beta or the probability of correctly rejecting the null hypothesis given that the null hypothesis is false.”Narration, Lesson 3; deck slides 4 to 5
Review: what power depends on: “All else being equal, power depends on the size of the effect to be determined.” “The type one error rate selected. A larger type one error rate provides more power.” “and the sample size all else being equal. The larger the sample size, the more statistical power.” “When there is a cluster structure, the intraclass correlation may also be a consideration.”Narration, Lesson 3; deck slide 6
How ERCTs are powered: “Suppose this experiment is sufficiently powered with a total sample size of 300 or 150 per condition.” Adding a third arm without adding participants loses power: “Will I have the same statistical power as before? Well, you know the answer to this rhetorical question. I will not have the same power as before.” “To maintain power, it will be necessary to obtain at least 150 more participants.” “Because to maintain the same power, each treatment control comparison must be based on 300 participants.”Narration, Lesson 3; deck slides 7 to 9
The factorial experiment is more efficient: “the factorial experiment requires many fewer experimental participants to maintain the same level of statistical power. In this case, 352 as compared to nearly 900 for the comparative treatment approach and over 1,400 for individual experiments.” The four-component example is compared in Figure 3.Narration, Lesson 3; deck slide 11
Small per-condition n is not a sign of low power: “In this experiment, there are only 22 participants per condition and yet power is at least8” (caption artefact: “at least8” for at least .8). “With an ERCT, a small per condition n is usually an indication that the experiment has low, possibly unacceptable power. By contrast, in a balanced factorial experiment, a small perition n does not necessarily indicate low power” (caption artefact: “perition” for per-condition).Narration, Lesson 3; deck slides 12 to 13
The logic behind powering factorial experiments: “In factorial experiments, per condition n does not determine power. What matters is the n per level of each factor. This n is going to be based on an aggregation of experimental conditions.” Lowercase n is sample size per experimental condition; uppercase N is used otherwise.Narration, Lesson 3; deck slide 14
Main effects aggregate half the sample per level: “the main effect of MI is the mean of conditions 9- 16 minus the mean of conditions 1 through 8.” Adding up, “the sample size for the yes level of MI is 176. Similarly, you will find that the aggregate sample size for the no level of MI is 176. In other words, the n per level is 176.” The PEER main effect “is computed by subtracting the mean of conditions 1 through 4 and 9 through 12 from the mean of conditions 5- 8 and 13- 16.” Again, the n per level is 176. The same logic holds for TEXT and MIND.Narration, Lesson 3; deck slides 15 to 16
Balance is what makes this work: “each level of every factor appears the same number of times at each level of every other factor” (the first and most important requirement for perfect balance). “Balance is what makes factorial experiments so efficient, particularly the 2 to the K experiment.” “The second requirement for perfect balance is equal sample sizes across experimental conditions.” Approximate balance is acceptable in practice: “it is a good idea to come as close to perfect balance as you can.”Narration, Lesson 3; deck slides 17 to 18
Adding a factor does not add participants: adding behavioral skills training (low/high) makes the example a 2^5: “The new experiment has double the number of experimental conditions. Previously we had 16 conditions. Now we have 32.” “It may surprise you to learn you will not need any additional participants. We can take the original 352 participants and divide them equally among the 32 conditions which results in 11 per condition. This experiment has the same amount of statistical power.”Narration, Lesson 3; deck slides 19 to 21
Why the power is unchanged: “There are still 176 participants in each level of MI. So the power for testing this main effect has not changed.” For the new factor, “The main effect of skills is the mean of the evenumbered conditions minus the mean of the oddnumbered conditions” (caption artefacts: “evenumbered”/“oddnumbered” for even-/odd-numbered). “the n for the high level of skills is 176 and the n for the low level of skills is also 176.”Narration, Lesson 3; deck slides 22 to 23
Please remember this: “adding one or more factors to a factorial experiment will increase the number of conditions in the experiment. But it is very often possible to add one or more factors to a factorial experiment without the need to increase the overall sample size.” “statistical power in a factorial experiment does not depend on the per condition sample size. Instead, it depends on the sample size per level of a factor.”Narration, Lesson 3; deck slide 24
More factors are often worth it: “it can be inefficient to conduct a factorial optimization trial with only two or three factors.” Still, “factors should never be added just for the sake of having more factors. If the research questions at hand can be addressed adequately with only two or three factors, then that's what is called for.” “often the greatest scientific yield comes from a factorial experiment with more factors.”Narration, Lesson 3; deck slide 25
The caveat about multiple testing: the claim that n = 11 per condition has as much power as the n = 22 design “assumes no correction for example a boneroni correction to maintain the type one error rate across multiple hypothesis tests” (caption artefact: “boneroni” for Bonferroni). “Whether or not to apply such a correction in an optimization trial is controversial. In our view, when the results of an optimization trial are being used to select components and component levels for inclusion in the optimized intervention, corrections like the boneroni are unnecessary.”Narration, Lesson 3; deck slide 28
Counterintuitive but true: “Factorial experiments can have very small percondition ends and yet maintain a high degree of statistical power. This is because power is driven by the n per level of a factor. This is a unique characteristic of the factorial experiment family” (caption artefact: “percondition ends” for per-condition n's). “An ERCT with n= 11 per condition would be extremely unlikely to be sufficiently powered. Yet, depending on circumstances, factorial experiments can have per condition ends as low as two or three and still be sufficiently powered, provided that the N per level is large enough.”Narration, Lesson 3; deck slides 29 to 30
Figure 3: Design options for the four-component HIV example (redrawn)
| Design | Number of experimental conditions | Participants needed to maintain power (alpha = .8, d = .3) | Can interactions be estimated? |
|---|
| Individual experiments | 8 | 1,408 | No |
| Comparative treatment | 5 | 880 | No |
| Factorial (main effect) | 16 | 352 | Yes |
Redrawn from Lesson 3 deck slide 11 (“Choosing an experimental design: Comparison of options”). The factorial experiment estimates interactions as well as main effects with far fewer participants. With N = 352 the per-condition n is 22, and each factor level is aggregated across 8 conditions, giving n per level of 176.Figure 4: Main effect of MI in the 2^4 example aggregates half the conditions (redrawn)
| Conditions | MI level | Contribution to MI main effect | Per-condition n | Aggregated n per level |
|---|
| 1 to 8 | No | minus side of the difference | 22 | 176 |
| 9 to 16 | Yes | plus side of the difference | 22 | 176 |
Redrawn from Lesson 3 deck slides 12 and 15. Main effect of MI = mean of conditions 9 to 16 minus mean of conditions 1 to 8. Each level of MI contains 8 x 22 = 176 participants, so power for the main effect is based on 176 per level, not 22 per condition.
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? There may be more than one.
- ☐ A factorial experiment with a small per-condition sample size is always underpowered.
- ☑ It is sometimes possible to add one or more factors to a factorial experiment without the need to increase the sample size to maintain statistical power.
- ☑ Power in a factorial analysis of variance is more closely related to the sample size per level of a factor than it is to the sample size per experimental condition.
- ☐ It is best to stick to factorial experiments with 2 or 3 factors; with more factors the experiments require massive numbers of participants.
Feedback when incorrect (source text, retained as provided): Answer A. is incorrect. A factorial experiment with a small per-condition sample size is not necessarily underpowered.
Answer D. is incorrect. The number of factors in a factorial experiment often has little to no effect on power, making experiments with more factors potentially more efficient.
Lesson 4: The Resource Demands of Factorial Experiments Involving More Than Two Levels in a Factor
The lesson contrasts adding factors (often free) with adding levels to a factor (costly), working through the 2 x 2 x 3 x 2 example in which TEXT gains a third level (No, Low, High).
Adding a level instead of a factor: “what would happen if instead of adding a factor to this four factor experiment, you wanted to add a level to a factor.” “suppose instead of text having levels no, and yes, it would have three levels, no, low, and high.”Narration, Lesson 4; deck slide 8
The design becomes 2 x 2 x 3 x 2: “This experiment is now a 2x 2x 3x 2 or a 23x3. This means there are 23r* 3 experimental conditions which makes 24 conditions” (caption artefacts: “23x3” for 2^3 x 3, “23r* 3” for 2^3 x 3). With four factors, MI, PEER and MIND at two levels and TEXT at three, there are 24 experimental conditions, and “at each level of each other factor, three levels of text appear. No, low and high.”Narration, Lesson 4; deck slides 9 to 10
No added participants: “The 352 participants are now divided among 24 conditions. This creates a slight variability in per condition n, but this is unimportant for our purposes here.”Narration, Lesson 4; deck slide 11
The MI main effect is unaffected: “This is now the mean of conditions 13- 24 minus the mean of conditions 1 through 12. Each level of MI still has 176 participants, so power remains the same. The addition of a third level to text hasn't changed power for detection of the MI main effect.”Narration, Lesson 4; deck slide 12
TEXT now has two main effects: “Now because there are three levels there are two comparisons. So there are two main effects. Let's say the desired comparisons are high versus no and low versus no.”Narration, Lesson 4; deck slide 13
High versus No loses power: “This main effect is the mean of conditions 5 6 11 12 17 18 23 and 24 minus the mean of conditions 1 2 7 8 13 14 19 and 20. The other conditions do contribute to the estimate of the error term but otherwise do not contribute to this main effect estimate. Thus, all else being equal, power will be reduced.” Only 8 conditions contribute on each side, so the per-level N drops from 176 to about 120.Narration, Lesson 4; deck slide 13
Restoring power costs participants: “it would be necessary to add about 176 more participants. This is an increase of 50%.”Narration, Lesson 4; deck slides 15 to 16
Low versus No is likely worse: “If no participants are added, this is also based on a perle sample size of n= 120” (caption artefact: “perle” for per-level). “power is likely to be reduced even further because the effect size for the low versus no comparison is probably smaller than the effect size for the high versus no comparison.” “restoring power to the original level would require even more than an additional n= 176 participants. How many more? This depends on the anticipated effect size.”Narration, Lesson 4; deck slides 17 to 18
Two conclusions: “it is often possible to add one or more factors to a factorial experiment and maintain power without adding participants. So adding one or more factors to a factorial optimization trial is often an efficient strategy.” “adding a level of a factor usually requires at least 50% more participants. So adding even one level to even one factor can be costly.”Narration, Lesson 4; deck slides 19 to 20
If you are considering more than two levels: “the 2 to the k factorial experiment is generally the most efficient.” “Addressing questions involving comparison of more than two levels tends to be resource intensive.” “the resource management principle says you have to ask yourself how important is the question about intermediate levels.” “Resources you spend here cannot be spent to address other questions that may be equally or more important.”Narration, Lesson 4; deck slide 21
A recommended strategy for dose questions: “Start by using two levels to ascertain that there is a difference between high and no. Or if you expect a curve linear dose response curve, compare no to whatever dose is hypothesized to produce the largest or most favorable effect. Then if there is a difference, examine intermediate levels in a subsequent experiment” (caption artefact: “curve linear” for curvilinear). “If there is no difference between high versus no, do you care about the intermediate levels?” Dose-response questions are returned to later in the course.Narration, Lesson 4; deck slide 22
Figure 5: Condition sets used for the High versus No TEXT comparison (redrawn)
| Side of the comparison | Conditions included | Per-level N |
|---|
| High level of TEXT (plus) | 5, 6, 11, 12, 17, 18, 23, 24 | 8 x ~15 = about 120 |
| No level of TEXT (minus) | 1, 2, 7, 8, 13, 14, 19, 20 | 8 x ~15 = about 120 |
| Remaining conditions | 3, 4, 9, 10, 15, 16, 21, 22 | contribute to the error term only |
Redrawn from Lesson 4 deck slide 13. Because the three-level TEXT factor is split into two comparisons, each comparison is based on only 8 of the 24 conditions on each side, reducing the per-level N from 176 to about 120 and lowering power.
Knowledge check
MULTIPLE CHOICE: Is the following statement true or false? In a factorial experiment, considerations such as the number of factors and the number of levels per factor have little to no effect on the number of participants needed to maintain a desired level of power.
Feedback when incorrect (source text, retained as provided): The number of factors often has little to no effect on sample size requirements to maintain power, but the number of levels per factor can have a huge effect, with more levels beyond 2 often requiring greatly increased sample sizes.
Lesson 5: Experimental Control in a Factorial Design
The lesson explains how experimental control works in a factorial experiment, where each factor has its own control level rather than the experiment having a single control condition.
Control in an ERCT: “In an ERCT, there is usually a single experimental condition designated as the control.” “In the control arm, treatment A and treatment B are set to off. The control participants may get some other treatment depending on the circumstances, but the treatments under examination are set to off.”Narration, Lesson 5; deck slide 4
Condition 1 is not THE control condition: “In condition one, all the factors are set to no. This looks like a control condition in an ERCT. But is condition one the control condition? Condition one is not considered the control condition in a factorial experiment. Each factor has its own control.”Narration, Lesson 5; deck slides 5 to 6
Each factor has its own control: “This concept of experimental control is very different from control in an ERCT. In a factorial experiment, one level of each factor serves as its control.” “for the MI factor conditions 1 through 8 in which MI is set to no serve as the control for MI. For the mind factor the oddnumbered conditions serve as the control” (caption artefact: “oddnumbered” for odd-numbered).Narration, Lesson 5; deck slide 6
Different factors can use different types of control: “Suppose an attention placebo is to be used as a control for motivational interviewing. Texts containing information about diet are to be used as a control for text message support and a peaceful nature video is to be used as a control for mindfulness meditation. This can readily be incorporated into the experimental design.”Narration, Lesson 5; deck slide 7
Who is in a control condition?: “Which participants are in a control condition? Arguably all of them except for those in condition 16. Most of the participants are in a treatment condition for some factors and a control condition for others.” “those assigned to condition four are in the control level of MI and peer and the experimental level of text and mind.” “Those assigned to condition 14 are in the control level of text and the experimental level of the remaining factors.” Only Condition 16 (all factors at their experimental level) puts a participant in no control level at all.Narration, Lesson 5; deck slides 9 to 11
Figure 6: The 2^4 factorial with factor-specific controls (redrawn)
| Condition | MI | PEER | TEXT | MIND |
|---|
| 1 | Attention placebo | No | Info about diet | Nature video |
| 2 | Attention placebo | No | Info about diet | Yes |
| 3 | Attention placebo | No | Yes | Nature video |
| 4 | Attention placebo | No | Yes | Yes |
| 5 | Attention placebo | Yes | Info about diet | Nature video |
| 6 | Attention placebo | Yes | Info about diet | Yes |
| 7 | Attention placebo | Yes | Yes | Nature video |
| 8 | Attention placebo | Yes | Yes | Yes |
| 9 | Yes | No | Info about diet | Nature video |
| 10 | Yes | No | Info about diet | Yes |
| 11 | Yes | No | Yes | Nature video |
| 12 | Yes | No | Yes | Yes |
| 13 | Yes | Yes | Info about diet | Nature video |
| 14 | Yes | Yes | Info about diet | Yes |
| 15 | Yes | Yes | Yes | Nature video |
| 16 | Yes | Yes | Yes | Yes |
Redrawn from Lesson 5 deck slide 7. Each factor has its own control procedure: attention placebo for MI, No for PEER, diet-information texts for TEXT, nature video for MIND. Only Condition 16 is at the experimental level of every factor; all other participants are in the control level of at least one factor.
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? There may be more than one.
- ☐ Every factorial experiment has a single experimental condition designated as the control.
- ☐ If different factors in a factorial experiment require different types of controls, it is usually advisable to break the experiment apart into several distinct experiments.
- ☑ In a factorial experiment, a single participant may be in the control group with respect to one factor and in the treatment group with respect to another.
- ☑ It is acceptable for different factors in a factorial experiment to have different types of control.
Feedback when incorrect (source text, retained as provided): Answer A. is incorrect. Factorial experiments do not have a single experimental condition designated as the control condition.
Answer B. is incorrect. Because each factor has its own control, it is usually not necessary to break the experiment apart into distinct experiments to maintain good experimental control.
Synthesis (my own reading: not part of the course)
Read methodologically, this module converts the factorial experiment from a design idea into working machinery: coding rules that make estimates interpretable, a power arithmetic that makes the design efficient, and a control concept that makes it coherent. The pivotal move is the redefinition of the estimand. An ERCT compares conditions that differ in one component, so its power budget is one comparison per component and adding a treatment arm forces new participants. A factorial experiment estimates a main effect as an average over half the conditions, so every participant contributes to every main effect: with N = 352 split over 16 conditions (n = 22), each factor level aggregates 176 participants, which is why power is comparable to a t-test on the full N and why adding a fifth factor merely re-slices the same participants into 32 conditions of 11.
Three technical points are easy to misread and worth fixing in your own notes. First, 2^k denotes k factors each at two levels, not a 2-by-k design; the narration is explicit that 2^3 and 2 x 3 are different designs, and the caption artefacts (“23x3” for 2^3 x 3) make the point in Lesson 4. Second, under effect coding each regression coefficient equals half the corresponding main effect or interaction (b1 = main effect of MI / 2, b4 = MI-by-PEER interaction / 2), while dummy-coded coefficients are simple effects at the reference levels of the other factors; the omnibus F is the same under both codings, but the individual hypothesis tests are not. Third, the efficiency claim has a boundary: adding factors is often free because the per-level N is unchanged, whereas adding a level to a factor halves the per-comparison N (176 to about 120 in the 2 x 2 x 3 x 2 example) and typically costs at least 50% more participants to restore power; the course's practical answer is to study dose-response questions with two levels first and examine intermediate levels only after a High-versus-No difference is established.
One caution for readers of the course materials: the Lesson 2 knowledge check marks both “True” and “False” as correct for a single multiple-choice statement, which cannot be a coherent answer key; the notes reproduce the marks exactly and flag the anomaly rather than choosing an intended answer. Caption transcription artefacts (Corsera, perition, boneroni, fstistic, Cougler, Fiser, pier, Zcore, at least8, 23x3, perle, evenumbered/oddnumbered, 74 for .74, “Guastafello”/“Guastafferro”/“Guapero” for Guastaferro) are preserved inside quotes and flagged. No answer-level “This is incorrect” feedback anomalies of the kind found in Modules 1 to 2 occur in this module's knowledge checks.
Glossary
- Effect coding
- Coding scheme for a 2^k factorial experiment that represents the two levels of each factor as −1 and +1 (no and yes), consistent with the course's definitions of main effects and interactions. Also called contrast or deviation coding in other texts.
- Dummy coding
- Coding scheme (also called reference category coding) that represents the two levels of each factor as 0 and 1, where 0 is the reference level; higher-order effect vectors are products of the first-order vectors. Coefficients are conditional (simple) effects at the reference levels of the other factors.
- Main effect
- The mean of the conditions at one level of a factor minus the mean of the conditions at the other level; under effect coding it is interpreted as the factor's effect averaged across all combinations of levels of the other factors.
- Interaction
- The extent to which the effect of one factor differs across levels of another; the order of an interaction is the number of factors involved (two-way, three-way, …).
- First-order / second-order / third-order effect
- The terms used under dummy coding instead of main effect and interaction, because dummy-coded estimates do not correspond to the usual main effect and interaction definitions.
- Omnibus F
- The overall model F test in a factorial ANOVA; identical under effect coding and dummy coding for the same data and model.
- Standard error
- In a perfectly balanced 2^k factorial experiment with effect coding, identical for all effect estimates; under dummy coding, larger for higher-order effects.
- Statistical power
- The probability of correctly rejecting the null hypothesis given that it is false; power = 1 − beta.
- Type I error / alpha
- Rejecting the null hypothesis given that it is true; alpha is the Type I error rate.
- Type II error / beta
- Failing to reject the null hypothesis given that it is false; beta is the Type II error rate.
- N per level
- The number of participants aggregated across all experimental conditions at one level of a factor; the quantity that drives power for a main effect in a factorial experiment.
- Balance (structure)
- Each level of every factor appears the same number of times at each level of every other factor; an absolute requirement.
- Balance (sample size)
- Equal sample sizes across experimental conditions; required for perfect balance, approximated in practice.
- Control condition (ERCT)
- A single experimental condition designated as the control, with the treatments under examination set to off.
- Control level (factorial)
- One level of each factor serves as that factor's control; different factors can use different types of control (for example attention placebo, diet-information texts, nature video).
- Attention placebo
- A control procedure matched to an intervention's attention or contact, used here as the control for motivational interviewing.
- Resource management principle
- The principle that the investigator must decide how important a research question (for example, about intermediate levels of a factor) is before committing additional resources, considering opportunity costs.
- Dose-response question
- A question about the effect of different amounts or levels of a component; the course recommends two-level comparisons first, with intermediate levels examined in a subsequent experiment.
Sources
- Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Set reading: Chapter 3 for all of Module 3.)
- Kugler, K. C., Dziak, J. J., & Trail, J. (2018). Coding and interpretation of effects in analysis of data from a factorial experiment. In L. M. Collins & K. C. Kugler (Eds.), Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer. (Cited on the Lesson 2 deck slide 22 as the deep-dive reference on effect versus dummy coding.)
- Course materials: CADIO, MOST from a Methodological Perspective, Module 3 (“Some Conceptual and Technical Aspects of the Factorial Experiment”), share link https://share.articulate.com/n28XvskSB1b5squx2_nq3, five video lessons and five slide decks.
Video list
Prepared 10 Sep 2026. Notes paraphrase the narration unless marked as verbatim quotes; figures are redrawn from the course slides; quotes follow the YouTube caption transcripts (durations are the last caption-panel timestamps, approximate). Caption transcription artefacts are preserved inside quotes and flagged. The course's five knowledge checks are reproduced with the course answer key; the Lesson 2 item marking both True and False correct is a flagged source anomaly. No separate quiz distinct from these items appears in the course.