Study notes · CADIO / NYU School of Global Public Health
Module 2: “The Factorial Optimization Trial and the Factorial Analysis of Variance (ANOVA)” Authors: Linda M. Collins and Kate Guastaferro (School of Global Public Health, New York University) Format: 7 lessons (Lesson 1, set reading, Lessons 2 to 6) · 6 videos (91 min) · 6 slide decks (PDF) · 5 knowledge checks Set reading: Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 3, plus the additional readings listed under the set reading lesson) Access: free via CADIO share links, or Coursera
Source inconsistency: course marks these options correct but attaches unrelated EASE feedback. Correctness marks follow course answer key; feedback is retained as a source anomaly.
Source anomalies found in Module 2 (flagged, not silently corrected). (1) In the Lesson 2 knowledge check the course marks three options correct for a single-answer question (“Which one of the following best states the resource management principle?”); the answer-level feedback attached to the second and third marked-correct options says they are incorrect. The answer key marks are reproduced exactly as in course.json; the contradictory feedback is retained, labelled as a source anomaly and not endorsed. (2) The question-level “feedback when incorrect” for the knowledge checks in Lessons 1, 2 and 3 is the gift-card text copied from a different item; it is retained verbatim, labelled and not endorsed. (3) The Lesson 1 slide and narration define effectiveness as “the extent to which the intervention does more harm than good under real world conditions”, inverting the standard definition (“more good than harm”, Flay 1986); the quote is preserved and flagged. (4) Caption transcription artefacts (for example “Gusteferro”, “Fischer”, “pier”, “defect size”, “equal ends”, “Corsera”, “multi-phase”) are preserved inside narration quotes and noted where they occur.
The module's argument in one paragraph.
Module 1 ended with the claim that MOST is not any particular experimental design; this module delivers the first piece of the machinery that makes the factorial experiment the workhorse optimisation trial design. The argument runs in three steps. First, the research questions of an optimisation trial, which components work, which interact, for whom and under what circumstances, are different from the single question an ERCT answers, and different questions call for different designs (Lesson 2). Second, the factorial experiment answers those questions by defining the effect of a component as its main effect: the average difference between all conditions at the higher and lower levels of that factor, so that every participant contributes to every estimate, and participants are effectively recycled; interactions express whether a component's effect is augmented or reduced in the presence of another (Lessons 3 to 4). Third, the analysis follows from the design: factorial ANOVA is a regression model with effect (−1,1) coding, in which each regression coefficient equals half the corresponding main effect or interaction, and in a perfectly balanced 2^k experiment the estimates of all main effects and interactions are uncorrelated with identical standard errors, so power is uniform across coefficients (Lessons 5 to 6). The through-line is efficiency: for the four-component HIV example, the 2^4 factorial needs 352 participants where four individual experiments need 1,408 and a comparative treatment experiment needs 880, and only the factorial can estimate interactions at all.
Lesson 1: The Preparation Phase and Specifying the Optimization Objective
The lesson reviews the preparation phase and shows how the optimization objective makes the definition of intervention EASE concrete. The running example throughout the module is the intervention to reduce viral load among HIV positive individuals who drink heavily, with five candidate components: motivational interviewing (no/yes), peer mentoring (no/yes), text message support (no/yes), mindfulness meditation (no/yes), and behavioral skills training (low/high intensity).
Purpose of the preparation phase: “The purpose of the preparation phase is to lay the groundwork for optimization.” Activities: deriving a detailed conceptual model (or revising an existing one), identifying a set of candidate intervention components, conducting any necessary pilot work, and identifying an optimization objective. The course is aimed at methodologists and offers only an abbreviated treatment of the preparation phase; a different Coursera course covers it in full.Narration, Lesson 1; slides 2.3 to 2.6
Conceptual model: “A carefully constructed and highly detailed conceptual model is the basis for every intervention that is optimized using MOST.” In the example, the model shows the behavioural factors leading to HIV viral load suppression: alcohol use (positive arrow to HIV viral load) and ART adherence (negative arrow) are the proximal causes; intentions to reduce alcohol use and to maintain ART adherence sit further upstream, and health beliefs, positive role models, social support, mental health status and behavioural skills are still more upstream. Each candidate component targets one causal factor: motivational interviewing targets health beliefs; peer mentoring provides positive role models; text message support provides a social support system; mindfulness meditation helps improve mental health; behavioral skills training improves behavioural skills.Narration, Lesson 1; slides 2.7 to 2.8 (conceptual model adapted from Collins, Kugler & Gwadz, 2016)
The conceptual model is not a statistical model: “I want to emphasize that this figure is not a structural equation model. It's not intended to represent how you are going to analyze the data. Instead, it is intended to express the role that each candidate component has in the intervention.” It expresses why each candidate component is under consideration; peer mentoring, for example, is included to offer a positive role model and model intentions, even though it might conceivably also improve behavioural skills.Narration, Lesson 1
Optimization objective: “The optimization objective expresses your definition of intervention EASE.” “It's your optimization objective, the investigator selects it.” The objective expresses the desired balance of intervention effectiveness against affordability, scalability and efficiency. The term optimization criterion is sometimes used instead: “the term optimization criterion is sometimes used instead of optimization objective. The terms are interchangeable.”Narration, Lesson 1; slides 2.16 to 2.18
Review of the four desiderata (with a flagged source error): effectiveness is “the extent to which the intervention does more harm than good under real world conditions”, the slide prints the same wording, which inverts the standard Flay (1986) definition (more good than harm); the inversion is a course error, flagged and not silently corrected. Affordability, which some publications call economy, is the extent to which the intervention is effective without exceeding budgetary constraints; scalability is the extent to which the intervention can be implemented widely with fidelity; efficiency is the extent to which the intervention avoids wasting time, money or other valuable resources. Intervention EASE is a strategic balance of the four desiderata: in the narration's words, “it is achieve by balancing effectiveness against affordability, scalability, and efficiency” (caption transcription drops the 'd' of 'achieved').Narration, Lesson 1; slides 2.11 to 2.15
Resources are implementation resources, not trial resources: “I want to clarify that here we are talking about resources required to implement the intervention, not resources required to conduct the optimization trial.” There will almost always be limits on money, staff time, participant time or other resources; or there may be no upper limit but a wish to avoid wasting resources. A full economic cost analysis may not be necessary; the investigator needs a sense of the key constraints and data on what each set of components costs in whatever terms cost is defined (for example money or staff time), noting economies of scale when components are delivered together. An upper limit may reflect “how much is whoever is going to pay for the intervention willing to pay for it” (willingness to pay).Narration, Lesson 1; slides 2.19 to 2.22
Examples of possible optimization objectives: (a) “I want an intervention made up of all active components.”the “all active components” objective; cost is not an explicit consideration, so it identifies an efficient intervention. (b) “I want to select the components that will provide the most effective intervention that can be implemented for less than, (say), $300 per person.”affordability, with “affordable” defined by the investigator. (c) “I want to select the components that will provide the most effective intervention that can be completed in less than 1 hour of staff time.”, scalability, at least in this respect, because an intervention taking more time would probably be subjected to ad hoc modifications. (d) “I want to select the set of components that represents the best value.”, a judgment call weighing performance against cost. (e) “Suppose you want to select the set of components that represents the best value without exceeding an implementation cost of say, $300 per person.”, a budget ceiling plus a value criterion; the best-performing intervention under the ceiling may not be the best value. (f) A time-efficiency objective: the intervention that delivers the best performance in relation to the amount of time it requires (staff time, participant time, or both).Narration, Lesson 1; slides 2.23 to 2.28
Why specify the optimization objective in the preparation phase: “Because this will help ensure that you collect the data you need during the optimization trial.” For example, if the objective is to complete the intervention in three hours or less, the trial must collect data on how long each combination of components takes; otherwise the objective cannot be used after the trial. Cost data and pilot testing of data collection procedures may be needed.Narration, Lesson 1; slide 2.34
Can the optimization objective be changed later? “Can you change the optimization objective later? Yes, you can.” Different optimization objectives can be used for different settings: a clinic able to afford $300 per person and a clinic able to afford $200 per person would optimise with different objectives, and the components selected for the two clinics “will not necessarily be a subset of the components selected using the $300 per person optimization objective.”Narration, Lesson 1; slides 2.35 to 2.36
The role of the optimization objective in decision-making: if cost is not a consideration, all components that showed substantial effects in the optimization trial are included; if cost is a consideration, “it is possible that only a subset of the four components that demonstrated effects in the optimization trial will ultimately be included in the optimized intervention.” Omitting one or more effective components may be necessary to meet the objective.Narration, Lesson 1; slides 2.31 to 2.33
Immediate scalability: “By our definition, a scalable intervention is implementable in the intended setting exactly as evaluated.” That is, no ad hoc modifications are needed to make it more affordable, more straightforward to implement, or less burdensome on staff or participants. During the preparation phase, identify stakeholders and key constraints on implementation; then MOST can be used to optimise an intervention that operates within them. “The idea is to develop an intervention with implementation in mind from the beginning, with the objective of achieving not only effectiveness but immediate scalability.”Narration, Lesson 1; slides 2.37 to 2.39
Figure 1: Conceptual model for the running HIV example (redrawn)
Redrawn from the Lesson 1 slide deck (slide 2.7, adapted from Collins, Kugler & Gwadz, 2016): the dashed box links each candidate component to the causal factor it targets; the figure is a conceptual model, not a structural equation model.
Knowledge check
MULTIPLE CHOICE: Which of the following could be used as an optimization objective? My optimization objective is to…
☐ …identify a combination of components that will show a statistically significant effect in an ERCT. (This is incorrect. This objective does not balance effectiveness, affordability, scalability, and efficiency.)
☑ …identify the combination of components that produces the best expected outcome for implementation costs of less than $500 per person. (That's correct!)
☐ …identify the combination of components that takes the least amount of time to implement. (This is incorrect. This objective does not balance effectiveness, affordability, scalability, and efficiency.)
☐ …conserve research resources by selecting a highly efficient optimization trial design. (This is incorrect. Although it is important to select a highly efficient optimization trial design, optimization is defined in terms of intervention implementation resources, not research resources.)
Feedback when incorrect (source text; source anomaly, not endorsed): This is incorrect. A gift card provided to incentivize completion of research surveys is included for research purposes only, and will not be a part of the intervention when it goes to scale. Therefore, it cannot be considered an intervention component.
Reading for All of Module 2
Set reading and supplementary sources for the module:
Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 3.)
Collins, L. M., Kugler, K. C., & Gwadz, M. V. (2016). Optimization of multicomponent behavioral and biobehavioral interventions for the prevention and treatment of HIV/AIDS. AIDS and Behavior, 20, 197 to 214.
Kugler, K. C., Dziak, J. J., & Trail, J. (2018). Coding and interpretation of effects in analysis of data from a factorial experiment. In L. M. Collins & K. C. Kugler (Eds.), Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer.
Fisher, R. A. (1971). The design of experiments. New York, NY: Hafner Publishing.
This lesson is a text reading list, not a video lesson; none of the material above is quoted against transcripts in this page.
Lesson 2: Conceptual Differences Between the Factorial Experiment and the ERCT
The lesson contrasts the research questions addressed by ERCTs and optimisation trials and grounds design selection in the resource management principle.
Why conduct an ERCT: “The objective of an ERCT is assessment of whether the intervention as a package outperforms a suitable control or comparison.” When the question is whether an intervention works as a package, “an experimental design in the ERCT family is the most appropriate.”Narration, Lesson 2; slides 2.5 to 2.6
Why conduct an optimization trial: “Here the objective is efficient assessment of the performance of individual intervention components.” The investigator may want to “weed out any underperforming components, including iatrogenic components”, get a sense of the magnitude of each component's effect, or examine whether a component's effect is augmented or reduced in the presence of another; “an experimental design not in the ERCT family is needed to address these kinds of questions.”Narration, Lesson 2; slides 2.7 to 2.8
Different questions, different designs: “Different research questions call for different experimental designs.” “It isn't possible to address every research question with the same experimental design, even one as versatile as the RCT.”Narration, Lesson 2; slide 2.9
What an optimization trial is good for: “An optimization trial is used to help build knowledge about which components work, which components interact with each other, and which components work for whom and under what circumstances.” It is less helpful for assessing overall effectiveness of a package and for whom the package works.Narration, Lesson 2; slide 2.9
What an evaluation trial is good for: “By contrast, an evaluation trial helps build knowledge about the overall effectiveness of an intervention as a package, and figuring out for whom the package works and under what circumstances.” It is less helpful for assessing individual components and interactions.Narration, Lesson 2; slide 2.9
Example questions each trial addresses: optimisation trial: “what is the effect of the motivational interviewing component on average?” and “Does peer mentoring have a larger effect when text message support is included?” Evaluation trial: “what is the effect of an intervention package made up of motivational interviewing, peer mentoring, text message support, mindfulness meditation, and a high intensity level of behavioral skills training?”Narration, Lesson 2; slide 2.10
The motivation for an optimization trial: “The information gathered by means of an optimization trial is used to select components and component levels to optimize the intervention and thereby achieve intervention EASE.” EASE is a strategic balance of the desiderata, achieved by balancing effectiveness against affordability, scalability and efficiency.Narration, Lesson 2; slides 2.11 to 2.12
Choice of design is critical: “Any experimental design is a possibility.” “The only requirement is that it must be selected based on the resource management principle.” “This means that out of the alternatives, the selected design best addresses the key research questions while making the best use of research resources.”Narration, Lesson 2; slide 2.13
Review of the resource management principle: the investigator using MOST “must strive to make the best and most efficient use of available resources when obtaining scientific information”; the most efficient design is the one that “best manages research resources, we will be discussing this soon, and most directly addresses the primary research questions.”Narration, Lesson 2; slides 2.14 to 2.15
Strategic resource management: instead of always selecting an ERCT, “manage the research resources strategically so as to gain the most information, gain the most reliable information, and move intervention science forward fastest.” Decide what information is most important and target resources there; choose a design appropriate to the type of intervention, with efficiency in mind, starting from the resources available. “By definition, MOST does not require an increase in research resources, but in most cases, it will require a realignment of research resources.”Narration, Lesson 2; slides 2.16 to 2.19
The 32-arm ERCT thought experiment: with five two-level components there are 2^5 or 32 possible combinations. “If the question is which of these combinations has the largest effect, a 32-arm ERCT would, at least in theory, provide a definitive answer.” In practice it would not be feasible to power, and it would not address questions about individual components or interactions.Narration, Lesson 2; slide 2.16
Decisions for candidate components 1 to 4 (no/yes): “For candidate components 1-4, the decisions would be, does a component have a detectable effect?” If the yes level outperforms the no level, inclusion is considered (possibly reconsidered against cost and time, depending on the optimization objective); if no effect is detected, the component is not considered further.Narration, Lesson 2; slide 2.22
Decisions for the behavioral skills training component: “One decision that needs to be made is, does the high intensity level outperform the low intensity level?” If it does, high intensity is considered (subject to the same cost and time reconsideration); if not, the low intensity level would probably be selected.Narration, Lesson 2; slide 2.23
You are always selecting a component level: “Note that you are always selecting a component level when the components are no and yes, selecting no means the component will not be included.” “When the levels are not no and yes, but instead reflect different intensities, doses, types, and so on, the component is going to be included no matter what the outcome of the optimization trial is.”Narration, Lesson 2; slides 2.22 to 2.23
What the trial must tell us: “For each component, we want to determine whether there is a difference between the higher and lower levels, and, ideally, we also wish to determine whether any of the components interact.” This information feeds the decisions about which components and levels go into the intervention package.Narration, Lesson 2; slide 2.24
The resource management principle (formal statement): “The resource management principle says that the investigator must carefully choose an experimental design so as to gather the information needed while making the most of, but not exceeding the available resources.” The experimenter needs a clearly specified set of research questions, knowledge of available resources, and knowledge of the resources each candidate design requires.Narration, Lesson 2; slides 2.25 to 2.28
Optimization trial designs: “In this course, we are focusing on the factorial design, which has a long history in agriculture, industrial engineering, and related fields.” “Other optimization trial designs like the fractional factorial design, the sequential multiple assignment randomized trial, or SMART, and the micro-randomized trial, MRT, are also members of the general family of factorial experimental designs.” SMART and MRT are used in optimisation of adaptive interventions; system identification comes from control engineering and is another approach to adaptive interventions (see the Rivera et al. chapter cited at the end of the lesson).Narration, Lesson 2; slides 2.29 to 2.30
Knowledge check
MULTIPLE CHOICE: Which one of the following best states the resource management principle?
☑ The investigator must choose an experimental design so as to gather the information needed while making the most of the available resources.
☑ An investigator is better off staying with one type of experimental design for all research. (This is incorrect. Different experimental designs may be appropriate for different research questions.)Original course feedback (source anomaly, not endorsed):
☑ Resources should be devoted to research management. (This is incorrect. Resources should be devoted to research management, but this is not what the resource management principle states.)Original course feedback (source anomaly, not endorsed):
☐ Different experimental designs all have about the same level of efficiency. (This is incorrect. Different experimental designs can vary greatly in efficiency with respect to a particular research question.)
Feedback when incorrect (source text; source anomaly, not endorsed): This is incorrect. A gift card provided to incentivize completion of research surveys is included for research purposes only, and will not be a part of the intervention when it goes to scale. Therefore, it cannot be considered an intervention component.
Lesson 3: Main Effects and Interactions
The lesson reviews the structure of the factorial experiment and what it estimates: main effects and interactions.
Factorial experiments are not “MOST designs”: “I want to start by observing that I've noticed that some people refer to factorial experiments as MOST designs, this is incorrect.” Several designs could be used in the optimization phase, and factorial experiments “have a long and distinguished history that predates MOST by more than a century.” “They were first used in the 19th century in agriculture.” “Sir Ronald A. Fisher created the factorial analysis of variance (ANOVA) a method of analyzing the data from a factorial experiment.”Narration, Lesson 3; slides 2.5 to 2.6
Definition of a factorial experiment: “In a factorial experiment, several independent variables are manipulated in a coordinated fashion.” “The independent variables are called factors.” This manipulation produces an array of experimental conditions. In the two-by-two (2^2) example, factors A and B each have levels no and yes, producing four experimental conditions. “This design includes all possible combinations of the factors A and B.” It is most common for a factorial design to include all possible combinations, but “some very useful factorial designs consist of a subset of the possible experimental conditions.” “These are called fractional factorial designs.”Narration, Lesson 3; slides 2.5 to 2.7
Main effect of factor A: “The main effect of factor A is the mean of the two conditions where factor A equals yes” minus “the mean of the two conditions where factor a equals no” (the lowercase 'a' is a caption artefact). “The main effect of factor A is interpreted as the effect of factor A averaged across the levels of factor B.”Narration, Lesson 3; slides 2.9 to 2.10
Main effect of factor B: “The main effect of factor B is the mean of the experimental conditions in which factor B is set to yes minus the mean of the experimental conditions in which factor B is set to no.” “The main effect of factor B is interpreted as the effect of factor B averaged across the levels of factor A.”Narration, Lesson 3; slides 2.9 to 2.10
Notation for factorial designs: a 2x2 experiment, two factors each with two levels, is a 2^2 or two-squared experiment; a 2x2x3 experiment has three factors, two with two levels and one with three levels; a 2^3 experiment has three factors each with two levels. “A 2^3 is not the same as a 2x3.” “factorial experiments in which all k factors have two levels are referred to as 2^k experiments.” The course focuses on 2^k experiments because “these tend to be the most efficient variety of factorial experiment.”Narration, Lesson 3; slides 2.13 to 2.16
The A x B interaction: the two-way (second order) interaction. “The order of an interaction refers to the number of factors involved in the interaction.” “One way to view this two way interaction is as the difference in the effect of factor A across the levels of factor B.” Equivalently, it is the difference in the effect of factor B across the levels of factor A. The interaction is computed as [(Mu4 - Mu2) - (Mu3 - Mu1)] divided by 2. “We call that two in the denominator, the coefficient correction.”Narration, Lesson 3; slides 2.19 to 2.20
How this relates to intervention optimization: “Each factor in a factorial experiment is used to evaluate the performance of a component.” “The main effect is an estimate of the component's effect on average.” The evaluation includes “consideration of whether and how it affects the performance of other components”, expressed in the interaction.Narration, Lesson 3; slide 2.21
Component versus factor: “A component is a part of an intervention, whereas a factor is an experimentally manipulated variable.” In an optimization trial, a factor is constructed by the experimenter to learn about the performance of a component. “Components, not factors, are selected or not selected for inclusion in an intervention.”Narration, Lesson 3; slide 2.22
Figure 2: The 2^2 factorial design and its main effects (redrawn)
Experimental condition
Factor A
Factor B
Mean outcome
1
no
no
Mu1
2
no
yes
Mu2
3
yes
no
Mu3
4
yes
yes
Mu4
Main effect of A = (Mu3 + Mu4)/2 − (Mu1 + Mu2)/2 (mean of conditions 3 and 4 minus mean of conditions 1 and 2). Main effect of B = (Mu2 + Mu4)/2 − (Mu1 + Mu3)/2. A x B interaction = [(Mu4 − Mu2) − (Mu3 − Mu1)]/2. Redrawn from slides 2.8 and 2.19.
Knowledge check
MULTIPLE CHOICE: A properly designed, conducted, and analyzed factorial experiment can be used to draw inference about which quantities?
☐ main effects only (This is incorrect. A factorial experiment can be used to draw inferences about main effects, but also about other quantities.)
☑ main effects and interactions
☐ the differences between the means of individual experimental conditions
☐ the effect of a treatment package (This is incorrect. An ERCT can be used to estimate the effect of a treatment package.)
Feedback when incorrect (source text; source anomaly, not endorsed): This is incorrect. A gift card provided to incentivize completion of research surveys is included for research purposes only, and will not be a part of the intervention when it goes to scale. Therefore, it cannot be considered an intervention component.
Lesson 4: Fundamental Differences Between the ERCT and the Factorial Experiment
The lesson compares three ways to examine the performance of four candidate components (motivational interviewing, peer mentoring, text message support, mindfulness meditation; all no/yes) and explains why the factorial experiment is far more efficient.
Design option A: four individual treatment/control experiments: “The individual experiments approach requires eight experimental conditions, that is, four experiments with two conditions each.” For 80% power with a standardized effect size of 0.3 (the caption says “standardized defect size”; a caption artefact), “we would require a sample size of 352 for each of the four experiments, for a total required sample size of 1,408.” This approach “does not enable estimation of interactions.”Narration, Lesson 4; slides 2.6 to 2.7
Design option B: comparative treatment experiment: “In a comparative treatment experiment, each of the components under consideration would be represented by an experimental condition in which that component is set to yes and all the others are set to no.” There is one condition per component plus a control condition in which all components are set to no. “There are other very similar experimental designs, such as the dismantling experiment, that have identical statistical properties.” “The comparative treatment experiment would require five experimental conditions in our example.” “To achieve a power of at least 0.8, you would need 880 participants.” It is “essentially a five arm ERCT” and still cannot estimate interactions.Narration, Lesson 4; slides 2.8 to 2.9
Design option C: the 2^4 factorial experiment: “This will have two to the fourth or 16 experimental conditions.” Each factor is set to yes in eight conditions and no in eight conditions. “The factorial design requires by far the largest number of experimental conditions, 16, as opposed to 8 for the individual experiments approach and 5 for the comparative treatment experiment. However, the factorial design requires many fewer participants, only 352 as opposed to 880 for the comparative treatment approach and 1,408 for the individual experiments approach.”Narration, Lesson 4; slides 2.10 to 2.12
ERCT purpose: “The purpose of the ERCT is direct comparison of individual experimental condition means.” In a typical two-arm ERCT, the purpose is direct comparison of the intervention condition mean against the control condition mean.Narration, Lesson 4; slide 2.13
Simple (conditional) effects versus main effects: in options A and B, the effect of motivational interviewing is the mean of a condition with MI set to yes (everything else no) versus a condition with everything set to no, “the equivalent of pulling out experimental condition nine, which is darkly shaded, and experimental condition one, lightly shaded, and comparing their means.” “This kind of effect is often called a simple effect in the social and behavioral sciences and is more widely known as a conditional effect.” By contrast, “In a factorial experiment, the effect of a factor is its main effect.” “The main effect of MI is the mean of conditions 9-16 minus the mean of conditions 1-8.”Narration, Lesson 4; slides 2.13 to 2.18
Every estimate uses all participants: “Notice that this estimate of the main effect of the peer mentoring factor is based on all of the experimental conditions and by extension all of the participants.” It is based on the same participants used for the main effect of MI, “they're just rearranged.” All of the experimental conditions are involved in each definition, “they are just combined in different ways to define the different effects.” “In a sense, experimental participants are recycled, by this I mean, each participant is used in multiple effect estimates, yet under many circumstances the effect estimates are uncorrelated, we will discuss this later.”Narration, Lesson 4; slides 2.18 to 2.21
The “genius” of factorial ANOVA: “In my opinion, this is the genius of the factorial analysis of variance.” “In a factorial analysis of variance of data from a 2^k experiment using effect coding, that is, (-1,1) coding”and “each main effect and interaction effect estimate is based on all participants.”Narration, Lesson 4; slide 2.21
Fisher on the wider inductive basis: the narration quotes Fisher: “any conclusion has a wider inductive basis when inferred from an experiment in which the quantities of other ingredients have been varied, than it would have from any amount of experimentation, in which these had been kept strictly constant” and “in fact, as the factorial arrangement well illustrates, we may, by deliberately varying in each case, some of the conditions of the experiment, achieve a wider inductive basis for our conclusions, without in any degree impairing their precision.” (The caption spells the name “Fischer”.)
Why define component effects as averages: “I would argue that it is desirable to define the effect of a component as an average across lots of different combinations of components and component levels.” First, “the continual optimization principle of MOST suggests that the intervention will evolve over time, new components will be devised, tested, and added to the intervention, and weak components will be removed”, so it makes sense to build from components whose effects obtain across many contexts. Second, “A second reason is that the intervention we designed is not always the one the participants complete.” “Participants do not always receive or comply with all of the components.”Narration, Lesson 4; slides 2.23 to 2.24
No direct comparison, no control condition: “Whereas the objective of an ERCT is direct comparison of the means of experimental conditions, the purpose of a factorial experiment is not direct comparison of any pair or subset of the experimental conditions.” “in a factorial experiment, the objective is not comparison of any condition or set of conditions to a single experimental condition that is designated as the control.” “This means that although condition 1 looks like a traditional control condition, it does not serve as the control in the illustrated factorial design.”Narration, Lesson 4; slide 2.25
Fundamental differences at a glance: “The ERCT was designed for direct comparison of a relatively small number of experimental conditions.” “Whereas the factorial experiment was designed for estimation of the main effect of each factor and interactions involving two or more factors.” Effects are estimated in the ERCT “via direct comparison of individual experimental conditions” and in the factorial “via construction of combinations of experimental conditions”. Finally, “in the ERCT, each participant usually contributes to a single effect estimate, but in the factorial experiment, each participant contributes to multiple effect estimates.”Narration, Lesson 4; slide 2.26
Figure 3: Comparing design options for four components (redrawn)
Design option
Experimental conditions
Sample size for 80% power (d = 0.3)
Interactions estimable?
A. Four individual treatment/control experiments
8 (4 x 2)
1,408 (352 per experiment)
No
B. Comparative treatment experiment (five-arm ERCT)
5
880
No
C. 2^4 factorial experiment
16
352
Yes
Redrawn from slides 2.7, 2.9 and 2.12. The factorial has the most conditions but the fewest participants, because every main effect estimate uses all participants.
Knowledge check
MULTIPLE CHOICE: Is the following statement true or false? Factorial experiments always require larger numbers of experimental participants than alternative approaches.
☐ True
☑ False
Lesson 5: The Factorial ANOVA as a Regression Model
The lesson expresses factorial ANOVA within the regression framework and develops effect (−1,1) coding for a 2^3 experiment with factors MI, PEER and TEXT (motivational interviewing, peer mentoring, text message support).
Notation review: k is the number of factors. “In this course, we will focus on experiments in which all k factors have two levels. These are referred to as 2^k experiments.” In the 2^2 example, the main effect of Factor A is the mean of experimental conditions 3 and 4 minus the mean of conditions 1 and 2, interpreted as “the effect of Factor A averaged across the levels of Factor B”; Factor B analogously uses conditions 2 and 4 minus conditions 1 and 3.Narration, Lesson 5; slides 2.4 to 2.6
The full ANOVA model for a 2^3 experiment: “In a full ANOVA model, for this 2^3 experiment, there are an intercept, three main effects.” “Three 2-way interactions, MI x PEER, MI x TEXT, and PEER x TEXT, and one 3-way interaction, MI x PEER x TEXT.” These effects “can be estimated via regression.” The outcome column now shows Y bars rather than mu, indicating estimated means from empirical data.Narration, Lesson 5; slides 2.10 to 2.11
The regression model: “Except for the intercept, all of the b weights in this regression equation correspond to main effects and interactions.” The equation read out in the narration is: y-hat = b0 + b1 X_MI + b2 X_PEER + b3 X_TEXT + b4 X_MI*PEER + b5 X_MI*TEXT + b6 X_PEER*TEXT + b7 X_MI*PEER*TEXT, where b1 to b3 correspond to main effects, b4 to b6 to two-way interactions, and b7 to the three-way interaction (formatting added; caption spacing artefacts removed).Narration, Lesson 5; slide 2.13
Dummy coding versus effect coding: prior regression experience may involve zeros and ones. “That is called either dummy coding or reference category coding.” “In this course, we use a different coding scheme called effect coding in a 2^k experiment, effect codes are ones and minus ones.” Effect coding is used “because it is consistent with Fischer's definitions of main effects and interactions” (caption spelling; Fisher).Narration, Lesson 5; slides 2.14 to 2.17
Assigning main effect codes: “everywhere a factor in the experimental design is set to no, there is a code of -1 in the table on the left.” “Everywhere a factor in the experimental design is set to, yes, there is a code of 1 in the table on the left.” Each experimental condition corresponds to one row of codes; “every participant in a particular experimental condition has the same set of effect codes.”Narration, Lesson 5; slides 2.16 to 2.18
Intercept and main effect coefficients: “In a regression model defining a factorial analysis of variance using effect coding, the intercept corresponds to the grand mean of the outcome variable.” “In other words, each regression coefficient corresponding to a main effect is equal to half that main effect.” “The standard errors are scaled accordingly, so the hypothesis tests are unaffected.”Narration, Lesson 5; slides 2.21 to 2.23
Coding interactions: “The effect codes for the interactions are obtained by multiplying the effect codes for the corresponding main effects.” For a two-way interaction the codes of the two main effects are multiplied; for a three-way interaction, the codes of three main effects are multiplied. In Condition 1, -1 x -1 = 1 is the code for the MI by PEER interaction; in Condition 3, -1 x 1 = -1.Narration, Lesson 5; slides 2.24 to 2.27
Interaction coefficients: “When effect coding is used, each regression coefficient corresponding to an interaction is equal to half that interaction.” “This holds only for one definition of the interaction. There are two subtly different definitions of the interaction in wide use.” The standard errors are scaled accordingly, so hypothesis tests are unaffected.Narration, Lesson 5; slides 2.28 to 2.29
The algebra in words: “you can show that if effect coding were used to analyze the results of the two to the third experiment we have presented as an example, b_0 equals the grand mean, b_1 equals half the main effect of MI, and b_4 equals half the interaction between MI and PEER.” The derivation is in the Kugler, Dziak & Trail chapter in the Collins & Kugler book (also listed in the set reading).Narration, Lesson 5; slide 2.30
What this has to do with optimization: “Optimization of a multicomponent intervention is the process of identifying a strategic balance of intervention effectiveness against affordability, scalability, and efficiency.” “The ŷs are estimates of the expected effectiveness associated with each experimental condition.” This definition of effectiveness differs from the treatment-control difference: “During the optimization phase, we define effectiveness in terms of ŷ.” “During the evaluation phase, we define it in terms of the treatment control difference.”Narration, Lesson 5; slides 2.31 to 2.33
Figure 4: Effect codes for the 2^3 experiment with MI, PEER and TEXT (redrawn)
Condition
MI
PEER
TEXT
X_MI
X_PEER
X_TEXT
X_MI*PEER
X_MI*TEXT
X_PEER*TEXT
X_MI*PEER*TEXT
1
no
no
no
−1
−1
−1
+1
+1
+1
−1
2
no
no
yes
−1
−1
+1
+1
−1
−1
+1
3
no
yes
no
−1
+1
−1
−1
+1
−1
+1
4
no
yes
yes
−1
+1
+1
−1
−1
+1
−1
5
yes
no
no
+1
−1
−1
−1
−1
+1
+1
6
yes
no
yes
+1
−1
+1
−1
+1
−1
−1
7
yes
yes
no
+1
+1
−1
+1
−1
−1
−1
8
yes
yes
yes
+1
+1
+1
+1
+1
+1
+1
Redrawn from slides 2.14, 2.16 and 2.24 to 2.26. Main effect codes are −1 for no and +1 for yes; interaction codes are the products of the corresponding main effect codes.
Lesson 6: The Balance Property of Factorial Experiments
The lesson defines the two aspects of balance and their consequences for estimation in a 2^3 experiment with MI, PEER and TEXT.
Balance is a defining characteristic: “The balance property is an important even defining characteristic of factorial experiments” (the caption runs the wording together; the intended reading is important, even defining). “Let's examine a balanced factorial experiment to see the two aspects of balance.”Narration, Lesson 6; slide 2.4
Aspect 1 of balance (structure): “In a balanced factorial experiment, each level of each factor appears the same number of times at each level of every other factor.” In the 2^3 example, the no level of TEXT appears twice at the no level of MI and twice at the yes level of MI; the same holds for the yes level of TEXT and for every factor pair. “The first aspect of balance is critical.”Narration, Lesson 6; slides 2.5 to 2.8
Background variability: within the yes level of MI (conditions 5 through 8), the other two factors vary; “This presence of what you might think of as background variability sometimes worries people who are thinking about conducting a factorial optimization trial for the first time.” “But in a balanced factorial experiment, the factors are varied exactly the same way across the no level of MI, that is conditions 1 through 4.”Narration, Lesson 6; slides 2.9 to 2.11
Why balance protects the main effect estimate: suppose PEER has a main effect so that every condition with PEER set to yes has a higher mean. “This will affect the conditions in which MI equals no and the conditions in which MI equals yes by the same amount. So there is no net impact on the estimate of the main effect of MI.” “The situation is more complicated if there are substantial interactions but the balance property makes estimation of interactions straightforward.” “Main effects can and of course should be interpreted in the light of any interactions.”Narration, Lesson 6; slides 2.10 to 2.12
Do you need all the conditions? Not necessarily, but: “First, it is highly recommended that you structure the experiment so that you preserve aspect one of the balance property.” “once you remove even a single condition from a factorial experiment, you will face trade-offs in terms of accuracy of estimation of effects.” Fractional factorial designs “preserve the balance property and make the trade-offs clear”, and any reduced or fractional design “must be selected with particular care.”Narration, Lesson 6; slide 2.14
Aspect 2 of balance (sample size): “The second aspect of balance is maintaining an equal number of participants in each and every experimental condition.” “In a perfectly balanced experiment, both aspects of balance hold.”Narration, Lesson 6; slide 2.15
Consequences of perfect balance with effect coding: “if effect coding is used to analyze the data from a perfectly balanced 2 to the K experiment, A the estimates of all main effects and interactions are uncorrelated and B the standard errors are identical for all of the effect estimates.” “This means that for a regression coefficient of a given size, statistical power is identical whether the regression coefficient corresponds to a main effect or an interaction.”Narration, Lesson 6; slide 2.15
Equal n in practice: “The first aspect of balance is critical. In practice, the second aspect of balance, equal ends, can be difficult to attain, particularly if you are conducting an experiment in a field setting” (the caption renders the intended phrase equal n's as “equal ends”). If condition sample sizes vary, the effect estimates become correlated to some extent: “In general, if the differences in sample size are not large, the correlations will be modest.” “The greater the variability in experimental condition sample size, the larger the correlations among the effect estimates.” “So it is a good idea to try to maintain equal ends to the extent that you can” (again, equal n's in the original wording).Narration, Lesson 6; slide 2.16
Figure 5: The 2^3 factorial experiment used to illustrate balance (redrawn)
Condition
MI
PEER
TEXT
Sample size
Mean outcome
1
no
no
no
n = 40
Ybar1
2
no
no
yes
n = 40
Ybar2
3
no
yes
no
n = 40
Ybar3
4
no
yes
yes
n = 40
Ybar4
5
yes
no
no
n = 40
Ybar5
6
yes
no
yes
n = 40
Ybar6
7
yes
yes
no
n = 40
Ybar7
8
yes
yes
yes
n = 40
Ybar8
Redrawn from the Lesson 6 deck (slides 2.5 to 2.15). Aspect 1: each level of each factor appears the same number of times at each level of every other factor. Aspect 2: equal sample size in every condition. The lesson closes the module: “Congratulations, you have completed module 2.”
Knowledge check
MULTIPLE RESPONSE: Which of the following are part of the definition of a perfectly balanced factorial experiment? Select all that apply.
☐ There are no interactions.
☐ For each positive main effect, there is a corresponding negative main effect.
☑ All experimental conditions have the same number of participants.
☑ Each level of each factor appears the same number of times at each level of every other factor.
Synthesis (my own reading: not part of the course)
Read methodologically, this module is the bridge from the conceptual argument of Module 1 to the design machinery used in the rest of the course. The pivotal move is the redefinition of the estimand. An ERCT defines the effect of a component as a simple (conditional) effect, the difference between two conditions that differ only on that component, which is why it needs one dedicated condition pair per component and cannot touch interactions. The factorial experiment redefines the effect of a component as a main effect, an average over conditions that differ in many other ways, and thereby converts a sample-size problem into a design problem: the same 352 participants yield four main effects and six interactions in the 2^4 example. The statistical price is that the interpretation is average-based, and the course quotes Fisher to justify exactly that: the conclusion has a wider inductive basis because the effect is observed across varied conditions rather than in a tightly controlled one.
Three technical points are easy to misread and worth fixing in your own notes. First, 2^k denotes k factors each at two levels, not a 2-by-k design; the narration is explicit that 2^3 and 2x3 are different designs. Second, under effect coding each regression coefficient equals half the corresponding main effect or interaction, not the effect itself; the hypothesis tests are unaffected because the standard errors scale identically. Third, the “balance property” has two aspects, the structural property that each level of each factor occurs equally often at each level of every other factor (critical, and preserved by fractional factorials), and equal sample sizes across conditions (desirable but often unattainable in the field). Only perfect balance plus effect coding delivers uncorrelated estimates with identical standard errors, the property that makes the 2^k design so attractive for optimisation trials.
One caution for readers of the literature: this module's Lesson 1 slide and narration define effectiveness as “the extent to which the intervention does more harm than good under real world conditions”, which inverts the standard Flay (1986) definition (“more good than harm”) used in Module 1 of the conceptual course. The inversion appears in the source material itself and is flagged in the notes rather than silently corrected. Similarly, the knowledge-check answer keys in this course contain an internally inconsistent item (Lesson 2) and question-level feedback copied from an unrelated item (Lessons 1 to 3); the notes reproduce the course exactly and label the anomalies.
Glossary
Preparation phase
Lay the groundwork for optimization: derive a detailed conceptual model, identify candidate components, conduct pilot work, and specify the optimization objective.
Conceptual model
A model of the process to be intervened upon, showing how each candidate component is hypothesized to affect the outcome pathway; the basis for every intervention optimized using MOST. Not a structural equation model.
Optimization objective
The investigator's definition of intervention EASE for a particular study (also called the optimization criterion); expresses the desired balance of effectiveness against affordability, scalability and efficiency, and is specified during the preparation phase.
Intervention EASE
A strategic balance of the four desiderata, Effectiveness, Affordability, Scalability, Efficiency, balancing effectiveness against the other three.
Immediate scalability
A scalable intervention is implementable in the intended setting exactly as evaluated, with no ad hoc modifications.
Optimization trial
An experiment conducted to gather some of the information needed for optimization, focusing on the performance of individual intervention components.
Evaluation RCT (ERCT)
A design whose objective is assessment of whether the intervention as a package outperforms a suitable control or comparison; direct comparison of individual experimental condition means.
Resource management principle
The investigator must carefully choose an experimental design so as to gather the information needed while making the most of, but not exceeding, the available resources; design selection must best address the key research questions while making the best use of research resources.
Factorial experiment
An experiment in which several independent variables (factors) are manipulated in a coordinated fashion, producing an array of experimental conditions; usually including all possible combinations of factor levels (2^k experiments in this course).
Factor
An experimentally manipulated variable; constructed by the experimenter to learn about the performance of a component.
Main effect
The mean of the conditions at one level of a factor minus the mean of the conditions at the other level; interpreted as the effect of the factor averaged across the levels of the other factors.
Interaction
The difference in the effect of one factor across the levels of another (two-way or second-order interaction for two factors); the order of an interaction is the number of factors involved.
Coefficient correction
The constant in the denominator of the interaction estimator (2 for a two-way interaction in this course's definition), which varies with the order of the interaction.
Simple (conditional) effect
An effect defined as the difference between two conditions that differ only on the component of interest, holding everything else constant; the kind of effect an ERCT estimates.
Comparative treatment experiment
A multi-arm design in which each component is represented by a condition with that component set to yes and all others set to no, plus a control condition with all components set to no; essentially a multi-arm ERCT.
Dismantling experiment
A design statistically identical to the comparative treatment experiment.
Factorial ANOVA
Analysis of variance of data from a factorial experiment, expressible within the general linear model regression framework.
Effect coding
Coding scheme for 2^k experiments using −1 and +1 (no and yes), consistent with the definitions of main effects and interactions; contrast with dummy (reference category) coding.
Balance (aspect 1)
Each level of each factor appears the same number of times at each level of every other factor.
Balance (aspect 2)
Equal sample size in each experimental condition.
Sources
Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Set reading: Chapter 3.)
Collins, L. M., Kugler, K. C., & Gwadz, M. V. (2016). Optimization of multicomponent behavioral and biobehavioral interventions for the prevention and treatment of HIV/AIDS. AIDS and Behavior, 20, 197 to 214. (Source of the conceptual model figure.)
Kugler, K. C., Dziak, J. J., & Trail, J. (2018). Coding and interpretation of effects in analysis of data from a factorial experiment. In L. M. Collins & K. C. Kugler (Eds.), Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer. (Cited in Lesson 5 for the effect-coding algebra.)
Fisher, R. A. (1971). The design of experiments. New York, NY: Hafner Publishing. (Source of the quotation on the wider inductive basis; cited at the end of Lesson 4.)
Flay, B. R. (1986). Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Preventive Medicine, 15, 451 to 474. (Cited for the effectiveness desideratum on the Lesson 1 slide, whose printed wording inverts the definition.)
Prepared 9 Sep 2026. Notes paraphrase the narration unless marked as verbatim quotes; figures are redrawn from the course slides; quotes follow the YouTube caption transcripts. Caption transcription artefacts are preserved inside quotes and flagged. The course's five knowledge checks are reproduced once each with the course answer key; the course presents no separate quiz distinct from these items.