Study notes · CADIO / NYU School of Global Public Health
Module 3: “Introduction to the Optimization Trial” Authors: Linda M. Collins and Kate Guastaferro (School of Global Public Health, New York University) Format: 8 lessons · 8 videos (79 min) · 8 slide decks (PDF) · 8 knowledge checks · 8-item quiz Set reading: Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 3 for Lessons 1 to 6; Chapter 8 for Lessons 7 to 8) Access: free via d3center / CADIO share links, or Coursera
The module's argument in one paragraph.
Module 3 moves from the preparation phase (Modules 1 to 2) into the optimization phase: the optimization trial, whose objective is “efficient assessment of intervention components”, how each component performs on average and in combination with others. The module's spine is the resource management principle: the investigator must make the best and most efficient use of available resources, so the experimental design is chosen to manage research resources well and most directly address the primary research questions. The workhorse design is the factorial experiment, most widely applicable, often the most efficient, and the root of most other optimization trial designs. Three properties make it economical: every participant's data contributes to every effect estimate (participants are “recycled”); power is driven by the N per level of each factor, not by per-condition n, so adding factors need not add participants; and effect coding with balanced designs keeps effect estimates uncorrelated with identical standard errors. Against this, adding a level to a factor costs at least 50% more participants, so 2^k experiments are the efficient default. The module closes by connecting intervention design to trial design: fixed interventions raise component-performance questions best answered by factorial designs; adaptive interventions raise sequential questions best answered by SMARTs (low adaptation intensity) or micro-randomized trials / system identification (high intensity).
Lesson 1: Importance of the Resource Management Principle
The lesson sets the selection rule for optimization trial designs.
Objective of an optimization trial: “The objective of an optimization trial is efficient assessment of intervention components, that is, how each component performs on average and how it performs in combination with other components.”Narration, Lesson 1
Why conduct one? You might consider an optimization trial if you want to weed out underperforming intervention components; get a sense of the magnitude of each component's effect; or examine whether the effect of a component is augmented or reduced in the presence of another (statistically, an interaction). The information is used to select components or component levels to optimize the intervention, to achieve intervention EASE.
Intervention EASE: “Intervention EASE is a strategic balance of the four desiderata for multicomponent interventions, in other words, a strategic balance of effectiveness against affordability, scalability, and efficiency.”Narration, Lesson 1
Resource management principle: “an investigator using MOST must strive to make the best and most efficient use of available resources when obtaining scientific information. Any experimental design is okay as long as it is the most efficient one.”Narration, Lesson 1
The most efficient design: “The most efficient design for an optimization trial is the one that best manages research resources (we will be discussing this soon) and most directly addresses the primary research questions.”Narration, Lesson 1
Scenario 1 (the running example of the module): an intervention scientist is developing an intervention aimed at reducing viral load among HIV positive heavy drinkers.
Candidate component
Levels
Motivational interviewing
no / yes
Peer mentoring
no / yes
Text message support
no / yes
Mindfulness meditation
no / yes
Behavioral skills training
low intensity / high intensity
Sometimes a subset of the candidate components is used to illustrate different points.
Why not just run a huge ERCT? A 32-arm ERCT for five components would in theory be definitive but is not feasible to power. Instead, manage research resources strategically: gain the most information, gain the most reliable information, and move intervention science forward the fastest. The investigator decides what information is most important and targets resources there; chooses the design most appropriate for the type of intervention being optimized; and chooses the design for efficiency.
Resources: “By definition, MOST does not require an increase in research resources, but in most cases, it will require a realignment of research resources.”Narration, Lesson 1
Groundwork before selecting an experimental design. The objective is to gather information that will be used in decision-making about which components and component levels go into the optimized intervention. The starting point is always: what decisions do I need to make? For components 1 to 4 (no/yes), the question is whether the component has a detectable effect, does the yes level outperform the no level? If yes, the component is considered for inclusion (it may still be left out to meet the optimization objective, e.g. affordability or scalability). If no, the component is not included. For behavioral skills training (low/high intensity), the question is whether the high-intensity level outperforms the low-intensity level; cost may force settling for the low level.
What we want to find out: for each component, whether there is a difference between the higher and lower levels; ideally, whether components interact.Narration, Lesson 1
The principle, precisely: “The resource management principle says that the investigator must carefully choose an experimental design so as to gather the information needed while making the best use of, but not exceeding, the available resources.” The experimenter must have a clearly specified set of research questions, know what resources are available, and know what resources each design requires, different designs can require different resources.Narration, Lesson 1
Why the factorial optimization trial is emphasized in this course: “it's the most widely applicable optimization trial and it's often the most efficient. Moreover, most, but not all, other optimization trial designs are rooted in the factorial experiment.”
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are consistent with the resource management principle of MOST? Select all that apply.
☑ It is important to identify the experimental design that most directly addresses the primary research questions.
☐ In general, experimental designs all cost about the same to implement.
☐ First select an experimental design, then take stock of what resources are available to implement it.
☑ It is important to manage research resources strategically.
Feedback when incorrect: In general, experimental designs all cost about the same to implement.This is incorrect. Different experimental designs under consideration application in the same study can differ widely in how much they cost to implement.First select an experimental design, then take stock of what resources are available to implement it.This is incorrect. The starting point for selection of an experimental design is always an assessment of what resources are available.
Lesson 2: Introduction to the Factorial Experiment
The lesson reviews the factorial experiment: what it is, what it estimates (main effects and interactions), and why coding matters for interpretation.
Not “MOST designs”: factorial experiments predate MOST by more than 100 years, and the factorial design is not the only experimental design used in MOST. Please do not refer to factorial experiments as MOST designs.Narration, Lesson 2
History. Factorial experiments were first used in the 19th century in agriculture. Sir Ronald Fisher created the factorial analysis of variance (ANOVA), a method of analyzing data from a factorial experiment.
Factorial experiment: “In a factorial experiment, several independent variables are manipulated in a coordinated fashion. These independent variables are called factors.” Ideally there are equal numbers of participants in each experimental condition; in field experiments imbalance is rare to avoid, but severe imbalance can cause problems.Narration, Lesson 2
2×2 (2²) example. Two factors, A and B, crossed to create four experimental conditions (cells), representing all four combinations of the two levels of A and B. The main effect of Factor A is the mean of the conditions in which A is set to yes (blue) minus the mean of the conditions in which A is set to no (yellow). Similarly for B.
Experimental condition
Factor A
Factor B
Outcome
1
No
No
Ȳ₁
2
No
Yes
Ȳ₂
3
Yes
No
Ȳ₃
4
Yes
Yes
Ȳ₄
Notation. A 2×2 experiment = two factors, each with two levels = 2². A 2×2×3 experiment = three factors, two with two levels and one with three levels = 2²×3. A 2×3 factorial experiment is an experiment with two factors, one with two levels, one with three levels. A 2×2×2 (2³) experiment has three factors, each with two levels, it is not a 2×3. This distinction matters because 2³ and 2×3 are very different designs: 2³ has 8 conditions, 2×3 has 6.
When all k factors have two levels, the general class is referred to as a 2^k experiment.
Main effect of Factor A: “The main effect of Factor A is the mean of experimental conditions 3 and 4 in blue, minus the mean of experimental conditions 1 and 2 in yellow. Note that this is interpreted as the effect of Factor A averaged over the levels of Factor B.”Narration, Lesson 2
Interaction of A and B: “Is the effect of Factor B the same across the levels of Factor A? If it is, there is no interaction. If the effect of Factor B varies across the levels of Factor A, then there is an interaction.” Equivalently: is the effect of Factor A the same across the levels of Factor B? You get the same answer.Narration, Lesson 2
How this relates to optimization: “each factor in a factorial experiment can be used to evaluate the performance of a component.”Narration, Lesson 2
Coding: “Coding refers to how the levels of a factor are represented numerically in data analysis. In this course, we are using effect coding. In effect coding, if a factor has two levels, the levels are represented by -1 and 1.” Dummy (reference category) coding uses 0 and 1. The omnibus F is the same under either coding; the main effect and interaction estimates are generally not the same. Effect-coded main effects and interactions should be interpreted as described here; dummy-coded ones generally should not.Narration, Lesson 2
Two points about effect coding. (1) In a properly conducted factorial experiment, effect coding produces main effect and interaction estimates that are all uncorrelated if ns are equal across experimental conditions; if ns are unequal, usually only modestly correlated, one reason to aim for equal ns. With dummy coding, effects are usually highly correlated. (2) In a properly conducted 2^k factorial experiment with equal ns, the standard errors are identical for all effect estimates, main effects and all interactions up to the k-way interaction, so, all else equal and given a particular anticipated effect size, power is identical for all these estimates.
Explained further in Section 3.11.2 of Collins (2018) and in detail in Kugler, Dziak, and Trail (2018).
Knowledge check
MULTIPLE RESPONSE: Which of the following statements about the factorial experimental design are correct? Select all that apply.
☑ A 2k factorial experiment has k factors, each with two and only two levels.
☐ A 23 factorial experiment can also be called a 2×3 factorial experiment.
☑ The main effect of Factor A is defined as Factor A’s effect averaged over all the levels of all the other factors in the experiment.
☐ Main effects and interaction effects are interpreted identically whatever kind of numerical coding is used to perform the statistical analysis.
Feedback when incorrect: A 23 factorial experiment can also be called a 2×3 factorial experiment.This is incorrect. A 23 factorial experiment has 3 factors each with 2 levels, whereas a 2×3 factorial experiment has 2 factors, one with 2 levels and one with 3 levels.Main effects and interaction effects are interpreted identically whatever kind of numerical coding is used to perform the statistical analysis.This is incorrect. Main and interaction effects are typically interpreted differently depending on the numerical coding used in the analysis. In this course we use effect (-1, 1) coding exclusively.
Lesson 3: Fundamental Differences between the Factorial Experiment and the ERCT
The lesson compares ways to experimentally manipulate four candidate components (motivational interviewing, peer mentoring, text message support, mindfulness meditation) and draws out the logic of factorial efficiency.
Three design options (N needed to maintain power ≥ .8, d = .3):
Design
Number of experimental conditions
Participants needed for power ≥ .8 (d = .3)
Can interactions be estimated?
A: Individual experiments (an ERCT on each component)
8
1,408
No
B: Comparative treatment experiment (control + one condition per component)
5
880
No
C: 2⁴ factorial experiment
16
352
Yes
These approaches examine subtly different definitions of the main effect, discussed in Collins, Dziak & Li (2009). Whether the factorial is more efficient than the comparative treatment experiment depends on the cost of implementing experimental conditions versus the cost of obtaining and retaining participants.
A fundamental difference in purpose: “the purpose of the ERCT is direct comparison of experimental condition means.” “This is not the purpose of a factorial experiment. Instead, the purpose of a factorial experiment is estimation of main effects and interactions.”Narration, Lesson 3
Why the factorial is so efficient. The main effect of MI is the mean of conditions 9 to 16 (MI = yes) minus the mean of conditions 1 to 8 (MI = no); the main effect of PEER is the mean of conditions 5 to 8 and 13 to 16 minus conditions 1 to 4 and 9 to 12; TEXT uses 3,4,7,8,11,12,15,16 minus 1,2,5,6,9,10,13,14; MIND is even-numbered minus odd-numbered conditions. In each case all participants contribute to the estimate: “You can think of this as 'recycling' of research participants.”
Fisher's point about averaging across conditions: “any conclusion has a wider inductive basis when inferred from an experiment in which the quantities of other ingredients have been varied, than it would have from any amount of experimentation, in which these had been kept strictly constant. In fact, as the factorial arrangement well illustrates, we may, by deliberately varying in each case some of the conditions of the experiment, achieve a wider inductive basis for our conclusions, without in any degree impairing their precision.”Narration, Lesson 3 (quoting Fisher, 1971, p. 102)
In an optimization trial, components will operate in conjunction with other components, so examining their performance on average across many other components makes sense.
ERCT
Factorial experiment
Designed for direct comparison of a small number of experimental conditions.
Designed for estimation of the main effect of each factor and interactions involving two or more factors.
Effects estimated via direct comparison of individual experimental conditions.
Effects estimated via construction of combinations of experimental conditions.
Each participant usually contributes to a single effect estimate.
Each participant contributes to multiple effect estimates.
Both are excellent designs. Neither is more efficient across the board; for each design there are situations where one is more efficient than the other.
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☑ Factorial experiments often require substantially fewer experimental participants than alternative approaches.
☑ In a factorial experiment, every participant’s data contributes to multiple effect estimates.
☑ The factorial experiment and the ERCT address different kinds of research questions.
☑ In a factorial experiment, a single participant may be in the treatment condition with respect to some factors and the control condition with respect to others.
Lesson 4: Why Factorial Experiments Can Be So Economical
The lesson explains the statistical logic: power depends on N per level of each factor, not per-condition n.
Statistical power (brief review): “A Type 1 error is defined as mistakenly rejecting the null hypothesis when the null hypothesis is true. The probability of making this error is designated with the Greek letter alpha. A Type II error is defined as failing to reject the null hypothesis when the null hypothesis is false. The probability of making this error is designated with the Greek letter beta. Statistical power = 1-beta; in other words, statistical power is the probability of correctly rejecting the null hypothesis when it is false.”Narration, Lesson 4
How ERCTs are powered. A two-arm ERCT comparing Treatment A with control, powered with 150 per condition (total 300). Adding Treatment B with the same expected effect size does not maintain power by dividing the 300 among three conditions, each treatment-control comparison must be based on 300 participants, so 150 more are needed (total 450).
The factorial contrast. The 2⁴ factorial has 16 experimental conditions and, at 352 total, 22 participants per condition. With ERCT training this looks underpowered: but:
An important difference: “With an ERCT, a small per-condition n is usually an indication that the experiment has low, possibly even unacceptable, power. By contrast, in a balanced factorial experiment, a small per-condition n does not necessarily indicate low power.”Narration, Lesson 4
What determines power in a factorial: “in factorial experiments, the per-condition n does not determine power. Instead, what matters is the N per level of each factor.”Narration, Lesson 4
Example. The main effect of MI = mean of conditions 9 to 16 minus mean of conditions 1 to 8; the N per level (No/Yes) is 176. The main effect of PEER = mean of conditions 5 to 8 and 13 to 16 minus 1 to 4 and 9 to 12; the N per level is still 176, so power remains the same. In a 2^k experiment, all participants contribute to all effect estimates, that is why the 2^k experiment is so efficient and economical.
Balance: “Balance is the reason for the efficiency of factorial experiments.” Two requirements: (a) each level of each factor appears the same number of times at each level of every other factor (an absolute requirement); (b) all experimental conditions have the same sample size (perfect balance; in practice usually approximate balance, which is nearly as efficient).Narration, Lesson 4
Adding a factor without adding participants. Adding behavioral skills training changes a 2⁴ to a 2⁵, doubling the number of experimental conditions to 32but with the same total N, the per-level N stays 176 for every factor (including the new one), so power for detection of all main effects is unchanged. The per-condition n drops to 11; that does not matter.
Remember: “all else being equal, adding one or more factors to a factorial experiment will increase the number of conditions in the experiment. But, it will not necessarily require more participants. In fact, it is very often possible to add one or more factors to a factorial experiment without the need to increase the sample size.”Narration, Lesson 4
Implication: it can be inefficient to conduct a factorial optimization trial with only two or three factors. If you can handle the additional experimental conditions logistically, it may be efficient to investigate a few more components.
Small per-condition n's: “Factorial experiments can have very small per-condition n's and yet maintain high statistical power.” The reason: “in factorial experiments, power is driven primarily by the n per level of a factor, not the number of participants per condition.” Depending on the circumstances, factorial experiments can have per-condition ns as low as 2 or 3 and still be sufficiently powered, provided the per-level sample size is large enough.Narration, Lesson 4
Knowledge check
MULTIPLE CHOICE: Which ONE of the following statements is NOT true?
☐ In a factorial experiment, statistical power is not driven primarily by the per-condition sample size.
☐ An ERCT with a per-condition sample size as low as 10 or 15 is unlikely to have sufficient statistical power.
☑ A factorial experiment with a per-condition sample size as low as 2 or 3 is unlikely to have sufficient statistical power.
☐ When an experimental condition is added to an ERCT, usually an increase in participants is needed to maintain power.
Lesson 5: Why the 2^k Experiment is So Economical
The lesson shows that adding a level to a factor is expensive, unlike adding a factor.
Adding a level, not a factor. Suppose TEXT changes from levels No/Yes to levels No/Low/High. The design still has four factors but is no longer 2^k: it is a 2×2×3×2 with 24 experimental conditions. The 352 participants are divided among 24 conditions (14 or 15 per condition, slight variability is unimportant).
Factor
Levels
Effect on power
MI (unchanged, 2 levels)
No / Yes
Main effect computed as before, averaged over more conditions; still 176 participants per level, power unchanged.
TEXT (now 3 levels)
No / Low / High
Two main effects: High vs No and Low vs No. Each is based on a per-level n of approximately 120 (conditions with TEXT = Low must be left out), so power is lower than for MI.
Bringing power back up. All else equal, the only remedy is more participants. Assuming both TEXT comparisons have about the same anticipated effect as MI, it would take about 176 more participants, an increase of 50%. If the Low vs No comparison has a smaller anticipated effect (likely), even more participants are needed.
The lesson's conclusion: “It is possible to add one or more factors to a factorial experiment and maintain power without adding participants. So, adding one or more factors to a factorial optimization trial is often an efficient strategy. However, adding a level of a factor usually requires at least 50% more participants, so adding even one level to a factor can be costly.”Narration, Lesson 5
Guidance: “the 2^k factorial experiment is generally the most efficient, and to ask yourself, do you really need that in-between level?” A continual-optimization strategy: start with two levels to ascertain a difference between High and No; if observed, examine Low vs No in a subsequent experiment. If no difference between High and No, you probably do not care about Low vs No.Narration, Lesson 5
Knowledge check
MULTIPLE CHOICE: All else being equal, increasing the number of levels of a factor to three or more typically requires an increase in the number of participants to maintain statistical power. Which of the following is the reason?
☐ Because the number of experimental conditions increases. (This is incorrect. EASE does not involve maximizing all four of these attributes of an intervention simultaneously.)
☑ Because the number of participants per level of the factor decreases. (This is incorrect. EASE does not involve balancing efficiency against affordability and scalability.)
Feedback when incorrect: This is incorrect. All else being equal, an increase in the number of experimental conditions alone does not decrease power in a factorial experiment, as long as the sample size per level of each factor remains the same.
Lesson 6: Experimental Control in a Factorial Design
The lesson explains how control works in a factorial experiment: each factor has its own control.
ERCT logic: “In an ERCT, there is usually a single condition designated as THE control.”Narration, Lesson 6
In a factorial experiment, Condition 1 (all factors set to No) is NOT “the” control. Effect estimates are based on aggregations of experimental conditions; the means of individual conditions are not directly compared. To compute the main effect of MI you do not compare conditions 9 to 16 to condition 1; you compare conditions 9 to 16 to conditions 1 to 8.
Each factor has its own control: “Instead, each factor has its own control. That is, one level of each factor serves as the control for that factor.” Conditions 1 to 8 form the control for the MI factor; conditions 1 to 4 and 9 to 12 serve as the control for PEER; and so on.Narration, Lesson 6
Different factors can use different types of control. Suppose an attention placebo is used as a control for Motivational Interviewing; texts containing information about diet for Text Message Support; and a peaceful nature video for Mindfulness Meditation. This is implemented by setting each factor's control level to that control treatment, the controls can be very different.
Control approach
Factor
Attention placebo
Motivational interviewing
No peer mentoring (No)
Peer mentoring
Texts with information about diet
Text message support
Peaceful nature video
Mindfulness meditation
Who is in a “control condition”? “the same individual participant can be in a control condition with respect to one factor and a treatment condition with respect to another factor.” For example, participants in condition 6 are in the control condition with respect to MI and TEXT and in the treatment condition with respect to PEER and MIND. Participants in condition 1 are in the control condition with respect to all four factors, but this is not THE control condition for the entire experiment. In fact, all participants are in a control condition except those in condition 16 (all factors set to Yes).Narration, Lesson 6
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☐ Like the ERCT, factorial experiments include a single particular experimental condition designated as the control condition. (This is incorrect. According to the definition of pilot study we are using in this course, investigators should not power pilot studies at all.)
☐ An investigator using a factorial design must identify a single control treatment that will work for all the factors in the experiment. (This is incorrect. Optimization trials are not pilot studies, although optimization trials should be pilot tested.)
☑ In a factorial experiment, each factor has its own control.
☑ In a factorial experiment, an individual participant may simultaneously be in the treatment condition of certain factors and the control condition of others. (This is incorrect. According to the definition of pilot study we are using in this course, the fundamental purpose of a pilot study is to establish feasibility of an approach that is under consideration for use in a larger study.)
☐ It is not possible to use a placebo control, such as an attention placebo, in a factorial experiment.
Feedback when incorrect: Like the ERCT, factorial experiments include a single particular experimental condition designated as the control condition.This is incorrect. The logic of experimental control differs between the ERCT and the factorial experiment. Factorial experiments typically do not include a single particular experimental condition designated as the control condition.An investigator using a factorial design must identify a single control treatment that will work for all the factors in the experiment.This is incorrect. Because in a factorial experiment each factor has its own control, different types of controls may be used for different factors as appropriate.It is not possible to use a placebo control, such as an attention placebo, in a factorial experiment.This is incorrect. Any type of control is possible in a factorial experiment.
Lesson 7: Brief Introduction to Adaptive Interventions
The lesson distinguishes fixed from adaptive intervention designs, and adaptive interventions by intensity of adaptation.
Why this matters for trial design: “recognizing whether you are optimizing a fixed or adaptive intervention is an important first step in selecting an optimization trial design. Moreover, if you are optimizing an adaptive intervention, the desired intensity of adaptation of the intervention is an additional consideration.”Narration, Lesson 7
Experimental design vs intervention design: “The term experimental design refers to the design of an experiment to gather information needed to develop, optimize, or evaluate an intervention. By contrast, the term intervention design refers to the specific details of the approach taken by an intervention, including the intervention components, the settings of the components, any eligibility requirements for participants, etc.” When you optimize an intervention using MOST, you are optimizing the intervention design.Narration, Lesson 7
Fixed intervention: “In a fixed intervention design, all participants are offered the same treatment.” They may not all receive it (refusal, missed sessions), but the design calls for the same treatment to be offered.Narration, Lesson 7
Adaptive intervention: “in adaptive interventions, intervention components and component levels are varied strategically across participants.” Different participants receive different doses or levels of certain components, or entirely different components. “An adaptive intervention is designed to give each participant the amount and type of intervention needed to achieve the desired outcome, and no more.” Varying treatment strategically conserves resources so participants who benefit from more intensive treatment can receive it.Narration, Lesson 7
Figure 1: Adaptive intervention to encourage adherence to HIV treatment
The running adaptive-intervention example: a sequence of pre-specified decision rules with a single decision point at Day 45.
Definition of adaptive intervention: “An adaptive intervention is a sequence of pre-specified decision rules that can be used to guide whether, how, or when - and based on which measures - to alter an intervention or intervention component (e.g., treatment type, duration, frequency or amount) at critical decision points during the course of care.”Narration, Lesson 7 (definition from Almirall, Nahum-Shani, Wang, & Kasari, 2018)
Terms illustrated. Pre-specified decision rules assign treatment to individuals; Day 45 is a decision point; adherence is the tailoring variable. Assignment of treatment is planned and systematic, not haphazard or made up on the spot.
Intensity of adaptation: “the degree to which an adaptive intervention has many decision points and/or short review intervals.” The review interval is the time allowed to elapse between decision points if there is more than one. The HIV example has very low intensity of adaptation: only one decision point. Weekly measurement with a weekly opportunity to change the intervention would be a higher intensity. Intensity of adaptation is different from intensity of dosage: “It is completely possible for an intervention to have a high intensity of adaptation and yet deliver a low dose or light-touch intervention”, the strategy used by many app-delivered behavioral interventions.Narration, Lesson 7
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☑ In a fixed intervention, all participants are offered the same treatment.
☑ In an adaptive intervention, different participants may be offered different types and/or doses of treatment.
☐ Intensity of adaptation refers to the intensity of the treatment dosages used in an adaptive intervention.
☑ The decision rules in an adaptive intervention are always pre-specified.
Feedback when incorrect: Intensity of adaptation refers to the intensity of the treatment dosages used in an adaptive intervention.This is incorrect. Intensity of adaptation refers to the degree to which an adaptive intervention has many decision points and/or short review intervals. It has nothing to do with dosage.
Lesson 8: Different Interventions, Different Optimization Trial Designs
The lesson connects intervention design to optimization trial design: different types of interventions generate different scientific questions, and the resource management principle then suggests different designs.
Design selection (review): “The most efficient design for an optimization trial is the one that: best manages research resources, and we discussed this in a previous lesson, and most directly addresses the primary research questions.” Which design most directly addresses the primary research questions depends partly on the type of intervention to be optimized.Narration, Lesson 8
Components in an adaptive intervention. In the adherence example, text messaging, behavioral skills training, peer mentoring, and motivational interviewing can all be considered components, as can the tailoring variable (adherence), the placement of the decision point and whether there is more than one, the decision rules themselves, and (if more than one decision point) the review interval. This is not an exhaustive list.
Intervention design
Typical scientific questions
Usually a good design choice
Fixed intervention
“How well does this component perform?” / “Is the high dose better than the low dose of this component?”
Factorial or fractional factorial experiment (fractional factorial: see Chapter 5 of the textbook; not covered in this course)
Adaptive intervention (low intensity of adaptation)
Sequential questions, is the initial treatment effective overall? Which second-line treatment for adherent participants? For non-adherent participants?
Adaptive intervention (high intensity of adaptation)
Frequent, small decisions about tailoring
Micro-randomized trial (MRT) or system identification experiment
Three questions for the adaptive example. Q1: Is the initial treatment (text messaging, peer mentoring, behavioral skills training) effective overall, does it outperform a comparison or control? Requires randomizing participants at the outset to initial treatment or control. Q2: For adherent participants after 45 days, is continuing text messaging only or text messaging plus peer mentoring the better second-line treatment? Requires waiting to Day 45, assessing adherence, and randomizing only adherent participants. Q3: For non-adherent participants, is continuing text messaging plus peer mentoring only, or adding motivational interviewing, better? Requires randomizing only non-adherent participants.
SMART: “Sets of questions like these cannot be addressed in a standard factorial experiment, but they can be addressed using a sequential multiple-assignment, randomized trial, or SMART.” SMARTs are useful primarily when the adaptive intervention being optimized is characterized by low intensity of adaptation. For high intensity, the micro-randomized trial or a system identification experiment may be a good choice.Narration, Lesson 8
Why the factorial is emphasized in this course: “it's the most widely applicable optimization trial, and because many other optimization trial designs are rooted in the factorial experiment, such as the fractional factorial, the SMART, and the MRT. System identification emerged from a different tradition.”
Start with the questions: “We recommend starting with the scientific questions, then using the resource management principle to select the most appropriate and efficient design. It's usually a mistake to start with the idea that you want to use a particular experimental design, although at times this can be tempting.” If the scientific questions are sufficiently clearly and explicitly stated, selection of the appropriate design is usually not difficult.Narration, Lesson 8
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☐ The design of an optimization trial must be selected based on the continual optimization principle.
☑ Optimization of fixed and adaptive interventions often require addressing different types of research questions.
☐ The factorial experiment is used in optimization of fixed interventions only.
☑ The sequential multiple-assignment randomized trial is an experimental design for optimization of adaptive interventions.
☑ Optimization of adaptive interventions with different intensities of adaptation may require different optimization trial designs.
Feedback when incorrect: The design of an optimization trial must be selected based on the continual optimization principle.This is incorrect. The design of an optimization trial must be selected based on the resource management principle.The factorial experiment is used in optimization of fixed interventions only.This is incorrect. The factorial experiment may be used in optimization of adaptive interventions, depending on what research questions are to be addressed.
Module quiz (self-test)
Reveal the 8 quiz items with answers
MULTIPLE RESPONSE: Which of the following statements are consistent with the resource management principle of MOST? Select all that apply.
☑ It is important to identify the experimental design that most directly addresses the primary research questions.
☐ In general, experimental designs all cost about the same to implement.
☐ First select an experimental design, then take stock of what resources are available to implement it.
☑ It is important to manage research resources strategically.
Feedback when incorrect: In general, experimental designs all cost about the same to implement.This is incorrect. Different experimental designs under consideration application in the same study can differ widely in how much they cost to implement.First select an experimental design, then take stock of what resources are available to implement it.This is incorrect. The starting point for selection of an experimental design is always an assessment of what resources are available.
MULTIPLE RESPONSE: Which of the following statements about the factorial experimental design are correct? Select all that apply.
☑ A 2k factorial experiment has k factors, each with two and only two levels.
☐ A 23 factorial experiment can also be called a 2×3 factorial experiment.
☑ The main effect of Factor A is defined as Factor A’s effect averaged over all the levels of all the other factors in the experiment.
☐ Main effects and interaction effects are interpreted identically whatever kind of numerical coding is used to perform the statistical analysis.
Feedback when incorrect: A 23 factorial experiment can also be called a 2×3 factorial experiment.This is incorrect. A 23 factorial experiment has 3 factors each with 2 levels, whereas a 2×3 factorial experiment has 2 factors, one with 2 levels and one with 3 levels.Main effects and interaction effects are interpreted identically whatever kind of numerical coding is used to perform the statistical analysis.This is incorrect. Main and interaction effects are typically interpreted differently depending on the numerical coding used in the analysis. In this course we use effect (-1, 1) coding exclusively.
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☑ Factorial experiments often require substantially fewer experimental participants than alternative approaches.
☑ In a factorial experiment, every participant’s data contributes to multiple effect estimates.
☑ The factorial experiment and the ERCT address different kinds of research questions.
☑ In a factorial experiment, a single participant may be in the treatment condition with respect to some factors and the control condition with respect to others.
MULTIPLE CHOICE: Which ONE of the following statements is NOT true?
☐ In a factorial experiment, statistical power is not driven primarily by the per-condition sample size.
☐ An ERCT with a per-condition sample size as low as 10 or 15 is unlikely to have sufficient statistical power.
☑ A factorial experiment with a per-condition sample size as low as 2 or 3 is unlikely to have sufficient statistical power.
☐ When an experimental condition is added to an ERCT, usually an increase in participants is needed to maintain power.
MULTIPLE CHOICE: All else being equal, increasing the number of levels of a factor to three or more typically requires an increase in the number of participants to maintain statistical power. Which of the following is the reason?
☐ Because the number of experimental conditions increases. (This is incorrect. EASE does not involve maximizing all four of these attributes of an intervention simultaneously.)
☑ Because the number of participants per level of the factor decreases. (This is incorrect. EASE does not involve balancing efficiency against affordability and scalability.)
Feedback when incorrect: This is incorrect. All else being equal, an increase in the number of experimental conditions alone does not decrease power in a factorial experiment, as long as the sample size per level of each factor remains the same.
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☐ Like the ERCT, factorial experiments include a single particular experimental condition designated as the control condition. (This is incorrect. According to the definition of pilot study we are using in this course, investigators should not power pilot studies at all.)
☐ An investigator using a factorial design must identify a single control treatment that will work for all the factors in the experiment. (This is incorrect. Optimization trials are not pilot studies, although optimization trials should be pilot tested.)
☑ In a factorial experiment, each factor has its own control.
☑ In a factorial experiment, an individual participant may simultaneously be in the treatment condition of certain factors and the control condition of others. (This is incorrect. According to the definition of pilot study we are using in this course, the fundamental purpose of a pilot study is to establish feasibility of an approach that is under consideration for use in a larger study.)
☐ It is not possible to use a placebo control, such as an attention placebo, in a factorial experiment.
Feedback when incorrect: Like the ERCT, factorial experiments include a single particular experimental condition designated as the control condition.This is incorrect. The logic of experimental control differs between the ERCT and the factorial experiment. Factorial experiments typically do not include a single particular experimental condition designated as the control condition.An investigator using a factorial design must identify a single control treatment that will work for all the factors in the experiment.This is incorrect. Because in a factorial experiment each factor has its own control, different types of controls may be used for different factors as appropriate.It is not possible to use a placebo control, such as an attention placebo, in a factorial experiment.This is incorrect. Any type of control is possible in a factorial experiment.
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☑ In a fixed intervention, all participants are offered the same treatment.
☑ In an adaptive intervention, different participants may be offered different types and/or doses of treatment.
☐ Intensity of adaptation refers to the intensity of the treatment dosages used in an adaptive intervention.
☑ The decision rules in an adaptive intervention are always pre-specified.
Feedback when incorrect: Intensity of adaptation refers to the intensity of the treatment dosages used in an adaptive intervention.This is incorrect. Intensity of adaptation refers to the degree to which an adaptive intervention has many decision points and/or short review intervals. It has nothing to do with dosage.
MULTIPLE RESPONSE: Which of the following statements are correct? Select all that apply.
☐ The design of an optimization trial must be selected based on the continual optimization principle.
☑ Optimization of fixed and adaptive interventions often require addressing different types of research questions.
☐ The factorial experiment is used in optimization of fixed interventions only.
☑ The sequential multiple-assignment randomized trial is an experimental design for optimization of adaptive interventions.
☑ Optimization of adaptive interventions with different intensities of adaptation may require different optimization trial designs.
Feedback when incorrect: The design of an optimization trial must be selected based on the continual optimization principle.This is incorrect. The design of an optimization trial must be selected based on the resource management principle.The factorial experiment is used in optimization of fixed interventions only.This is incorrect. The factorial experiment may be used in optimization of adaptive interventions, depending on what research questions are to be addressed.
Synthesis (my own reading: not part of the course)
For readers who work with the 2021 MRC/NIHR framework for developing and evaluating complex interventions, this module is the payoff of the framework's “identify and optimise” language: the factorial experiment is the most economical way to answer the question an evaluation trial cannot, which components earn their place? The MRC framework lists factorial designs among the options for optimisation; Module 3 is the reason why the factorial (and especially the 2^k variant) is the default. The statistical core is worth internalising precisely: power in a balanced factorial is a function of the N per level of each factor, which is why a 2⁴ with 22 per condition can be well-powered while a two-arm trial with 22 per arm would not be. This is the single most common source of reviewer confusion about factorial optimization trials, a small per-condition n is not evidence of low power in a balanced factorial, and a reviewer (or a reviewer-facing proposal) should be told so explicitly.
For behaviour-change readers, the fixed-versus-adaptive distinction maps onto a familiar split: most behaviour change interventions evaluated as packages are fixed intervention designs, and their optimisation questions (“how well does this component perform?”, “is the high dose better than the low dose?”) are exactly what a factorial or fractional factorial trial answers. Adaptive interventions, sequential multiple-assignment (SMART), micro-randomized (MRT), or control-engineering (system identification) designs, are the same logic applied when the intervention itself is sequenced or tailored. The resource management principle disciplines the choice: you are not choosing a favourite design; you are stating the questions, then choosing the most efficient design that answers them.
One translation note for grant writing: Module 3's “factorial experiment” is the complete factorial; the course explicitly defers fractional factorial designs to Chapter 5 of Collins (2018), when a complete factorial is logistically impossible (large k), the fractional factorial is the resource-management answer, and this module's efficiency arguments are the reason it is worth reading further. Likewise, the course's emphasis on effect coding (-1, 1) is a substantive choice, not notation preference: under dummy coding, main effects and interactions in a factorial ANOVA are generally not interpretable in the standard way, and the nice properties (uncorrelated estimates, equal standard errors, equal power) hold under effect coding with (approximate) balance.
Glossary
Optimization trial
An experiment conducted to gather information needed for optimization; its objective is efficient assessment of intervention components, how each performs on average and in combination with others.
Resource management principle
The principle that the investigator must make the best and most efficient use of available resources when obtaining scientific information; any experimental design is okay as long as it is the most efficient one, the one that best manages research resources and most directly addresses the primary research questions.
Intervention EASE
A strategic balance of the four desiderata for multicomponent interventions, effectiveness against affordability, scalability, and efficiency.
Factorial experiment
An experiment in which several independent variables (factors) are manipulated in a coordinated fashion; the purpose is estimation of main effects and interactions.
Factor
An independent variable manipulated in a factorial experiment; in a 2^k experiment each of the k factors has two levels.
Main effect (of a factor)
The effect of a factor averaged over all the levels of all the other factors in the experiment; the mean of the conditions at one level minus the mean of the conditions at the other level.
Interaction (of two factors)
Whether the effect of one factor is the same across the levels of the other; if it varies, there is an interaction.
2^k experiment
A factorial experiment with k factors, each with two and only two levels; the design with 2^k experimental conditions.
Effect coding
Numerical representation of factor levels used in this course: for two levels, -1 and 1. Under effect coding, the standard textbook definitions of main effects and interactions apply.
Dummy (reference category) coding
Numerical representation using 0 and 1 for two levels; omnibus F is the same as under effect coding, but main effects and interactions generally should not be interpreted in the standard way.
Balance
The reason for the efficiency of factorial experiments: (a) each level of each factor appears the same number of times at each level of every other factor; (b) all experimental conditions have the same sample size (approximate balance is usually acceptable).
N per level
The sample size at each level of a factor; in a 2^k experiment, power for main effects is driven by N per level, not per-condition n.
Experimental design
The design of an experiment to gather information needed to develop, optimize, or evaluate an intervention.
Intervention design
The specific details of the approach taken by an intervention: components, component settings, eligibility requirements, etc.
Fixed intervention
An intervention design in which all participants are offered the same treatment.
Adaptive intervention
A sequence of pre-specified decision rules guiding whether, how, or when, and based on which measures, to alter an intervention or component at critical decision points during the course of care.
Intensity of adaptation
The degree to which an adaptive intervention has many decision points and/or short review intervals; distinct from intensity of dosage.
Decision point
A point during the course of care at which the pre-specified decision rules may alter the intervention.
Tailoring variable
The measure on which each participant's value, together with the decision rules, determines what additional treatment is provided.
SMART
Sequential multiple-assignment randomized trial: an experimental design for optimization of adaptive interventions, useful primarily when the adaptive intervention has low intensity of adaptation.
Micro-randomized trial (MRT)
An experimental design for optimization of adaptive interventions with high intensity of adaptation.
System identification
An experimental approach from control engineering for optimizing adaptive interventions with high intensity of adaptation.
Sources
Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Set reading: Chapter 3 for Lessons 1 to 6; Chapter 8 for Lessons 7 to 8. Cited in Lessons 2, 4, 7.)
Collins, L. M., & Kugler, K. C. (Eds.). (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer. (Cited in Lesson 2.)
Almirall, D., Nahum-Shani, I., Wang, L., & Kasari, C. (2018). Experimental designs for research on adaptive interventions: Singly and sequentially randomized trials. In Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer. (Cited in Lessons 7 and 8 for the adaptive-intervention definition and the SMART.)
Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202 to 224. (Cited in Lesson 3 for the subtly different definitions of the main effect across design options.)
Fisher, R. A. (1971). The design of experiments. New York: Hafner Publishing. (Quoted in Lesson 3, p. 102.)
Klasnja, P., Hekler, E. B., Shiffman, S., Boruvka, A., Almirall, D., Tewari, A., & Murphy, S. A. (2015). Microrandomized trials: An experimental design for developing just-in-time adaptive interventions. Health Psychology, 34(S), 1220. (Cited in Lesson 8.)
Rivera, D. E., Hekler, E. B., Savage, J. S., & Downs, D. S. (2018). Intensively adaptive interventions using control systems engineering: Two illustrative examples. In Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer. (Cited in Lesson 8.)
Kugler, K. C., Dziak, J. J., & Trail, J. B. (2018). (Cited in Lesson 2 for coding and interpretation of effects in factorial ANOVA.)
Video list
Lesson 1, Importance of the Resource Management Principle, qrct4vnZ6s8 (10:46)
Lesson 2, Introduction to the Factorial Experiment, 5XF_6Td1kAQ (12:40)
Lesson 3, Fundamental Differences between the Factorial Experiment and the ERCT, OSMb_VUj5IE (11:47)
Lesson 4, Why Factorial Experiments Can Be So Economical, dbqCJiPDBgg (13:05)
Lesson 5, Why the 2^k Experiment is So Economical, evwbfJ3rZ14 (7:52)
Lesson 6, Experimental Control in a Factorial Design, 2c318ELkQf0 (3:45)
Lesson 7, Brief Introduction to Adaptive Interventions, J5nm-Rpqlmw (10:10)
Lesson 8, Different Interventions, Different Optimization Trial Designs, x9NnhOpqnHA (9:07)
Prepared 4 Sep 2026. Notes paraphrase the narration unless marked as verbatim quotes; figures are redrawn from the course slides. Slide decks and the full question JSON are in the module package. Total video time: 79:12.