← MOST study notes index · CADIO / NYU School of Global Public Health
MOST from a Methodological Perspective: Module 5
Study notes · CADIO / NYU School of Global Public Health
Module 5: “Additional Optimization Trial Designs”
Authors: Linda M. Collins and Kate Guastaferro (School of Global Public Health, New York University)
Format: 6 lessons (Lesson 1, set reading, Lessons 2 to 5) · 5 videos · 5 slide decks (PDF) · 5 knowledge checks
Set reading: Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 8, plus the cluster, adaptive and hybrid-design references listed under the set-reading lesson)
Source anomalies found in Module 5 (flagged, not silently corrected). (1) The Lesson 3 and Lesson 4 knowledge checks are true/false items for which the course marks both “True” and “False” as correct; both checkboxes are reproduced exactly as in course.json and each item is flagged, not endorsed. (2) The caption transcripts contain transcription artefacts, for example “Corsera” for Coursera, “Guapero” for Guastaferro, “sessation” for cessation, “lozenes” for lozenges, “Zakadall” for Dziak (the deck slide 17 cites Dziak et al., 2012 and Murray, 1998), “Nahome Shai”/“Nah Shai” for Nahum-Shani, “Almaral” for Almirall, “Casari” for Kasari, “SMRT”/“smart” for SMART, “youmish.edu” for umich.edu, “perle n” for per-level n, “curvalinear” for curvilinear, “Tanner at all”/“tenor at all” for Tanner et al., “MIAL 0” for MI = 0, “1 in6” for 1/16, and “about 350, 880 or,400” for the deck’s 352, 880 and 1,408; these are preserved inside narration quotes and noted where they occur.
The module's argument in one paragraph.
This module widens the design repertoire of the optimization phase. Three strands run through it. First, cluster structure: data are frequently nested (schools, clinics, families), and nesting matters in two ways, it opens the door to contamination between experimental conditions, and, when whole clusters are assigned to conditions, it inflates the sample size required for power through the design effect D = 1 + (n − 1) × ICC_X × ICC_Y (Lesson 1). The design effect is benign under within-cluster assignment with equal assignment probabilities (ICC_X = 0, so D = 1, provided contamination is negligible and the intervention targets individuals), and it explodes under between-cluster assignment (ICC_X = 1), where larger clusters and larger outcome ICCs push D up; the practical remedies are replicating the whole experiment within each cluster and, where necessary, fractional factorial designs. Second, adaptive interventions, sequences of pre-specified decision rules, change the research questions, and therefore the design: factorial and fractional factorial experiments suit fixed interventions; SMARTs suit adaptive interventions with low intensity of adaptation; MRTs and system identification experiments suit higher intensities; and the choice must start from the scientific questions and the resource management principle (Lessons 2 and 3). Third, two design families that look attractive but are resource-hungry: dose-response questions, where adding a level to a factor usually costs at least 50% more participants and establishing the dose can cost far more (Lesson 4), and hybrid evaluation-optimization designs, which combine an evaluation RCT with a factorial test of components to be added to a previously optimized intervention, and can speed up the pipeline but do not support modelling the removal of components (Lesson 5).
Lesson 1: Recognizing and Dealing with a Cluster Structure
The lesson defines cluster structure (nesting), distinguishes pre-existing from experiment-induced clusters, explains the two threats clustering poses (contamination; interventions that target the cluster level), and works through the consequence for statistical power: it depends on whether randomisation happens within or between clusters, and is quantified by the design effect D with its two intraclass correlations.
Cluster structure (nesting) defined: “It is also called nesting.” “A cluster structure occurs whenever participants are members of clusters such as schools, neighborhoods, medical practices, and the like.” “Participants within clusters may tend to resemble each other more than they resemble participants sampled at random from the overall target population.”Narration, Lesson 1; deck slides 4 to 5
How clustering can arise: “In most situations, participants are not randomly assigned to their cluster.” “Or even if participants are randomly assigned to a cluster, such as a therapy group, an experimental treatment may cause them to become more similar to each other.” “For example, participants randomly assigned to an ongoing support group for HIV treatment regimen adherence might influence each other over time.”Narration, Lesson 1; deck slides 5 to 7
Pre-existing and experiment-induced clusters: “Clusters may be pre-existing that is existing before the optimization trial and experiment induced that is established by the optimization trial.” “Examples of pre-existing clusters include clinics or medical practices, schools or classrooms, families, faith congregations, communities.” “Experiment induced clusters are formed as part of an experiment.” “suppose one candidate component is a treatment that is delivered in a group setting. In this case, participants may become more similar over the course of treatment because of group interaction.” “In fact, the more successful the treatment is, the more likely it is that the participants will become more similar.”Narration, Lesson 1; deck slides 6 to 7
Why clustering matters 1: contamination: “if not handled properly, clustering can result in contamination between experimental conditions.” “Contamination occurs when treatment quote unquote bleeds across experimental conditions.” “That is when information or resources from a treatment are shared with the control group.” “Contamination tends to reduce treatment effects.”Narration, Lesson 1; deck slides 8 to 10
The married-couple example: “Suppose a married couple are both participants in an optimization trial in the smoking sessation area.” “One spouse is assigned to a treatment in which nicotine lozenes are provided and the other spouse is assigned to a control condition that is not provided with any nicotine replacement therapy.” “Suppose the spouse in the treatment condition gives some of the nicotine lozenes to the spouse in the control condition.” “You can see the kind of havoc this would wreak on an experiment.”Narration, Lesson 1; deck slides 8 to 10
Why clustering matters 2: the level the intervention targets: “The intervention may be aimed at both the individual and cluster level or even solely at the cluster level.” “For example, consider a school-based drug abuse prevention program delivered in classrooms.” “If there could be a substantial amount of contamination or the intervention is aimed at both the individual and cluster level or solely at the cluster level then it does not make sense for individuals within a cluster to be in different experimental conditions.”Narration, Lesson 1; deck slides 11 to 12
Two types of random assignment: “There are two types of random assignment within cluster and between cluster.” “In within cluster randomization, also called individual randomization, an individual in a given cluster may be assigned to any experimental condition.” “In between cluster randomization, also called cluster randomization, entire clusters are assigned to experimental conditions.” “A cluster may be assigned to any condition, but all individuals in a cluster are in the same condition.”Narration, Lesson 1; deck slide 13
The primary-care practices example: “If within cluster randomization is used, each individual patient is randomly assigned to an experimental condition.” “Within a single primary care practice, there are patients assigned to any of the different experimental conditions.” “By contrast, if between cluster randomization is used, each practice is randomly assigned to an experimental condition.” “Within a primary care practice, all of the patients are in a single experimental condition.”Narration, Lesson 1; deck slide 14
The effect of clustering on power: “The effect of clustering on statistical power depends on which type of random assignment is used.” “If you use within cluster random assignment and it is safe to assume there will be little or no contamination, the cluster structure has little effect on power.” “If you use between cluster random assignment, cluster structure can have a huge effect on power.”Narration, Lesson 1; deck slides 15 to 16
The design effect D: “Now, the design effect D is a multiplier that is used to determine what sample size is needed in the presence of clustering.” “If you would need n participants without a cluster structure, you need d * n with a cluster structure.” “suppose a power analysis done ignoring cluster structure indicates you need a sample size of 400, but you have a cluster structure and you compute D and find it is 1.2. This means you would need 400 * 1.2 = 480 participants.”Narration, Lesson 1; deck slides 16 to 17
Intraclass correlation (ICC): “the degree of within cluster similarity in the outcome Y is expressed in a statistic called the intraclass correlation.”Narration, Lesson 1; deck slide 18
The formula for D: “the design effect D is equal to 1 + the quantity N -1 * the product of the interclass correlation of the X's and the interclass correlation of the Y's.” “Here n is the number per cluster.” “So you use an approximate typical N for power.” (The narration says “interclass” where the deck slides 19 to 24 write “intraclass”; both refer to the intraclass correlations of the predictors and the outcome.)Narration, Lesson 1; deck slide 19
Why the ICC of X is usually omitted: “You may have seen the expression for D previously, but without the interclass correlation for the X's.” “If the experiment in question is an ERCT, there's only one X variable, the variable that represents the treatment control assignment.” “Thus, the ICC for the X variable would be one and so it is usually left out of the expression for simplicity.” “But as we have seen when the experiment being powered is a factorial experiment there are lots of X variables that is the codes representing the main effects and interactions.”Narration, Lesson 1; deck slide 19
Within-cluster assignment implies ICC_X = 0, so D = 1: “It turns out that with within cluster randomization, if all participants have the same assignment probabilities, the interclass correlation for the X's equals zero.” “One way to think about this is that knowing a participant's cluster membership does not tell you anything about what experimental condition they are in.” “So the second term of the expression for D becomes zero and therefore D equals 1.” “The conclusion is that the cluster structure has little to no effect on power.”Narration, Lesson 1; deck slides 20 to 23
Ensuring equal assignment probabilities: “Essentially this amounts to replicating the entire experiment within each cluster.” “suppose you are conducting a 2 to the 4th experiment in five different clinics.” “Then you would implement the entire 2 to the 4th experiment in each of the clinics.” “A participant in any clinic would have a 1 in6 chance of being assigned to any one of the experimental conditions.”Narration, Lesson 1; deck slides 21 to 22
The caveat on within-cluster assignment: “we are assuming that contamination is not a problem.” “If it is, power may be reduced but not through the design effect.” “Instead, power would be reduced because the effect size would be smaller.”Narration, Lesson 1; deck slides 20 to 23
Between-cluster assignment implies ICC_X = 1: “When between cluster random assignment is used, cluster membership determines experimental condition.” “Therefore, the interclass correlation of the x's is one.” “So, the expression for d becomes the one you may be more familiar with.” “That is d = 1 plus the quantity n minus one * the interclass correlation of y.”Narration, Lesson 1; deck slide 24
What makes D larger: “All else being equal, D is larger when the cluster size is larger.” “This means that in studies where larger clusters are being assigned to experimental conditions such as entire schools, the design effect is greater than it is where smaller clusters such as families are assigned.” “D is also of course larger when the interclass correlation of Y is larger.”Narration, Lesson 1; deck slides 24 to 25
Two considerations for factorial optimization trials with between-cluster assignment: “The first consideration is will I have enough power.” “How you go about determining whether you will have enough power in the presence of clustering is essentially the same for a factorial optimization trial as it would be for an ERCT.” “The second consideration is given that I have enough power will I have enough clusters to assign to experimental conditions.” “You can assign as few as one cluster per condition, but it is a good idea to assign at least two clusters per condition.” “Then, if an entire cluster drops out, you won't have an empty cell.”Narration, Lesson 1; deck slides 26 to 27
The fractional factorial remedy: “You may want to consider a fractional factorial design.” “In a fractional factorial design, you include a subset of the experimental conditions.” “It's a subset that is carefully selected to preserve the balance properties of the factorial experiment.” “This will reduce the number of experimental conditions that need to be populated.” “Fractional factorial designs make exactly the same sample size requirements as complete factorial experiments.” “However, this number of clusters would be spread across fewer experimental conditions.”Narration, Lesson 1; deck slide 27
Experiment-induced clustering in factorial trials: “Experimentinduced clustering introduces some special considerations.” “For example, if you are randomly assigning participants to clusters, you may be able to control cluster size.” “Or you may have clustering in only one level of a factor.” “Experiment-induced clustering in both optimization trials and ERCTs is an open research area.”Narration, Lesson 1; deck slide 28
Lesson 1 references: “It is covered in depth in Murray's 1998 book which is a classic of course.” “For more information, I refer you to the journal article by Zakadall published in 2012.” “see the Nahome Shai and Zak chapter and the Nah Shai at all article both published in 2018.” (Caption artefacts: “Zakadall” is Dziak, deck slide 17 cites Dziak et al., 2012; “Nahome Shai”/“Nah Shai” is Nahum-Shani.)Narration, Lesson 1; deck slide 31
Figure 1: Cluster structure and the design effect (redrawn)
| Element | Within-cluster randomisation (individual) | Between-cluster randomisation (cluster) |
|---|
| Who is randomised | Each individual; a cluster can contain several conditions | Entire clusters; everyone in a cluster shares one condition |
| Cluster membership and condition | Reveals nothing about condition | Determines condition |
| ICC of the X variables | ICC_X = 0 when assignment probabilities are equal (replicate the whole experiment in each cluster) | ICC_X = 1 |
| Design effect | D = 1, so little to no effect on power (given negligible contamination) | D = 1 + (n − 1) × ICC_Y; can be a huge effect on power |
| What drives D | - | Larger number per cluster and larger ICC_Y make D larger (schools larger than families) |
Redrawn from Lesson 1 deck slides 4 to 25. The general expression shown on deck slide 19 is D = 1 + (n − 1) × ICC_X × ICC_Y, with n the number per cluster. Example from slides 16 to 17: if a power analysis ignoring clustering needs N = 400 and D = 1.2, the study needs 400 × 1.2 = 480 participants.
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are true? There may be more than one.
- ☐ Clustering in data occurs only when participants are in pre-existing groups, such as families or schools.
- ☑ Contamination between experimental conditions tends to reduce treatment effects.
- ☑ When conducting within-cluster random assignment it is a good idea to replicate the entire experiment within each cluster.
- ☐ Within-cluster random assignment can help prevent contamination.
Feedback when incorrect (source text, retained as provided): Answer A. is incorrect. Clustering can also be experiment-induced.
Answer D. is incorrect. Between-cluster random assignment can help prevent contamination.
Reading for All of Module 5
This lesson has no video or slide deck; it lists the set reading for the module (spacing normalised; names as listed by the course).
- Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 8 in the course textbook.)
- Dziak, J. J., Nahum-Shani, I., & Collins, L. M. (2012). Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations. Psychological Methods, 17, 153 to 175.
- Nahum-Shani, I., Dziak, J. J., & Collins, L. M. (2018). Multilevel factorial designs with experiment-induced clustering. Psychological Methods, 23, 458 to 479.
- Almirall, D., Nahum-Shani, I., Wang, L., & Kasari, C. (2018). Experimental designs for research on adaptive interventions: Singly and sequentially randomized trials. In Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer.
- Klasnja, P., Hekler, E. B., Shiffman, S., Boruvka, A., Almirall, D., Tewari, A., & Murphy, S. A. (2015). Microrandomized trials: An experimental design for developing just-in-time adaptive interventions. Health Psychology, 34(S), 1220.
- Murray, D. M. (1998). Design and analysis of group-randomized trials. Oxford: Oxford University Press.
- Nahum-Shani, I., & Dziak, J. J. (2018). Multilevel factorial designs in intervention development. In L. M. Collins and K. C. Kugler (eds.), The multiphase optimization strategy (MOST): Advanced topics. New York: Springer.
- Rivera, D. E., Hekler, E. B., Savage, J. S., & Downs, D. S. (2018). Intensively adaptive interventions using control systems engineering: Two illustrative examples. In Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer.
- Tanner, A. E., Guastaferro, K. M., Rulison, K., Wyrick, D. L., Milroy, J. M., Bhandari, S., Thorpe, S. Y., Ware, S., Miller, A. M., & Collins, L. M. (2021). A hybrid evaluation-optimization trial to evaluate an intervention targeting the intersection of alcohol and sex in college students and simultaneously test an additional component aimed at preventing sexual violence. Annals of Behavioral Medicine, 55, 1184 to 1187.
Lesson 2: Brief Introduction to Adaptive Interventions
The lesson draws the distinction between experimental design and intervention design, contrasts fixed and adaptive interventions, walks through an adaptive HIV-adherence example (decision rule at Day 45), defines the adaptive intervention formally, and introduces intensity of adaptation.
Experimental design versus intervention design: “The distinction between experimental design and intervention design.” “Experimental design in this context is the design of an experiment to gather information needed to develop, optimize or evaluate an intervention.” “By contrast, intervention design is the specific details of the approach taken by an intervention, including the intervention components, the settings or levels of the components, any eligibility requirements for participants, etc.” “you might be surprised at how often these two types of designs are confused, particularly in conversation.”Narration, Lesson 2; deck slides 6 to 7 (slide 7 cites Collins, 2018, p. 291)
Fixed interventions: “The design of a fixed intervention calls for all participants to be offered the same treatment.” “Note that participants may not all receive the same treatment.” “A participant may choose not to participate in some aspect of treatment, for example, even though this aspect of treatment is included in the intervention design.”Narration, Lesson 2; deck slides 8 to 9
Adaptive interventions: “in adaptive interventions, intervention components and component levels are varied strategically across participants.” “The premise here is that different people may have better outcomes with different amounts or types of intervention.” “An adaptive intervention is designed to give each individual the amount and type of intervention needed to achieve the desired outcome.” “And this part is important, no more intervention than is needed.”Narration, Lesson 2; deck slides 10 to 11
Why not give every participant everything: “You could ensure that every participant gets the amount and type of intervention needed by providing all participants with every type of intervention at the maximum level.” “However, this would be exhausting for the participants and a waste of resources that could be deployed elsewhere.”Narration, Lesson 2; deck slide 11
The adaptive example (an intervention to encourage adherence to HIV treatment): “In this intervention, all participants receive text messaging, peer mentoring, and behavioral skills training at the outset.” “Adherence is then monitored for 45 days.” “On day 45, it is determined whether or not each participant has been adherent.” “Those who have been adherent continue with text messaging only and no longer receive any further peer mentoring or behavioral skills training.” “Those who are not yet adherent continue with text messaging and peer mentoring, no longer receive behavioral skills training, and are provided with motivational interviewing.”Narration, Lesson 2; deck slides 12, 14 to 16
Definition of an adaptive intervention (course source): “An adaptive intervention is a sequence of prespecified decision rules that can be used to guide whether, how, or when, and based on which measures to alter an intervention or intervention component, eg treatment type, duration, frequency, or amount at critical decision points during the course of care.” (The lesson attributes this definition to the 2018 chapter by “Almaral, Nahum, Shani, Wang, and Casari”, i.e. Almirall, Nahum-Shani, Wang and Kasari, 2018.)Narration, Lesson 2; deck slide 13
The pre-specified decision rules in the example: “First, at the outset, all participants are provided with text messaging for the duration of the intervention and are provided with peer mentoring and behavioral skills training for 45 days.” “Starting on day 45, any participants who are determined to be adherent based on the assessment will continue to be provided with text messaging only.” “Any participants who are determined to be non-adherent based on the assessment will continue to be provided with text messaging and peer mentoring and in addition will begin to be provided with motivational interviewing.”Narration, Lesson 2; deck slides 14 to 16
Tailoring variable: “Note that the decision rule expresses a predetermined strategy for assigning treatment based on a tailoring variable which in this case is a measure of an intermediate outcome.”Narration, Lesson 2; deck slides 14 to 16
Intensity of adaptation: “Intensity of adaptation is the degree to which an adaptive intervention has many decision points and or short review intervals.” “You can think of this as how rapidly and intensively the adaptation takes place.” “Our example has very low intensity of adaptation.” “There is only one decision point and it occurs 45 days into the intervention.” “An example of an intervention with a higher intensity of adaptation would be the same intervention but with a weekly decision point.”Narration, Lesson 2; deck slides 17 to 18
Intensity of adaptation is not intensity of dosage: “Intensity of adaptation is different from intensity of dosage.” “An intervention can employ a high intensity of adaptation to deliver a low dose or light touch intervention.” “Interventions delivered via devices, for example, smartphones, often combine high intensity of adaptation with a light touch treatment.” “For example, there may be several decision points per day, but the treatment may be as simple as a brief text message.”Narration, Lesson 2; deck slides 17 to 20
Further resources: “I encourage you to take a look at the website of the D3 center at the University of Michigan.” “It has lots of valuable resources for people interested in adaptive interventions.” “The website can be found at d3c.isr.youmish.edu” (caption artefact: “youmish.edu” for umich.edu).Narration, Lesson 2; deck slides 17 to 20
Figure 2: The adaptive intervention example (redrawn)
| Stage | What happens |
|---|
| At the outset | All participants receive text messaging (indefinitely), plus peer mentoring and behavioural skills training for 45 days |
| Day 45 assessment | Adherence to the HIV treatment protocol (the tailoring variable) is assessed; behavioural skills training ceases |
| Decision rule, adherent | Continue with text messaging only |
| Decision rule, non-adherent | Continue with text messaging and peer mentoring, and in addition begin motivational interviewing |
Redrawn from Lesson 2 deck slides 12 and 14 to 16 (diagram reprinted in the course from Collins, 2018). The example has one decision point at Day 45, so it has very low intensity of adaptation; a weekly decision point would be a higher-intensity version (slides 17 to 18).
Knowledge check
MULTIPLE RESPONSE: Which of the following statements about adaptive interventions are true? There may be more than one.
- ☐ In an adaptive intervention, the intervention may be adjusted on an ad hoc basis, based on the judgement of implementation staff.
- ☑ An adaptive intervention is designed to give each individual the amount and type of treatment needed, and no more.
- ☑ Adaptive interventions involve tailoring variables and decision rules.
Feedback when incorrect (source text, retained as provided): Answer A. is incorrect. Adaptive interventions are not adjusted on an ad hoc basis; rather, they rely on prespecified tailoring variables and decision rules.
Lesson 3: Different Types of Interventions, Different Optimization Trial Designs
The lesson reviews the resource management principle, lists the aspects of the adaptive example that could be treated as components, and shows how the type of intervention (fixed or adaptive, and the intensity of adaptation) points towards different designs: factorial and fractional factorial experiments, SMARTs, MRTs and system identification experiments.
The resource management principle reviewed: “In module one, you learned that an investigator using most must strive to make the best and most efficient use of available resources when obtaining scientific information.” “Any experimental design is okay as long as it is the most efficient one.” “The most efficient design for an optimization trial is the one that best manages research resources.” “We discussed this in a previous lesson and most directly addresses the primary research questions.”Narration, Lesson 3; deck slides 6 to 7
Design choice depends on the intervention type: “Which experimental design directly addresses the primary research questions depends partly on the type of intervention to be optimized.” “For a fixed intervention, the question is usually how well does this component perform or is the high dose better than the low dose of this component.” “For these types of research questions, factorial and fractional factorial experiments often are appropriate.”Narration, Lesson 3; deck slides 9, 11
What can count as a component in the adaptive example: “Remember that anything that can usefully be separated out for scientific study can be considered a component.” “Another component to consider is the tailoring variable itself.” “The decision point can also be considered a component.” “If there is more than one decision point, another potential component is the length of the interval between decision points. Sometimes called the review interval.”Narration, Lesson 3; deck slide 8
Adaptive question 1: “One question might be, is the initial treatment, which in the example consists of text messaging, peer mentoring, and behavioral skills training, effective overall, does it outperform a comparison or control treatment?” “Addressing question one requires randomly assigning participants at the outset to the initial treatment or a control.”Narration, Lesson 3; deck slides 11 to 13
Adaptive question 2: “A second question might be for participants who are adherent after 45 days, which is a better second line treatment continue with text messaging only, which is what we had in the hypothetical example, or continue with text messaging and add peer mentoring.” “Addressing question two requires waiting until day 45, assessing adherence, and identifying adherent participants, and then randomizing only the adherent participants to either continue to receive text messaging only or continue with text messaging and add peer mentoring.”Narration, Lesson 3; deck slides 14 to 15
Adaptive question 3: “A third question might be for participants who are non-adherent after 45 days which is a better secondline treatment continue with text messaging plus peer mentoring only or continue with these treatments and add motivational interviewing which is the treatment we had in the hypothetical example.” “Question three requires waiting until day 45, assessing adherence and identifying non-adherent participants and randomizing only non-adherent participants to continue to receive text messaging and peer mentoring only or continue to receive text messaging and peer mentoring and add motivational interviewing.”Narration, Lesson 3; deck slides 16 to 17
SMART: “Sets of questions like these can be addressed using a sequential multiple assignment randomized trial, also known by the acronym SMRT.” “SMRTs are useful when the adaptive intervention being optimized has a relatively low intensity of adaptation.” (Caption artefact: “SMRT” for SMART, as printed on deck slide 18.)Narration, Lesson 3; deck slide 18
MRT and system identification experiments: “For adaptive interventions with higher intensity of adaptation, a good choice is often the micro randomized trial MRT or a system identification experiment.”Narration, Lesson 3; deck slide 19
Factorial experiments as the foundation: “In this course, we are emphasizing the factorial optimization trial because first it remains the most widely applicable optimization trial and second many other optimization trial designs are rooted in the factorial experiment.” “The fractional factorial, smart and micro randomized trial are all rooted in the factorial experiment.” “So to understand these designs well, it is necessary to understand the factorial experiment.”Narration, Lesson 3; deck slide 20
Which design for which intervention: “smarts and MRTs are often good choices for optimization of adaptive interventions.” “However, factorial and fractional factorial experiments are sometimes the appropriate choice for optimization of adaptive interventions depending on the research questions at hand.”Narration, Lesson 3; deck slide 22
The selection rule: “When selecting an experimental design for an optimization trial, remember optimization of different types of interventions can generate different types of scientific questions.” “For different types of scientific questions, the resource management principle may suggest different experimental designs.” “start with the scientific questions, then use the resource management principle to select the most appropriate and efficient experimental design.” “If the scientific questions are sufficiently clearly and explicitly stated, selection of the appropriate experimental design is usually straightforward.”Narration, Lesson 3; deck slides 21 to 23
Figure 3: From intervention type to candidate optimisation trial designs (redrawn)
| Intervention type / research question | Designs the lesson points to |
|---|
| Fixed intervention, how well does this component perform, or is the high dose better than the low dose? | Factorial and fractional factorial experiments are often appropriate |
| Adaptive intervention, low intensity of adaptation, initial-treatment and second-line questions answered by randomising at the outset and at the decision point | Sequential multiple assignment randomized trial (SMART) |
| Adaptive intervention, higher intensity of adaptation | Micro-randomised trial (MRT) or system identification experiment |
| Any intervention, selection rule | Start with the scientific questions; use the resource management principle to select the most appropriate and efficient design; factorial and fractional factorial experiments remain options for adaptive interventions depending on the questions |
Redrawn from Lesson 3 deck slides 6 to 23. The lesson stresses that fractional factorial, SMART and MRT designs are all rooted in the factorial experiment (slide 20).
Knowledge check
MULTIPLE CHOICE: Is the following statement true or false? Depending on the exact research questions and type of intervention, either a factorial experiment, SMART, or MRT may be used for optimization of an adaptive intervention.
Flagged source anomaly: for this true/false item the course marks both “True” and “False” as correct. Both marks are reproduced exactly as they appear in the course source; the item is preserved and flagged rather than silently corrected, and is not endorsed.
Lesson 4: Addressing Dose-Response Questions in an Optimization Trial
The lesson recaps the resource requirements of the design options, shows what adding a level to a factor costs, then treats dose-response questions in two modes: deciding whether to devote resources to dose at all, and establishing the dose. Both are resource-intensive.
The design options recap: “You've seen this table before. It shows the resource requirements of three different options for investigating the performance of these four intervention components.” “Depending on the design approach selected, you would need about 350, 880 or,400 participants and would implement 5 8 or 16 experimental conditions.” (Caption artefact: “or,400” for 1,408, the deck table gives 352, 880 and 1,408 participants for the factorial, comparative-treatment and individual-experiments options.)Narration, Lesson 4; deck slides 6 to 7
The four-factor factorial: “If you decided to conduct a factorial experiment, the design would look like this.” “The design would have 16 experimental conditions and there would be 22 participants per condition for a total n of 352.”Narration, Lesson 4; deck slides 7 to 8
Adding a level instead of a factor: “What would happen if instead of adding a factor, you wanted to add a level to a factor?” “For example, instead of no and yes, the level of text would be no, low, and high.” “The new design would be a 2x 2x 3x two. So it would have 24 experimental conditions.” “The 352 participants must now be divided among the 24 conditions.”Narration, Lesson 4; deck slides 8 to 12
Per-level sample size is what drives power: “what matters for statistical power is not the n per condition but the n per level of each factor.” “The N per level for MI, peer and mind is the same as it was before the additional level was added to text.” “That is the perle N for these main effects remains 176.” “However, things are different for the text factor which now has three levels.”Narration, Lesson 4; deck slides 11 to 14
Power falls for the three-level factor: “There are now two main effects associated with this factor.” “You can see that the perle n is now about 120 as compared to the previous perle n of 176. Power will be reduced.” “Unless the dose response relation is curvalinear, the low versus no comparison will likely have a smaller effect size than the high versus low comparison, further reducing power.”Narration, Lesson 4; deck slides 12 to 14
The cost of the extra level: “All else being equal, it will be necessary to obtain more participants.” “Suppose both main effects of text are expected to be about the same size as the other, eg the main effect of MI.” “This is an unreasonably optimistic scenario, but let's use it for the sake of argument.” “It would be necessary to increase the sample size by at least 50%.”Narration, Lesson 4; deck slides 15 to 17
Conclusions on factors versus levels: “First, it is possible to add one or more factors and maintain power without adding participants.” “So adding one or more factors to a factorial optimization trial is often an efficient strategy.” “Second, adding a level of a factor usually requires at least 50% more participants.” “So adding even one level to a factor can be costly.” “This does not mean don't ever use designs that have more than two levels per factor.”Narration, Lesson 4; deck slides 18 to 19
What dose-response means here: “In this lesson, when we talk about dose response, we are referring to establishing the appropriate dose of a component.” “For example, the appropriate number of supportive text messages per week, peer mentoring sessions, or motivational interviewing sessions.” “These kinds of questions call for a dose response experiment.”Narration, Lesson 4; deck slides 20 to 22
The six-level design: “Suppose you want to change this experiment so that you can establish both whether MI has a detectable effect and which is the appropriate dose of motivational interviewing 1 2 3 four or five sessions.” “This would mean the MI factor would have six levels 0 through 5 and the design is a 6x 2x 2x 2 with 48 experimental conditions.” “This would be seven or eight participants per condition.” “You can see immediately that if you do this for the MI factor, the perle n is now only 352 / 6 or approximately 59.” “Compare this to the 2 to the k design where the perle n was 176.”Narration, Lesson 4; deck slides 22 to 23
Approach one: none versus high: “In approach one, you compare the mean of the conditions in which MI equals zero to the mean of the conditions in which MI equals 5.” “This is essentially a none versus high comparison in which you leave out all conditions in which MI equals 1, 2, 3, or 4.” “This effect is probably closest to the main effect in the original 2 to the K experiment.”Narration, Lesson 4; deck slides 24 to 25
Approach one cost: “To achieve power comparable to the 2 to the K experiment, you need to bring the MI= 0 and MI= 5 levels up to the previous sample size.” “This means adding 117 participants to each of these levels. The total increase is at least 234. the overall n would be at least 586, a 66% increase.” “This design is now unbalanced unless you increase all levels of MI.” “So some effect estimates will be correlated and power will vary for estimation of some effects.”Narration, Lesson 4; deck slide 25
Approach two: none versus any: “The second approach is to compare the mean of the conditions in which MI equals zero to the mean of all other conditions.” “This is essentially a comparison of none versus any.” “The sample size per level is very uneven.” “n equals about 293 in the any level because any is an aggregate of the MI= 1 2 3 4 and 5 levels but only about 59 in the none MIAL 0 level” (caption artefact: “MIAL 0” for MI = 0).Narration, Lesson 4; deck slide 26
Approach two cost: “to achieve power comparable to the 2 to the k experiment you would need to increase the n in the mi=0 level to at least 176.” “So under unrealistically optimistic assumptions, an increase of at least 117 participants is required for a total n of 469, a 33% increase over the 2 to the k experiment.” “But this is probably a gross underestimate.”Narration, Lesson 4; deck slide 26
Dose-response needs adjacent-dosage comparisons: “To establish dose response, you have to compare the means of adjacent dosages MI= 2 to MI= 1, MI= 3 to MI= 2 and so on.” “What power will you have for these comparisons? Not considering an adjustment for type one error rate.” “This is likely to be relatively small in comparison to say the mi=0 to mi= 5 comparison.”Narration, Lesson 4; deck slides 27 to 28
Power under both approaches is weak: “If you took approach number one, you have n= 176 at two levels of MI. MI= 0 and MI= 5.” “If you assign fewer than N= 176, power will be correspondingly smaller.” “Most of the comparisons will involve a perle n of about 59. So it's unlikely power will be great.” “Note that both of these approaches treat dose response as secondary.”Narration, Lesson 4; deck slides 29 to 30
The two-experiment strategy: “one strategy is to plan on two experiments.” “You can start by using two levels to ascertain that there is a difference between high or whatever dose is expected to produce the best outcome and none.” “If there is no difference between high versus none, do you care about the intermediate levels? Usually, you don't.” “Once it has been established that there is a difference between high and none, you can examine intermediate levels in a subsequent experiment to find the right dose.”Narration, Lesson 4; deck slides 27 to 33
Dose-response as the primary question: “What if dose response is the primary research question?” “In our example, to maintain power again ignoring any correction for type one error rate, you would need n= 176 at each of the six levels of MI.” “Why? This is essentially a six arm ERCT.” “In my opinion, it probably does not make sense to do this unless you have established that the component has an effect.”Narration, Lesson 4; deck slides 29 to 33
Figure 4: The arithmetic of levels and doses (redrawn)
| Design | Conditions | Per-level n for the factor at issue | Note |
|---|
| 2 × 2 × 2 × 2 factorial (MI, PEER, TEXT, MIND each no/yes) | 16 | 176 per level (22 per condition, N = 352) | The reference design |
| 2 × 2 × 3 × 2 (TEXT becomes no/low/high) | 24 | TEXT falls to about 120 per level; other factors stay at 176 | Restoring power needs at least 50% more participants |
| 6 × 2 × 2 × 2 (MI at doses 0 to 5) | 48 | MI falls to about 59 per level | Approach one (0 vs 5): +234 participants (N ≥ 586, +66%), unbalanced unless all levels increased; approach two (0 vs any): +117 optimistic (N = 469, +33%), “probably a gross underestimate” |
Redrawn from Lesson 4 deck slides 6 to 7 and 13 to 30. The deck's options comparison (slides 6 to 7) gives about 352 participants for the 16-condition factorial, 880 for the 5-condition comparative-treatment option and 1,408 for four individual experiments (8 experimental conditions), with only the factorial allowing interactions to be estimated.
Knowledge check
MULTIPLE CHOICE: Is the following statement true or false? Addressing dose-response questions can be resource-intensive.
Flagged source anomaly: for this true/false item the course marks both “True” and “False” as correct. Both marks are reproduced exactly as they appear in the course source; the item is preserved and flagged rather than silently corrected, and is not endorsed.
Lesson 5: Hybrid Evaluation-Optimization Designs
The lesson introduces the continual optimization principle, defines the hybrid evaluation-optimization trial, works through the Tanner et al. (2021) proof-of-concept example (a 2 × 2 combining evaluation of an optimised intervention with a test of a candidate component), and states when hybrids are most useful.
The continual optimization principle: “According to the continual optimization principle, most offers the opportunity to keep improving an intervention by repeatedly conducting optimization trials and based on the results adding or replacing intervention components to achieve a stated optimization. objective.”Narration, Lesson 5; deck slide 3
Combining evaluation and optimization in one experiment: “Of course, the process of continual optimization can be very timeconsuming under some circumstances.” “One way to shorten the process of intervention optimization and evaluation is to conduct evaluation phase research and optimization phase research in a single experiment.” “This is not always possible, but it is sometimes possible and may be worth considering.”Narration, Lesson 5; deck slides 4 to 5
The Tanner et al. example: “Tanner at all 2021 had previously optimized an intervention to prevent excessive drinking and risky sex in college students.” “The next step was evaluation of the optimized intervention in an ERCT.” “they learned that college officials wanted them to add to the intervention a component aimed at prevention of sexual victimization.” “The team did not want to add a component to the intervention without assessing its performance.” “So they decided to conduct a hybrid evaluation optimization trial.”Narration, Lesson 5; deck slides 6 to 8
The 2 × 2 design: “This was a simple 2x two factorial experiment because it was intended to be a proof of concept of the idea of a hybrid evaluation optimization trial.” “one factor was essentially the ERCT.” “The levels of this factor are intervention and active control.” “The second factor is the sexual victimization component which is a candidate component with levels included and not included.”Narration, Lesson 5; deck slides 7 to 10
Effects the design can estimate: “This design enables estimation of the main effect of the optimized intervention.” “This would be the mean of experimental conditions 1 and two minus the mean of experimental conditions three and four.” “This design also enables estimation of the main effect of the candidate component. This would be the mean of experimental conditions 1 and three minus the mean of experimental conditions 2 and four.” “Finally, the design enables estimation of the interaction between these two factors.”Narration, Lesson 5; deck slides 7 to 10
Why the interaction matters before concluding: “Consideration of this interaction is important before drawing any conclusions about the effectiveness of the optimized intervention.”Narration, Lesson 5; deck slides 7 to 10
Proof of concept, and the more efficient design: “As I mentioned above, the tenor at all experiment provided proof of concept.” “A more efficient design would have included more than one candidate component.” “In fact, as you know by now, this experiment could readily have included several candidate components.”Narration, Lesson 5; deck slides 11 to 12
Adding, not substituting: “A hybrid evaluation optimization design is most useful when the candidate components are to be added to the previously optimized intervention rather than substituted for components currently in the intervention.” “Based on the results of the hybrid trial, it is possible to model what the outcome would be if you added the components you tested.” “It is not possible to model outcomes if you removed some components from the previously optimized intervention.” “To do that would require a different kind of experiment.”Narration, Lesson 5; deck slides 11 to 12
Speed, limits, and further hybrids: “A hybrid evaluation optimization trial can potentially dramatically speed up the intervention optimization process, but their usefulness is limited to situations in which the candidate components are to be added to the previously optimized intervention along with whatever components already make up the intervention.” “A lot of work is being done currently on combining different kinds of optimization trial designs to create hybrid designs.” “For example, people working in this area are developing hybrids of standard factorials and sequential multiple assignment randomized trials.”Narration, Lesson 5; deck slides 13 to 14
Figure 5: The hybrid evaluation-optimization 2 × 2 (redrawn)
| Experimental condition | Previously optimised intervention | Candidate component: sexual victimisation |
|---|
| 1 | Intervention | Included |
| 2 | Intervention | Not included |
| 3 | Active control | Included |
| 4 | Active control | Not included |
Redrawn from Lesson 5 deck slides 7 to 10. Main effect of the optimised intervention = (1 + 2) − (3 + 4); main effect of the sexual victimisation component = (1 + 3) − (2 + 4); interaction = [(1 − 3) − (2 − 4)] / 2. Based on Tanner et al. (2021).
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? There may be more than one.
- ☑ The continual optimization principle states that intervention optimization is an ongoing process of intervention improvement.
- ☑ A hybrid evaluation-optimization trial can sometimes be used to collect data needed for evaluation and optimization in a single experiment.
- ☑ An evaluation-optimization trial requires that the candidate components be under consideration for addition to the intervention, not for replacement of components currently in the intervention.
Feedback when incorrect (source text, retained as provided): You did not select all the correct responses.
Synthesis (my own reading: not part of the course)
Read together, this module is about designs the course treats as “also possible” rather than default, and the discipline the resource management principle imposes on choosing them. The cluster lesson is the technical core. Its logic is that clustering damages an experiment through two distinct channels, and only the second is the design effect. The contamination channel reduces the effect size and therefore power regardless of randomisation scheme; the design-effect channel operates only when cluster membership predicts condition, i.e. under between-cluster assignment, where D = 1 + (n − 1) × ICC_Y multiplies the sample size requirement. The within-cluster case is rescued by replicating the entire experiment in each cluster, which equalises assignment probabilities and makes ICC_X, and hence the design effect, zero. Two implications are easy to miss: first, the factorial structure is what makes the omitted ICC_X term matter at all, because a factorial has many X variables rather than one treatment indicator; second, the within-cluster remedy does nothing about contamination, so the preconditions are as important as the arithmetic.
The adaptive-intervention lessons reorganise the design menu around the questions rather than the machinery. A fixed intervention’s questions (“does this component work; is high better than low”) map naturally onto factorial and fractional factorial experiments; an adaptive intervention generates stage-by-stage questions (initial treatment, then second-line choices for adherent and non-adherent subgroups) that map onto SMARTs, with MRTs and system identification experiments for high-adaptation designs. The module’s repeated point, start with the scientific questions, then apply the resource management principle, is the reason it can say both that SMARTs and MRTs are often good choices for adaptive interventions and that factorial designs remain appropriate for adaptive interventions depending on the questions, and also the most widely applicable option overall.
The last two lessons price the exotic options. Dose-response work is expensive because it spends degrees of freedom on within-factor distinctions: adding one level to one factor converts 176-per-level structure into roughly 120-per-level for that factor, forces at least a 50% sample-size increase to restore power under optimistic assumptions, and a full dose-finding experiment (six MI levels) leaves about 59 per level, with adjacent-dosage comparisons the weakest of all. The lesson’s advice, establish first that a component has an effect, then find the dose in a second experiment, is a sequencing argument as much as a power argument. Hybrid evaluation-optimisation designs are the mirror image: cheap in time but narrow in scope. Because the factorial estimates the effect of adding the candidate component to the optimised intervention, removal cannot be modelled; the design is therefore a tool for components being trialled as additions, and its logic assumes the optimised intervention is itself being evaluated in the same experiment. The two flagged true/false anomalies (Lessons 3 and 4) both mark “True” and “False” correct; as with earlier modules, the notes preserve the source marks and flag the incoherence rather than guessing an intended key. Caption transcription artefacts (listed in the anomaly note above) are preserved inside quotes.
Glossary
- Cluster structure (nesting)
- Data structure in which participants are members of clusters, such as schools, neighbourhoods or medical practices, and participants within clusters may resemble each other more than participants sampled at random from the target population.
- Pre-existing cluster
- A cluster that exists before the optimisation trial (clinics, schools, families, faith congregations, communities).
- Experiment-induced cluster
- A cluster formed as part of an experiment, for example a group-delivered treatment that makes participants more similar over time; more likely the more successful the treatment is.
- Contamination
- Treatment “bleeding” across experimental conditions, i.e. information or resources from a treatment being shared with the control group; it tends to reduce treatment effects.
- Within-cluster randomisation
- Also called individual randomisation: an individual in a given cluster may be assigned to any experimental condition.
- Between-cluster randomisation
- Also called cluster randomisation: entire clusters are assigned to experimental conditions, and all individuals in a cluster share one condition.
- Design effect (D)
- A multiplier used to determine the sample size needed in the presence of clustering: D times the size needed without clustering. General form D = 1 + (n − 1) × ICC_X × ICC_Y; under between-cluster assignment it reduces to D = 1 + (n − 1) × ICC_Y, and under within-cluster assignment with equal assignment probabilities D = 1.
- Intraclass correlation (ICC)
- The degree of within-cluster similarity; ICC_Y for the outcome and ICC_X for the predictor codes (main effects and interactions).
- Fractional factorial design
- A design including a carefully selected subset of the experimental conditions that preserves the balance properties of the factorial experiment; it needs the same sample size as the complete factorial but spreads the required clusters across fewer conditions.
- Experimental design vs intervention design
- Experimental design is the design of an experiment to gather information needed to develop, optimise or evaluate an intervention; intervention design is the specific details of the intervention itself (components, settings or levels, eligibility requirements).
- Fixed intervention
- An intervention whose design calls for all participants to be offered the same treatment (some may not receive all of it).
- Adaptive intervention
- A sequence of pre-specified decision rules that guide whether, how, or when, and based on which measures, to alter an intervention or component (e.g. treatment type, duration, frequency or amount) at critical decision points during the course of care; designed to give each individual the amount and type needed, and no more.
- Decision rule
- A predetermined strategy for assigning treatment based on a tailoring variable.
- Tailoring variable
- The measure on which a decision rule depends; in the course example, adherence to the HIV treatment protocol, a measure of an intermediate outcome.
- Decision point
- The point in the course of care at which a decision rule is applied (in the example, Day 45).
- Review interval
- The interval between decision points when there is more than one.
- Intensity of adaptation
- The degree to which an adaptive intervention has many decision points and/or short review intervals; distinct from intensity of dosage, so a high-adaptation intervention can deliver a light-touch treatment.
- SMART
- Sequential multiple assignment randomized trial: a design for adaptive interventions with relatively low intensity of adaptation, in which participants may be re-randomised at each decision point.
- MRT / system identification experiment
- Designs the lesson points to for adaptive interventions with higher intensity of adaptation (micro-randomised trial; system identification experiment).
- Resource management principle
- The MOST principle that the investigator must strive to make the best and most efficient use of available resources when obtaining scientific information; the design chosen should best manage research resources and most directly address the primary research questions.
- Dose-response question
- A question about the appropriate dose of a component (e.g. number of supportive text messages per week); such questions call for a dose-response experiment and tend to be resource-intensive.
- Hybrid evaluation-optimization trial
- A design that conducts evaluation-phase and optimization-phase research in a single experiment, typically evaluating a previously optimised intervention while testing candidate components to be added; most useful for components being considered as additions, not replacements.
- Continual optimization principle
- The MOST principle that intervention optimisation is an ongoing process of improvement through repeated optimisation trials, adding or replacing components to achieve a stated optimisation objective.
Sources
- Course materials: CADIO, MOST from a Methodological Perspective, Module 5 (“Additional Optimization Trial Designs”), share link https://share.articulate.com/bZuNFklhv0Gqrl2AaW49z, six lessons (Lesson 1, set reading, Lessons 2 to 5), five video lessons, five slide decks.
- Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Set reading: Chapter 8; Figures 2 and 5 are reprinted in the course from this text.)
- Dziak, J. J., Nahum-Shani, I., & Collins, L. M. (2012). Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations. Psychological Methods, 17, 153 to 175. (Cited in Lesson 1; deck slide 17 cites “Dziak et al., 2012”.)
- Nahum-Shani, I., Dziak, J. J., & Collins, L. M. (2018). Multilevel factorial designs with experiment-induced clustering. Psychological Methods, 23, 458 to 479. (Cited in Lesson 1.)
- Nahum-Shani, I., & Dziak, J. J. (2018). Multilevel factorial designs in intervention development. In L. M. Collins and K. C. Kugler (eds.), The multiphase optimization strategy (MOST): Advanced topics. New York: Springer. (Cited in Lesson 1.)
- Almirall, D., Nahum-Shani, I., Wang, L., & Kasari, C. (2018). Experimental designs for research on adaptive interventions: Singly and sequentially randomized trials. In Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer. (Source of the adaptive-intervention definition in Lesson 2 and the SMART discussion in Lesson 3.)
- Klasnja, P., Hekler, E. B., Shiffman, S., Boruvka, A., Almirall, D., Tewari, A., & Murphy, S. A. (2015). Microrandomized trials: An experimental design for developing just-in-time adaptive interventions. Health Psychology, 34(S), 1220. (Cited on the MRT slide.)
- Murray, D. M. (1998). Design and analysis of group-randomized trials. Oxford: Oxford University Press. (Cited in Lesson 1; “a classic of course”.)
- Rivera, D. E., Hekler, E. B., Savage, J. S., & Downs, D. S. (2018). Intensively adaptive interventions using control systems engineering: Two illustrative examples. In Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. New York: Springer. (Cited on the system identification slide.)
- Tanner, A. E., Guastaferro, K. M., Rulison, K., Wyrick, D. L., Milroy, J. M., Bhandari, S., Thorpe, S. Y., Ware, S., Miller, A. M., & Collins, L. M. (2021). A hybrid evaluation-optimization trial to evaluate an intervention targeting the intersection of alcohol and sex in college students and simultaneously test an additional component aimed at preventing sexual violence. Annals of Behavioral Medicine, 55, 1184 to 1187. (The hybrid example in Lesson 5.)
Video list
Prepared 12 Sep 2026. Notes paraphrase the narration unless marked as verbatim quotes; figures are redrawn from the course slides; quotes follow the YouTube caption transcripts. Transcripts were retrieved on 12 Sep 2026 through the browser transcript panel (Show transcript) on each video page, which is the supported route on this host; each transcript carries a video ID and timestamp coverage (L1 0:00 to 15:23, 112 segments; L2 0:01 to 9:33, 72; L3 0:01 to 10:03, 72; L4 0:01 to 15:07, 114; L5 0:01 to 6:31, 51). Caption transcription artefacts are preserved inside quotes and flagged in the anomaly note. Slide images were extracted from the five deck PDFs (136 slides) into 24 montages; all montages were processed with OCR and key montages were additionally inspected visually (Lesson 1 slides 1 to 6 and 13 to 24; Lesson 2 slides 7 to 18; Lesson 4 slides 7 to 12; Lesson 5 slides 7 to 17). Some deck titles in the PDF text layer are template text and do not match the slide images; slide numbers cited here follow the numbers printed on the slides themselves. The course's five knowledge checks are reproduced with the course answer key; the Lesson 3 and Lesson 4 items marking both True and False correct are flagged source anomalies, not endorsed. No separate quiz distinct from these items appears in the course.