← MOST study notes index · CADIO / NYU School of Global Public Health
MOST from a Methodological Perspective: Module 4
Study notes · CADIO / NYU School of Global Public Health
Module 4: “Powering a Factorial Optimization Trial”
Authors: Linda M. Collins and Kate Guastaferro (School of Global Public Health, New York University)
Format: 7 lessons (Lesson 1, set reading, Lessons 2 to 6) · 6 videos · 6 slide decks (PDF) · 6 knowledge checks
Set reading: Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 4, plus Collins, Dziak & Li 2009; Cohen 1988; Fleiss 1986 listed under the set-reading lesson)
Source anomalies found in Module 4 (flagged, not silently corrected). (1) In the Lesson 4 knowledge check the course marks both “True” and “False” as correct for a single multiple-choice statement; both checkboxes are reproduced exactly as in course.json and the item is flagged, not endorsed. (2) The caption transcripts contain transcription artefacts, for example “Corsera” for Coursera, “Guapero” for Guastaferro, “Fiser”/“Fischer”/“Fisher” for Fisher, “pier” for peer, “tstistic” for t statistic, “non-entral” for non-central, “0.05 5” for 0.05, “2 to.3” for 0.2 to 0.3, “.5 range” for 0.5 range, “is8” for is .8, “about4” for about .4, “at least 08” for at least .8, “power of8” for power of .8, “7 to8” for 0.7 to 0.8, “116th” for 1/16th, “coen”/“Coen”/“co-en” for Cohen, “flice”/“FLC”/“FLEC”/“FCE”/“Fleece” for Fleiss, “R equals.5” for R equals .5, “decision-m” for decision-making, “basian” for Bayesian, “y ais”/“x ais” for y axis/x axis; these are preserved inside narration quotes and noted where they occur.
The module's argument in one paragraph.
This module answers the question “how many participants does a factorial optimization trial need, and what does that number mean?” in four steps. First, it clears the ground: an optimization trial is not a pilot study, and the pilot study's logic of feasibility (no hypothesis testing, no power analysis) does not apply to it (Lesson 1). Second, it sets up the standard power machinery, null and alternative hypotheses, Type I and Type II error, power as 1 − beta, and the dependence of power on effect size, sample size and alpha, and recommends expressing effect size as a standardized regression coefficient rather than Cohen's D (Lesson 2). Third, it restates the interpretation of main effects and interactions in the factorial experiment: main effects are averaged across the other factors, interactions are regression coefficients whose power in a balanced 2^k is identical to main effects of the same size, and plotting predicted (marginal) means is the clearest way to read them (Lesson 3). Fourth, it resolves the Cohen-versus-Fleiss “paradox” by showing the two authors define the interaction differently, Fleiss omits the coefficient correction (a denominator of 2 for a two-way interaction), and this choice changes the relation between effect size and the regression coefficient, so power results (for example 351 participants for a raw difference of 3 as a main effect versus a fourfold increase to detect the same raw difference as a Fleiss-defined two-way interaction) must always be tied to an explicit definition (Lesson 4). Lessons 5 and 6 then frame the choices: conclusion-priority versus decision-priority perspectives change the acceptable Type I/Type II balance, and a practical power analysis should be based on the smallest effect size of interest, expressed as a standardized regression coefficient, with covariates used only when their correlation with the outcome is substantial.
Lesson 1: The Pilot Study
The lesson defines a pilot study, contrasts it with an optimization trial, and draws the implications for powering an experiment. The course's definition comes from an article by Leon, Davis and Kramer (2011) in the Journal of Psychiatric Research.
Pilot study defined: “The fundamental purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to be used in a larger scale study.” “Note the emphasis here on feasibility, not drawing scientific conclusions of any kind.”Narration, Lesson 1; deck slides 5 to 6
A pilot study is not a hypothesis-testing study: “a pilot study is not a hypothesis testing study.” “Therefore, no inferential statistical tests should be proposed in a pilot study protocol.” “In other words, if it is a pilot study, you are not going to do hypothesis testing.”Narration, Lesson 1; deck slide 6
No power analysis in a pilot: “power analyses should not be presented.” “A pilot sample size is instead based on the pragmatics of recruitment and the necessities for examining feasibility.” “the reason for conducting a power analysis is to support scientific decisionmaking, that is hypothesis testing.”Narration, Lesson 1; deck slides 7 to 8
Purpose is to try out procedures: “the purpose of pilot studies is to try out study procedures and establish feasibility.”Narration, Lesson 1; deck slide 8
Optimization trials are not pilot studies: “By the Leon at all definition of a pilot study, optimization trials are not pilot studies” (caption artefact: “Leon at all” for Leon, Davis and Kramer). “The one exception is a pilot study to prepare for a fully powered optimization trial.”Narration, Lesson 1; deck slides 9 to 10
Three points of difference: “pilot studies are designed with no regard for statistical power, whereas optimization trials are designed to be adequately powered.” “pilot studies do not need experimental control because no scientific conclusions are going to be drawn based on a pilot study.” “the primary purpose of a pilot study is to prepare for a subsequent trial that will support statistical inference and scientific decision-making.”Narration, Lesson 1; deck slides 11 to 12
Pilot test the optimization trial itself: “Optimization trials are not pilot studies, but it is a good idea to pilot test an optimization trial before you run it.” “The best approach where possible is to pilot the entire optimization trial as designed.” “any pilot testing, including a pilot of the entire optimization trial, is conducted during the preparation phase.”Narration, Lesson 1; deck slides 12 to 14
Implication for powering: “if you are planning a pilot study for the preparation phase of most, there is no need for a power analysis for the pilot study.” “If you are planning an optimization trial, a power analysis is needed.”Narration, Lesson 1; deck slides 14 to 15
Knowledge check
MULTIPLE CHOICE: Is the following statement true or false? An optimization trial is a type of pilot study.
Feedback when incorrect (source text, retained as provided): This is incorrect. Although it is usually necessary to pilot test an optimization trial, optimization trials are not pilot studies because they are fully powered, carefully controlled experiments that address important scientific questions.
Reading for All of Module 4
This lesson has no video or slide deck; it lists the set reading for the module.
- Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Chapter 4 in the course textbook; a free PDF may be available through university libraries.)
- Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202 to 224.
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
- Fleiss, J. L. (1986). The design and analysis of clinical experiments. New York, NY: Wiley.
Lesson 2: Understanding the Basics of Powering a Factorial Experiment
The lesson reviews the logic of hypothesis testing with the t test, defines Type I error, Type II error and statistical power, and shows how power depends on effect size, sample size and alpha. It recommends expressing effect size as a standardized regression coefficient.
Review of the t test logic: “To analyze the data from a factorial optimization trial, you will use factorial analysis of variance ANOVA.” “The null hypothesis states that mu1 - mu2 equals 0.” “The familiar test statistic t equals the quantity yar1 minus y bar 2 divided by the standard deviation of y divided by the per condition sample size” (caption artefacts: “yar1” for Y-bar-1, “mu1 - mu2” for mu1 minus mu2).Narration, Lesson 2; deck slides 3 to 6
The p value and the decision rule: “What is the probability that I would observe a tstistic at least as large as the one I observed in my sample?” (caption artefact: “tstistic” for t statistic). “If the answer to this question is the probability is low, you decide the null hypothesis is false.”Narration, Lesson 2; deck slides 7 to 8
Type I and Type II errors: “A type one error occurs when the investigator rejects the null hypothesis when it is in fact true.” “A type two error occurs when the investigator does not reject the null hypothesis when it is in fact false.”Narration, Lesson 2; deck slide 9
Power is 1 minus beta: “Power is also a conditional probability but it is the probability of making a correct decision when the null hypothesis is false.” “It is the complement of beta.” “In other words, it is 1 minus beta.”Narration, Lesson 2; deck slide 10
What power depends on: “One consideration is sample size.” “All else being equal, a larger sample size will provide more power.” “Another consideration is the size of the effect being detected.” “All else being equal, a larger effect size means more power.” “Another consideration affecting power is the choice of the type one error rate alpha.” “All else being equal, a larger alpha means more power.”Narration, Lesson 2; deck slides 13 to 17
Expressing effect size: “A first step in conducting a power analysis is often expressing the effect size in a standardized measure.” “One often used effect size measure is Cohen's D.” “D is usually computed by taking the difference between the two means being compared and dividing by the standard deviation.”Narration, Lesson 2; deck slide 18
Benchmarks for D and beta star: “A small D is in the 2 to.3 range.” “A medium D is in the.5 range.” “And a large D is8 or greater” (caption artefacts: “2 to.3” for 0.2 to 0.3, “.5 range” for 0.5 range, “is8” for is .8). “When the predictors are uncorrelated, beta star is a function of the correlation between X and the outcome Y.” “Also a small beta star is equal to about 0.1.” “A medium beta star is equal to about 0.25.” “A large beta star is equal to about4 or greater” (caption artefact: “about4” for about .4).Narration, Lesson 2; deck slides 18 to 20
How to power a factorial experiment: “You can choose the desired alpha, the desired level of power, and the expected effect size, and then establish what sample size is needed to achieve the desired level of power.” “Or sometimes the maximum sample size is predetermined, perhaps by the available budget.” “you might want to specify the desired alpha, desired level of power, and upper limit on sample size, and use that information to determine the minimum effect you expect to be able to detect with the available level of resources.”Narration, Lesson 2; deck slides 21 to 22
Figure 1: What affects statistical power, and effect-size benchmarks (redrawn)
| Quantity | Effect on power (all else equal) | Caption benchmark (as narrated) |
|---|
| Sample size | Larger sample, more power | - |
| Effect size (raw difference mu1 − mu2) | Larger effect, more power | - |
| Type I error rate alpha | Larger alpha, more power | - |
| Cohen's D | Small D in the 2 to .3 range | Medium D in the .5 range; large D is .8 or greater |
| Standardized regression coefficient beta star | Small about 0.1 | Medium about 0.25; large about .4 or greater |
Redrawn from Lesson 2 deck slides 13 to 20. The narration's caption benchmarks contain artefacts (“2 to.3” for 0.2 to 0.3, “is8” for is.8, “about4” for about.4) which are preserved in the quotes above; the intended values are given here in the table only as an interpretive aid.
Knowledge check
MULTIPLE RESPONSE: Which of the following can affect statistical power in a given study? Select all that apply.
- ☐ Whether or not the effect size is expressed in terms of a regression coefficient
- ☑ Sample size
- ☑ Choice of Type I error rate
- ☑ Effect size magnitude
Feedback when incorrect (source text, retained as provided): Answer A. is incorrect. How the effect size is expressed does not affect statistical power.
Lesson 3: Interpretation of Main Effects and Interaction Effects
The lesson reviews how to interpret the main effects and interactions estimated in a factorial experiment, using the HIV viral-load example (MI, peer mentoring, text message support at no/yes levels), and distinguishes synergistic and antagonistic interactions. Interpretation relies on plots of predicted (marginal) means.
Definition of a main effect: “Our definition of the main effect of factor A is the effect of factor A averaged across all levels of all other factors.” “the main effect of MI is the mean of conditions 5 through 8 shaded darker minus the mean of conditions 1 through 4 shaded lighter.”Narration, Lesson 3; deck slides 5 to 6
Effect-coded versus dummy-coded: “This effectcoded main effect, which is consistent with the original definition of the main effect provided by Fiser, is not typically the same as the dummy-coded first order effect” (caption artefact: “Fiser” for Fisher). “The dummy-coded first order effect of MI is the effect of MI with the other factors set to no.” “I would like to suggest to you that Fischer's definition is more appropriate for intervention optimization.”Narration, Lesson 3; deck slides 6 to 7
Fisher's rationale, as quoted by the instructor: “Any conclusion has a wider inductive basis when inferred from an experiment in which the quantities of other ingredients have been varied than it would have from any amount of experimentation in which these had been kept strictly constant.”Narration quoting R. A. Fisher, Lesson 3; deck slide 12 (the narrator attributes this quote to “Fiser”/“Fischer”; the slide cites Fisher, 1971, p. 102)
Robustness reading of a main effect: “Components are delivered in the context of other components, usually lots of them.” “If a component demonstrates a strong main effect in the presence of other factors, this suggests robustness to variation in the intervention, such as errors that result in not all components being delivered and future changes in the intervention, that is addition or replacement of components.”Narration, Lesson 3; deck slide 11
Definition of an interaction: “The effectcoded definition of an interaction between factor A and factor B assuming each factor has two levels is as follows.” “It is 1/ half the difference between the effect of factor A at level one of factor B and the effect of factor A at level two of factor B averaged across all the levels of all of the other factors in the experiment” (caption artefact: “1/ half” for one-half).Narration, Lesson 3; deck slide 9
Worry 1: interactions contaminate main effects: “My response to this is that of course interactions should always be considered when interpreting main effects.” “in an effectcoded analysis of variance of data from a balanced factorial experiment, main effects and interactions are uncorrelated.” “When dummy coding is used, the first order and second order effects are highly correlated.”Narration, Lesson 3; deck slides 10 to 11
Worry 1 continued: ignoring interactions does not work: “interactions do not go away just because an investigator chooses not to estimate them.” “Instead, they become inextricably combined with other effects.” “My feeling is that it is better to embrace interactions, study them and come to understand them.”Narration, Lesson 3; deck slide 11
Worry 2: insufficient power to detect interactions: “consider a regression weight beta subj within a balanced 2 to the k factorial experiment.” “Statistical power for detection of beta subj is identical irrespective of whether beta subj corresponds to a main effect or an interaction of any order.” “This does not hold when dummy coding is used.” “It also does not necessarily hold when one or more factors have more than two levels.”Narration, Lesson 3; deck slide 12
Plot interactions with predicted means: “It is always a good idea to plot an interaction to interpret it.” “To do this, you need predicted means based on y hats.” “These will often be marginal means that is means computed by collapsing across one or more factors.” “when the lines on a plot like this are parallel, it indicates there is no interaction.”Narration, Lesson 3; deck slides 13 to 15
Synergistic and antagonistic interactions: “In a synergistic interaction, the combined effect of two or more factors is more favorable than would be expected based solely on the main effects.” “In an antagonistic interaction, the combined effect of two or more factors is less favorable than would be expected based solely on the main effects.”Narration, Lesson 3; deck slide 16
Antagonism does not rule out inclusion: “an antagonistic interaction between two components does not necessarily mean you should not include both components in the intervention.” “Notice that the combination of MI set to yes and peer set to yes still produces the best outcome.”Narration, Lesson 3; deck slides 21 to 22
Three-way interactions: “A three-way interaction means that a two-way interaction differs across a third factor.” “One way to depict three-way interactions graphically is to show a plot for each level of the third factor.”Narration, Lesson 3; deck slide 23
Figure 2: Reading main effects and interactions from plots of predicted means (redrawn)
| Pattern in a two-line plot (MI on one axis, PEER lines) | Interpretation |
|---|
| Parallel lines; vertical gap between them | Main effect of MI, no MI-by-PEER interaction |
| Lines overlap or have the same average height | No main effect of PEER |
| Lines not parallel; solid line tipped up when both factors are yes | Synergistic MI-by-PEER interaction |
| Lines not parallel; solid line tipped down when both factors are yes | Antagonistic MI-by-PEER interaction |
Redrawn from Lesson 3 deck slides 14 to 21. In the plotted examples the outcome is adherence (“adhere” in the narration) on the y axis; marginal means collapse over the third factor (TEXT). An antagonistic interaction can still leave the both-yes condition the best cell, so it does not by itself rule out including both components.
Knowledge check
MULTIPLE CHOICE: Is the following statement true or false? If there is an antagonistic interaction between two factors, this means the corresponding components should not both be included in the intervention being optimized.
Feedback when incorrect (source text, retained as provided): This is incorrect. The presence of an antagonistic interaction between two factors does not rule out including both of the corresponding components.
Lesson 4: Two Different Ways of Defining the Interaction
The lesson resolves the apparent contradiction between Cohen and Fleiss on power for detecting interactions: the two authors use subtly different definitions of the interaction, which differ by a “coefficient correction”.
The apparent paradox: “Cohen has written that in a 2 to the K experiment, power is the same for detection of comparably sized main effects and interactions.” “However, FLC has written that power is considerably lower for detection of interactions as compared with main effects, all else being equal” (caption artefact: “FLC” for Fleiss). “These are two unimpeachable sources making what appears to be contradictory statements.”Narration, Lesson 4; deck slides 3 to 4
Resolution: different definitions: “It turns out that yes, they can both be correct because they are using subtly different definitions of the interaction.” “Both Fleece and Cohen are using effect coding in their definitions” (caption artefact: “Fleece” for Fleiss). “The difference lies in whether or not a quantity we will call the coefficient correction is included.”Narration, Lesson 4; deck slide 4
Dot notation for marginal means: “a dot in a subscript indicates a mean over a particular factor.” “In other words, a marginal mean.” “mu bar sub plus dot represents the marginal mean when mi equals yes and pier equals yes averaged over the two levels of text.”Narration, Lesson 4; deck slide 6
The two definitions: “According to the Cohen definition, the MI by interaction is defined as follows.” “In the Cohen definition, this quantity is divided by two.” “The FLC definition is exactly the same except there is no denominator.” “They are identical except the Cohen definition includes a denominator of two.”Narration, Lesson 4; deck slides 7 to 8
The coefficient correction: “We are going to refer to that two in the denominator of the Coen definition as the coefficient correction” (caption artefact: “Coen” for Cohen). “It's a two for a two-way interaction, but as we will see in a moment, it varies depending on the order of the interaction.” “It's a two in the denominator for the two-way interaction and a four in the denominator for the three-way interaction.”Narration, Lesson 4; deck slides 8 to 9
Main effect in dot notation: “the main effect of MI is equal to mu bar sub plus dot dot minus muar sub minus dot dot” (caption artefact: “muar” for mu-bar). “the main effect is a linear combination of all of the experimental condition means each multiplied by a coefficient of one quarter.”Narration, Lesson 4; deck slide 10
Same scale under the Cohen definition: “in the coen definition of the interaction the interaction is on the same scale as the main effect” (caption artefact: “coen” for Cohen). “In other words the coefficient correction ensures that all main effects and interactions are on the same scale.”Narration, Lesson 4; deck slide 11
Interaction star and the regression coefficient: “We will refer to such interactions as the interaction star.” “in an effectcoded analysis, the regression coefficient corresponding to a main effect is always equal to 1/2 the main effect.” “the regression coefficient corresponding to an interaction is equal to 1/2 the interaction effect if the coen definition of the interaction the one that includes the coefficient correction is used.” “This holds for interactions of any order.”Narration, Lesson 4; deck slides 12 to 13
Fleiss definition changes the relation: “If the flice definition of the interaction is used, the relation to the regression coefficient is more complicated” (caption artefact: “flice” for Fleiss). “the regression coefficient corresponding to the two-way MIPER interaction.” “This is equal to the FLC definition divided by four.”Narration, Lesson 4; deck slide 13
The table of relations: “When the Coen definition is used, the regression coefficient is always equal to 1/2 the interaction.” “When the FLC definition is used, the regression coefficient equals 1/4 of a two-way interaction, 1/8 of a three-way interaction, and 116th of a four-way interaction” (caption artefact: “116th” for 1/16th).Narration, Lesson 4; deck slide 14
Dr. B's example: main effects: “Suppose Dr. B is planning a 2 to the K factorial experiment and wants to detect main effects corresponding to a raw difference between means of three.” “Remember this is equivalent to an unstandardized B weight of 1.5.” “You conduct a power analysis and you find that to achieve power of at least 0.8 you need 351 participants.”Narration, Lesson 4; deck slides 15 to 16
Dr. B's example: interactions: “Suppose Dr. B is thinking in terms of the Cohen definition.” “So the answer to Dr. B's question is yes there is power of8 to detect interactions the same size as the main effects” (caption artefact: “power of8” for power of .8). “Now suppose that instead Dr. B is thinking in terms of the flice definition.” “the raw difference of three corresponds to a B weight of 75” (caption artefact: “75” for .75). “To maintain power of8 it would be necessary to increase the sample size by a factor of four” (caption artefact: “power of8” for power of .8).Narration, Lesson 4; deck slides 16 to 17
Practical guidance: “Again I want to emphasize there is no right or wrong here.” “You can use either the Cohen or FLEC definition of the interaction” (caption artefact: “FLEC” for Fleiss). “The important thing is to be clear and consistent about which one you are using.” “Those of us working in intervention optimization find it easier to think in terms of the regression coefficient for all effects.”Narration, Lesson 4; deck slide 18
Figure 3: Relation between the effect-coded regression coefficient and the two interaction definitions (redrawn)
| Definition of the interaction | Regression coefficient (effect-coded, balanced 2^k) |
|---|
| Cohen (with coefficient correction) | 1/2 the interaction, for interactions of any order |
| Fleiss (without coefficient correction) | 1/4 of a two-way interaction; 1/8 of a three-way interaction; 1/16 of a four-way interaction |
Redrawn from Lesson 4 deck slide 14. Under the Fleiss definition the relation between the effect size and the regression coefficient depends on the order of the interaction, which is why a raw difference of 3 (B weight 1.5 under the Cohen definition) corresponds to a B weight of .75 under the Fleiss definition for a two-way interaction, and comparable power then requires four times the sample size.
Knowledge check
MULTIPLE CHOICE: Is the following statement true or false? There are two different definitions of the interaction in wide use, and which one is used makes a difference in a statistical power analysis.
Feedback when incorrect (source text, retained as provided): This is incorrect. Be sure you understand the difference between the two definitions.
Lesson 5: Two Distinct Perspectives on Research: Conclusion-Priority and Decision-Priority
The lesson contrasts two perspectives on the purpose of a study: the conclusion-priority perspective (hypothesis testing, scientific conclusions) and the decision-priority perspective (deciding which components and levels to include in the optimized intervention), and shows how they lead to different Type I/Type II choices.
Review of the error rates: “A type one error is rejection of the null hypothesis given that it is true or in other words mistakenly concluding an effect exists when it does not.” “A type two error is a failure to reject the null hypothesis given that it is false or in other words mistakenly overlooking an effect.”Narration, Lesson 5; deck slides 3 to 4
The conclusion-priority perspective: “if you obtain significant results at P less than 0.05, you conclude that the effect exists and you can reject the null hypothesis.” “We will call this entire perspective the conclusion priority perspective.”Narration, Lesson 5; deck slides 5 to 6
The decision-priority perspective: “Here the purpose of conducting a study is to enable you to make a decision about which components and component levels are to make up the optimized intervention.” “Every component examined in the optimization trial will either be included in the optimized intervention or not.” “We will call this the decision priority perspective.”Narration, Lesson 5; deck slides 6 to 8
The resource management principle: “when powering an optimization trial, the resource management principle is a critical consideration.” “you are going to make the best use of these resources to get the largest amount of and highest quality information you can to enable you to make that decision.”Narration, Lesson 5; deck slides 7 to 8
Conventions in the conclusion-priority perspective: “the maximum acceptable alpha is 0.05 and the minimum acceptable power is in the 7 to8 range” (caption artefact: “7 to8” for 0.7 to 0.8). “So a beta in the 2 to.3 range is acceptable” (caption artefact: “2 to.3” for 0.2 to 0.3). “the type 2 error rate is routinely four to six times the type one error rate.”Narration, Lesson 5; deck slide 9
Error trade-off in the decision-priority perspective: “Overlooking an effective component may be at least as painful or costly as mistakenly concluding that a component is effective.” “The resource management principle suggests consider raising the type one error rate to enable more power with the sample size you can afford.” “You are strategically accepting a higher type one error rate to reduce the type 2 error rate.”Narration, Lesson 5; deck slides 10 to 12
The worked example: “You want to maintain power of at least 08” (caption artefact: “at least 08” for at least .8). “You conduct a power analysis using your anticipated effect size and an alpha of 0.05 and find you will need a sample size of 126.” “Suppose you are comfortable with an alpha of 0.10.” “You find that an alpha of 100 reduces the required sample size to 98” (caption artefact: “an alpha of 100” for an alpha of .10). “So with an alpha of 010, the optimization trial can be accomplished within your resource limitations” (caption artefact: “010” for .10).Narration, Lesson 5; deck slides 12 to 13
Which perspective for which phase: “both perspectives are appropriate, but at different times and for different purposes.” “In the optimization phase of most, typically an investigator works from the decision priority perspective.” “In the evaluation phase, when the statistical and clinical significance of the optimized intervention is being tested, typically the investigator works from the conclusion priority perspective.”Narration, Lesson 5; deck slides 14 to 15
Summarising the differences: “From the conclusion priority perspective, it is okay to conclude not to conclude.” “By contrast, from the decision priority perspective, it is always necessary to make a decision.” “From the conclusion priority perspective, hypothesis testing is a primary objective.” “From the decision priority perspective, hypothesis testing is one tool for decision-making and it is just that, a tool that exists to help the investigator in decision-m” (caption artefact: “decision-m” for decision-making).Narration, Lesson 5; deck slide 16
A Bayesian note: “You may be thinking that in many ways a basian framework would be a more natural fit for most and I don't disagree” (caption artefact: “basian” for Bayesian). “the reality is it will be some time before a purely basian approach will be widely acceptable particularly in grant proposals.”Narration, Lesson 5; deck slide 17
Figure 4: Conclusion-priority versus decision-priority (redrawn)
| Dimension | Conclusion-priority | Decision-priority |
|---|
| Purpose of the study | Draw scientific conclusions about effects | Decide which components/levels to include |
| Is “we do not know” acceptable? | Yes: okay to conclude not to conclude | No: a decision is always necessary |
| Role of hypothesis testing | Primary objective | One tool for decision-making |
| Type I error convention | Alpha no more than 0.05 | Selected by the investigator, partly via the resource management principle |
| Type II error | Routinely 4 to 6 times the Type I rate | Balance determined to best meet the study's needs |
| Phase of MOST | Evaluation phase | Optimization phase |
Redrawn from Lesson 5 deck slides 9 to 16. In the decision-priority example, raising alpha from 0.05 to 0.10 reduces the required sample size from 126 to 98 participants at the cost of a higher Type I error rate.
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? There may be more than one.
- ☑ When an investigator is working from the conclusion-priority perspective, it is usually important to select a Type I error rate no larger than .05.
- ☑ An investigator working from the decision-priority perspective must consider how painful or expensive Type II errors are relative to Type I errors, and may wish to adjust these error rates accordingly.
- ☐ In the conclusion-priority perspective, investigators usually try to keep the Type I and Type II error rates about the same.
- ☑ The decision-priority perspective is usually appropriate for the optimization phase of MOST.
Feedback when incorrect (source text, retained as provided): Answer C. is incorrect. In the conclusion-priority perspective the Type II error rate is routinely 4 to 6 times the Type II error rate.
Lesson 6: Putting It All Together to Conduct a Power Analysis
The lesson integrates the module into a practical checklist for powering a factorial optimization trial.
Considerations to work through with your team: “One consideration is which effects are of primary scientific interest.” “Another consideration is how will you express the effect sizes?” “If you are powering to detect interactions, are you using the Cohen or FLCE definition?” (caption artefact: “FLCE” for Fleiss). “Are you planning to include a coariant such as a pretest in the statistical analysis?” (caption artefact: “coariant” for covariate). “are you considering the power analysis from a conclusion priority or decision priority perspective?”Narration, Lesson 6; deck slides 3 to 4
Which effects matter: “Factorial experiments are typically powered to detect main effects.” “You can power to detect interactions, but typically power analyses do not include higher order interactions.” “Identification of effects of primary scientific interest should be based on the conceptual model which we discussed briefly earlier in the course.”Narration, Lesson 6; deck slide 5
Express effect sizes as standardized regression coefficients: “We recommend using the standardized regression coefficient as an effect size measure.” “This is a straightforward approach that helps to avoid confusion about the definition of the interaction.”Narration, Lesson 6; deck slide 6
Actual size versus minimum size you care about: “Effect sizes for a power analysis of an optimization trial can be based on either the actual size you believe the effect will be or the minimum effect size that you care about.” “Maybe a component has to show a main effect standardized regression coefficient of greater than or equal to say 0.1 to be considered for inclusion.” “If the effect is less than this, you do not care what it is.” “So there is no need to be powered to detect it.”Narration, Lesson 6; deck slides 7 to 8
Where interactions come into play: “The conceptual model may specify certain important interactions as part of the treatment mechanism.” “In this case, you may want to maintain a desired level of power to detect these interactions.” “Otherwise, you can think of interactions as the rest of the story.”Narration, Lesson 6; deck slide 9
Power is the same for any regression weight in a balanced 2^k: “for data from a balanced 2 to the k experiment analyzed using effect coding, there is the same amount of statistical power associated with any regression weight of a given size irrespective of whether the coefficient represents a main effect or an interaction.” “This means all else being equal in a 2 to the k experiment power is the same for regression coefficients corresponding to main effects and interactions.”Narration, Lesson 6; deck slide 10
Covariates: “Adding a covariant such as a pretest will often reduce sample size requirements” (caption artefact: “covariant” for covariate). “However, do this only if the correlation is substantial above about R equals.5” (caption artefact: “R equals.5” for R equals .5). “Otherwise, including a coariant may cause a loss of power” (caption artefact: “coariant” for covariate).Narration, Lesson 6; deck slide 11
Error-rate choices: “If you are working from the conclusion priority perspective, you probably need to stick with alpha less than or equal to 0.05.” “If you are working from the decision priority perspective, you may want a different balance of type one and type two error rates.” “It may make sense to use an alpha in the 0.1 to 0.15 range, but I realize this is a controversial recommendation.” “remember you will be following up your optimization trial with an ERCT from a conclusion priority perspective using an alpha of 0.05 or less.”Narration, Lesson 6; deck slides 12 to 13
Base the analysis on the smallest effect size: “You can handle this by identifying the smallest effect size and basing the power analysis on this effect size.” “This will provide enough power for detection of any larger effects.”Narration, Lesson 6; deck slide 14
Software and multiple-testing caveat: “You can use any power analysis software to conduct a power analysis for a factorial optimization trial.” “Just be sure it is clear whether the software is assuming effect or dummy coding.” “Everything we have said assumes you are not using a correction for multiple hypothesis tests.” “We will come back to this later in this course.”Narration, Lesson 6; deck slides 14 to 15
Knowledge check
MULTIPLE RESPONSE: Which of the following statements are correct? There may be more than one.
- ☐ It is a good strategy to base the power analysis for an optimization trial on the largest effect size of interest.
- ☑ If an optimization trial is powered to detect a regression coefficient of a particular magnitude, power is equivalent for detection of all other effects of that magnitude or greater.
- ☑ Consider a balanced factorial experiment in which all factors have exactly two levels. Suppose the data from this experiment are to be analyzed using an effect-coded factorial analysis of variance. Then power is identical for detection of a regression coefficient of a given size, whether the coefficient corresponds to a main effect or an interaction.
Feedback when incorrect (source text, retained as provided): Answer A. is incorrect. An optimization trial should be powered to detect the smallest effect size of interest.
Synthesis (my own reading: not part of the course)
Read methodologically, this module converts the power-analysis question into a question about definitions, and then into a question about decision rules. The pilot-study lesson sets the boundary: power analysis belongs to experiments that support scientific decision-making, which is exactly what an optimization trial is designed to do, so the pilot-study prohibition on power analysis does not transfer. The Lesson 4 material is the technical heart. The “paradox” between Cohen and Fleiss is resolved by noticing that the two definitions differ by a constant (the coefficient correction), and the constant changes the mapping between an effect size expressed as a raw difference and the regression coefficient the power analysis actually uses: under the Cohen definition every coefficient is half its interaction, under the Fleiss definition the coefficient is one quarter (two-way), one eighth (three-way) or one sixteenth (four-way) of the interaction, so the same raw difference demands a fourfold larger sample when powering a Fleiss-defined two-way interaction. The lesson's practical advice, think in terms of regression coefficients, effectively chooses the Cohen definition while keeping the analysis definitionally explicit.
Three further points are easy to misread. First, in a balanced 2^k with effect coding, power is identical for any regression coefficient of a given size regardless of whether it is a main effect or an interaction (Lesson 3 and Lesson 6); this is what makes factorial optimization trials efficient, and it is why the Lesson 3 worry that interactions lack power is, for this design family, misplaced. Second, the interaction definition matters only when effect sizes are expressed as raw differences; expressing them as standardized regression coefficients sidesteps the ambiguity. Third, the choice between conclusion-priority and decision-priority perspectives is a real choice about error costs: the decision-priority perspective deliberately allows alpha above 0.05 when resources are constrained, and the module's example (126 to 98 participants by raising alpha from 0.05 to 0.10) makes the arithmetic of that trade explicit.
One caution for readers of the course materials: the Lesson 4 knowledge check marks both “True” and “False” as correct for a single multiple-choice statement, which cannot be a coherent answer key; the notes reproduce the marks exactly and flag the anomaly rather than choosing an intended answer. Caption transcription artefacts (Corsera, Guapero, Fiser/Fischer, pier, tstistic, non-entral, 0.05 5, 2 to.3, is8, about4, at least 08, power of8, 7 to8, 116th, coen/Coen/co-en for Cohen, flice/FLC/FLEC/FCE/Fleece for Fleiss, R equals.5, decision-m, basian, y ais/x ais, muar, 75 for .75) are preserved inside quotes and flagged. No answer-level “This is incorrect” feedback anomalies occur in this module's knowledge checks.
Glossary
- Pilot study
- A study whose fundamental purpose is to examine the feasibility of an approach intended for use in a larger study; it is not a hypothesis-testing study, no inferential tests are proposed, and no power analysis is presented (after Leon, Davis & Kramer, 2011).
- Optimization trial
- A fully powered, carefully controlled experiment whose data are used in statistical inference and scientific decision-making; not a pilot study, though pilot testing an optimization trial before running it is recommended.
- Preparation phase / optimization phase
- Phases of MOST; pilot testing is conducted in the preparation phase, and the optimization trial itself in the optimization phase.
- Type I error / alpha
- Rejecting the null hypothesis when it is true; alpha is the Type I error rate.
- Type II error / beta
- Failing to reject the null hypothesis when it is false; beta is the Type II error rate.
- Statistical power
- The probability of making a correct decision when the null hypothesis is false; power = 1 − beta.
- Central / non-central distribution
- The distribution of the test statistic when the null hypothesis is true (central) versus under a particular alternative hypothesis (non-central); power estimates compare the two.
- Cohen's D
- A standardized effect size: the difference between two means divided by the standard deviation; small D in the 0.2 to 0.3 range, medium about 0.5, large 0.8 or greater.
- Standardized regression coefficient (beta star)
- The recommended effect-size measure for powering optimization trials; small about 0.1, medium about 0.25, large about 0.4 or greater.
- Main effect
- The effect of a factor averaged across all levels of all other factors; in effect coding, half of it equals the corresponding regression coefficient.
- Interaction
- In this course, defined as the corresponding regression coefficient in an effect-coded balanced factorial analysis.
- Synergistic / antagonistic interaction
- A combined effect more (synergistic) or less (antagonistic) favorable than expected from the main effects alone.
- Coefficient correction
- The denominator included in the Cohen definition of the interaction (2 for a two-way interaction, 4 for a three-way interaction) and omitted in the Fleiss definition; it keeps main effects and interactions on the same scale.
- Interaction star
- The interaction defined without the coefficient correction (the Fleiss definition).
- Conclusion-priority perspective
- The hypothesis-testing perspective in which alpha is capped at 0.05, Type II error is routinely 4 to 6 times the Type I error, and “we do not know” is an acceptable conclusion; appropriate for the evaluation phase.
- Decision-priority perspective
- The perspective in which the study's purpose is deciding which components and levels to include; error rates are balanced via the resource management principle; appropriate for the optimization phase.
- Resource management principle
- The principle that resources should be used to obtain the largest amount of the highest quality information needed for the decision at hand, including strategically trading Type I and Type II error rates.
- Smallest effect size of interest
- The minimum effect the team cares to detect; powering for it provides enough power for detection of any larger effects.
Sources
- Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). New York: Springer. (Set reading: Chapter 4 for all of Module 4.)
- Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202 to 224. (Cited in the set-reading lesson.)
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. (Cited in the set-reading lesson and Lesson 4 on the two interaction definitions.)
- Fleiss, J. L. (1986). The design and analysis of clinical experiments. New York, NY: Wiley. (Cited in the set-reading lesson and Lesson 4 on the two interaction definitions.)
- Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626 to 629. (Cited by name in Lesson 1 as “Leon Davis and Kramer”, 2011.)
- Course materials: CADIO, MOST from a Methodological Perspective, Module 4 (“Powering a Factorial Optimization Trial”), share link https://share.articulate.com/8W66mqBJzaAlA5O311wZ7, six video lessons and six slide decks.
Video list
Prepared 11 Sep 2026. Notes paraphrase the narration unless marked as verbatim quotes; figures are redrawn from the course slides; quotes follow the YouTube caption transcripts. This module's captions were retrieved through the youtubetotranscript.com service because the host's direct YouTube API access is currently bot-walled (the browser transcript panel never receives segment content and the innertube player endpoint returns LOGIN_REQUIRED); the transcripts were content-verified against each lesson's topic and the course's own lesson titles, and duration timestamps are not available from this route. Caption transcription artefacts are preserved inside quotes and flagged. The course's six knowledge checks are reproduced with the course answer key; the Lesson 4 item marking both True and False correct is a flagged source anomaly. No separate quiz distinct from these items appears in the course.