Adaptive tutoring is the best-evidenced adaptation, and the effect is smaller than the promise

Intelligent tutoring is the only adaptive system in this module backed by pooled effect sizes and moderator analyses. The argument has three parts: the benchmark describes tutoring rather than adaptivity, the pools sit far below it and fall further under stricter designs, and the evidence best supports a machine that supplies corrected practice rather than one that teaches.

If adaptivity is the active ingredient, the learner model and the sequencing rules are where the effort belongs; if corrected practice is, the model is a scheduler and the effort belongs in the opportunities to answer, err and receive correction. The record does not settle which, and the comparisons differ, so the number moves. The tenth of ten units closes the sequence by asking what the best-evidenced adaptation delivers, and why the effect is smaller than the promise it inherited.

The two-sigma benchmark measures tutoring, not adaptivity

Bloom (1984) reported that students tutored one to one, or in pairs, performed about two standard deviations above students taught conventionally in a class of about thirty, so the average tutored student stood above about 98 per cent of the control class, and mastery learning alone produced about one standard deviation.

Corbett and Anderson (1995) built the machine version. Their ACT Programming Tutor held a production-rule model of programming knowledge and estimated the probability that the learner had learned each rule, the process they named knowledge tracing, then chose the next exercise from those estimates until the model judged each rule mastered.

The pooled estimates sit far below the inherited benchmark

VanLehn (2011) reviewed the experiments comparing human tutoring, computer tutoring and no tutoring and found the inherited beliefs unsupported: human tutoring came out at about 0.79 standard deviations and step-based computer tutoring at about 0.76.

Kulik and Fletcher (2016) meta-analysed 50 controlled evaluations and reported a median gain of 0.66 standard deviations over conventional instruction. The gain depended heavily on whether the test was written locally or standardised, so a reported effect is partly a statement about who wrote it.

Ma and colleagues (2014) pooled 107 effect sizes from 14,321 participants and reported the effect by comparison: about 0.42 against teacher-led large-group instruction, 0.57 against non-intelligent computer-based instruction, and 0.35 against textbooks, but about minus 0.11 against individual human tutoring, not statistically significant, so the machine beat the lecture and the workbook and did not beat the human tutor, locating the benefit in the tutoring rather than in the intelligence.

The comparison, the measure and the period set the number

Two narrower analyses push lower. Steenbergen-Hu and Cooper (2013) pooled 34 independent samples from 26 reports on school mathematics published between 1997 and 2010 and found at most a small positive effect, with sizes from about 0.01 to about 0.09. Their college analysis (Steenbergen-Hu and Cooper, 2014) found about 0.32 to about 0.37 across 39 studies of 22 systems, less than human tutoring and weaker in later studies, so the range across the pool is wider than the gap between any system and its comparison.

Ma and colleagues (2014) price the comparison condition: one body of studies yields a large effect against no instruction and a nil effect against a human tutor. Kulik and Fletcher (2016) price the outcome measure, and Steenbergen-Hu and Cooper (2013) show that population and period matter, since their effects came from school mathematics in a fixed window.

The evidence licenses corrected practice, not a teaching model

Nye, Graesser and Hu (2014) reviewed seventeen years of AutoTutor, a natural-language tutor built on human-inspired strategies, and reported average gains of about 0.3 standard deviations against reading text for an equivalent time, with deep reasoning questions, feedback and adaptation to learner state doing the work. That evidence is real and lives inside a lineage that took expert labour to build.

AutoTutor's mechanism is concrete enough to follow. Each question carries two prepared lists, one of the good answers or steps it expects and the other of the misconceptions it anticipates, and the student answers in short fragments across several turns. The tutor matches each fragment against those lists, starting with a pump such as 'what else', then a hint towards a missing idea, a prompt for a single word, and finally the answer the student has still not produced. A fragment that matches a misconception draws its correction, so the adaptation is pattern matching rather than a full reading of the student's reasoning. The matching runs on English fragments, so the learner it describes is at home in English.

Reich and Ruiperez-Valiente (2019) show what scaling the vehicle delivers: most learners never return after the first year, participation concentrated in the wealthiest countries, and low completion rates unimproved over six years. Siemens and Baker (2012) describe the infrastructure underneath, the educational data mining and learning analytics communities with their different leanings. Zawacki-Richter and colleagues (2019) screened 2,656 publications and synthesised 146, concluding that critical reflection on challenges and risks is almost absent and the link to theory weak.

The contested question: is the model the active ingredient?

First, adaptivity is the active ingredient. Its strongest support is the expertise reversal effect reviewed by Kalyuga (2007): support that helps a novice can hinder a learner with more prior knowledge, so the relative effectiveness of a method reverses as expertise grows, and a fixed default must fail for part of every audience. Koedinger, Corbett and Perfetti (2012) supply the framework in which a team can test this.

Second, corrected practice is the active ingredient, supplied with or without a model. Koedinger and colleagues (2023) fitted growth models to 27 datasets covering about 1.3 million observations from 6,946 learners, and found that a typical learner needed about 7.2 practice opportunities per knowledge component while differing little in rate. If rate barely varies, what a system must adapt is the number and timing of opportunities, not the pace, so a well-designed fixed sequence may capture most of the benefit. Ma and colleagues (2014) support the reading, since the advantage over a human tutor was nil.

Third, some authors doubt the label. Steenbergen-Hu and Cooper (2013) found effects from about 0.01 to 0.09 where tutoring is most deployed, and later studies weaker than earlier ones.

What would settle it is a design that has not been run at scale: hold the material and the number of practice opportunities fixed, vary only whether item order is chosen by a tracked model or fixed in advance, and measure a standardised outcome after a delay. If adaptive sequencing wins with practice held constant, the model works; if the conditions are indistinguishable, the model only schedules.

The adaptivity position rests on a review of the expertise reversal effect (Kalyuga, 2007) and a predictive framework rather than a trial (Koedinger, Corbett and Perfetti, 2012); the corrected-practice position on growth models fitted to twenty-seven datasets (Koedinger et al., 2023) and the pooled comparison against a human tutor (Ma et al., 2014); the doubt position on two meta-analyses that found effects from about 0.01 to 0.09 in school mathematics (Steenbergen-Hu and Cooper, 2013, 2014). While the dispute is open, hold the practice opportunities fixed and vary only the sequencing before claiming the model works. The strongest objection is that the best system-level evidence comes from one lineage (Nye, Graesser and Hu, 2014).

Where this is heading

Nye and colleagues (2014) close their review asking whether language models now read a student's reasoning as AutoTutor did or merely produce fluent explanations. The first controlled test of the question is not encouraging: Bastani and colleagues (2025) gave nearly a thousand high-school students a GPT-4 tutor for about fifteen per cent of the curriculum and then measured an unaided exam. Practice performance rose sharply, but the unguarded arm scored 17 per cent worse on that exam, and the safeguards removed the harm without adding a gain, so fluency did not convert into learning. Koedinger and colleagues (2023) move the target: schedule enough opportunities and protect the mastery threshold. On this evidence the machine tutor's contribution is to make corrected practice affordable, not to reproduce the tutor.

Sources

• Bloom, B. S. (1984). The 2 sigma problem: the search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher. doi:10.2307/1175554 Reported that tutored students performed about two standard deviations above conventionally taught students, with mastery learning alone at about one, and posed the search for group methods that could approach tutoring. The promise against which adaptive tutoring is measured, and a claim about tutoring rather than adaptation.
• VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist. doi:10.1080/00461520.2011.611369 Reviewed the experiments comparing human tutoring, computer tutoring and no tutoring and found the inherited beliefs wrong, with human tutoring at about 0.79 standard deviations and step-based computer tutoring at about 0.76. Shrinks the benchmark and shows the machine approaching the human tutor.
• Kulik, J. A. and Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Review of Educational Research. doi:10.3102/0034654315581420 Meta-analysed 50 controlled evaluations and reported a median gain of 0.66 standard deviations over conventional instruction, with the size depending heavily on whether the test was locally developed or standardised. Supplies the largest pooled estimate and the reason to distrust test alignment.
• Steenbergen-Hu, S. and Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K-12 students' mathematical learning. Journal of Educational Psychology. doi:10.1037/a0032447 Analysed 26 reports containing 34 independent samples from 1997 to 2010 and found at most a small positive effect, with sizes from about 0.01 to about 0.09. The sharpest counterweight to the large pooled estimates.
• Ma, W., Adesope, O. O., Nesbit, J. C. and Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. Journal of Educational Psychology. doi:10.1037/a0037123 Pooled 107 effect sizes from 14,321 participants and reported the effect by comparison: about 0.42 against teacher-led large-group instruction, 0.57 against non-intelligent computer-based instruction, 0.35 against textbooks, and no significant difference against individual human tutoring. Shows the comparison condition, not the adaptation, sets the number.
• Corbett, A. T. and Anderson, J. R. (1995). Knowledge tracing: modelling the acquisition of procedural knowledge. User Modelling and User-Adapted Interaction. doi:10.1007/BF01099821 Described a tutor built around a production-rule model that maintained a probability of mastery for each rule and chose the next exercise from those estimates until each was judged mastered. The origin of the learner model in tutoring, and of adaptation's technical meaning.
• Koedinger, K. R., Corbett, A. T. and Perfetti, C. (2012). The knowledge-learning-instruction framework: bridging the science-practice chasm to enhance robust student learning. Cognitive Science. doi:10.1111/j.1551-6709.2012.01245.x Proposed a framework linking knowledge types, learning processes and instructional principles so that an intervention can be predicted to work or not. The causal account the pooled estimates lack, and an attempt to explain why some adaptations help.
• Nye, B. D., Graesser, A. C. and Hu, X. (2014). AutoTutor and family: a review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education. doi:10.1007/s40593-014-0029-5 Reviewed seventeen years of a natural-language tutor and reported average gains of about 0.3 standard deviations against reading text for an equivalent time, with deep reasoning questions, feedback and adaptation to learner state doing the work. The strongest system-level evidence that conversational adaptation can teach.
• Siemens, G. and Baker, R. S. J. D. (2012). Learning analytics and educational data mining: towards communication and collaboration. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge. doi:10.1145/2330601.2330661 Set out the relationship between the two communities that analyse learner-generated data, their shared interest in using it and their differing leanings on discovery. The data infrastructure adaptive learning depends on, and the interpretive problem inside it.
• Kalyuga, S. (2007). Expertise reversal effect and its implications for learner-tailored instruction. Educational Psychology Review. doi:10.1007/s10648-007-9054-3 Reviewed findings that instructional support helping a novice can hinder a learner with more prior knowledge, so the relative effectiveness of a method reverses as expertise develops. The strongest theoretical reason a fixed default must fail.
• Reich, J. and Ruiperez-Valiente, J. A. (2019). The MOOC pivot. Science. doi:10.1126/science.aav7958 Analysed Harvard and MIT course data on edX and reported three patterns: most learners never return after the first year, participation growth concentrated in the wealthiest countries, and low completion rates unimproved over six years. The deployment evidence that scaling the vehicle does not deliver learning.
• Steenbergen-Hu, S. and Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students' academic learning. Journal of Educational Psychology. doi:10.1037/a0034752 Analysed 35 reports containing 39 studies of 22 tutoring systems and found a moderate effect of about 0.32 to about 0.37, less effective than human tutoring, with earlier studies significantly greater than later ones. The shrinkage over time inside a single review.
• Koedinger, K. R., Carvalho, P. F., Liu, R. and McLaughlin, E. A. (2023). An astonishing regularity in student learning rate. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.2221311120 Fitted growth models to 27 datasets covering about 1.3 million observations from 6,946 learners and found that a typical learner needed about 7.2 practice opportunities per knowledge component while differing little in rate of learning. Changes what an adaptive system should adapt to, from pace to opportunities.
• Zawacki-Richter, O., Marin, V. I., Bond, M. and Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education: where are the educators? International Journal of Educational Technology in Higher Education. doi:10.1186/s41239-019-0171-0 Screened 2,656 publications from 2007 to 2018 and synthesised 146, sorting them into four application areas, and concluded that critical reflection on challenges and risks is almost absent and the connection to pedagogical theory weak. Shows the research attention sitting away from the evidence that matters here.
• Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakci, O. and Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.2422633122 Gave nearly a thousand high-school students an unguarded GPT-4 tutor, a safeguarded one prompted to give hints rather than answers, or no access. Practice performance rose sharply, but the unguarded arm scored seventeen per cent worse on a later unaided exam, a harm removed but not reversed by the safeguards. One subject, one grade band, a short exposure.