In Part 3, we looked at research suggesting that attributing Direct Instruction’s success to its imitators was likely a bad idea. Today we’re going to examine what research says about those imitators in general, but first we need to look at two big issues with that research. The first issue is that the variety of available programs with their differences in alignment, format, pacing, and quality of support means that available studies typically look at a single, specific scripted program. This means we can’t generalize results across all scripted curricula. The other problem – and it’s a big one – is that research on scripted programs is often either produced by the universities that created them or paid for by the commercial entity that is marketing the program. Both of those situations affect the validity of the findings.* The research on DI’s imitators is sometimes questionable, occasionally contradictory, and frequently unimpressive in terms of positive effects on student learning, particularly for certain students. All the programs mentioned below are scripted programs; the research is categorized by issue. Consider:
Conflict of Interest
- Several studies found statistically significant positive influences on reading achievement with Success for All (SFA), but the chairman of the Success for All Foundation was a coauthor on all of those studies1.
- This is a tad softer, but some educational publications arguing for (and against) scripted curricula cite a person who is being paid to help states select and implement such curricula and/or representatives of companies that supply such programs. In one article, the author questions a person asserting the high quality of scripted programs, noting the assertion is unsupported by specific research. Another article quoted the same person,** who appeared to argue that scripted programs are necessary because teachers just aren’t smart enough to develop lessons that are aligned to standards. There is, however, an argument to be made that commercial entities (and their champions) don’t seem capable of understanding the difference between a standard and an objective so I’m not sure this guy should be pointing fingers.
Contradictory Findings
- Literature compiled by the creators of Open Court in 1998 cited studies from California and Arizona that ELLs using Success For All (SFA) scored higher on English reading measures than students using a different program. However, just 4 years later in 2002 a 3-year study from Miami-Dade County Schools in Florida found that ELLs in SFA schools actually made smaller gains in English proficiency than ELLs in schools not using SFA.
- One study found that Direct Instruction (the original scripted program) produced some positive results in students’ reading achievement in grades 1-52. However, a different study3 found that Direct Instruction benefited suburban students but not urban ones, and that individual teachers’ characteristics significantly influenced class reading achievement. There’s some irony here because DI was originally developed as an intervention for economically disadvantaged students, who are more prevalent in urban districts. It’s reasonable to assume that something has changed in DI since its inception in the 1960s. It’s also important because it means that the particular population of students using DI has a direct effect on its efficacy. The other bit of irony is that a program meant to control for teacher differences was shown to be effective for some teachers but not others.
- Two studies of Open Court Reading conducted a decade apart first found statistically significant positive effects4, then mixed positive and negative effects5. There may be some confounders here because a lot of change can occur in ten years – teacher preparedness, the body of state standards, the characteristics of the students in the districts studied, the content and structure of the state test, and of course, the program itself.
Mixed or Negative Results
- A 2002 study of Open Court reading in California using the Stanford Achievement Test (SAT 9) found that 57% of Open Court students fell into the bottom quartile while only 28% of students in a non-scripted program scored in the bottom quartile.
- The What Works Clearinghouse database shows that Open Court, Success for All, and Direct Instruction have positive, mixed, and zero effects, respectively6. In general, their analyses draw heavily on published academic research, so are likely to be reliable.
- An analysis of phonics instruction found no significant difference in mean reading scores over two years in classrooms with a scripted phonics program (SRA Reading Mastery) versus a non-scripted one7.
- It’s possible that even if a scripted program produces a boost in achievement, it is short lived. NAEP data from the last 10 years has shown that performance has not yet returned to pre-Covid levels and recovery between demographic groups is not equal, with the most marginalized students either continuing to lose ground or gaining at rates that are far below their wealthy peers. Since we know that marginalized populations have had the greatest and longest exposure to scripted programs, this should not be the case if those programs did what their marketing departments assured they would.
We aren’t able to see the experience level or measures of competence of the teachers in all these studies or whether they are fully certified. The most we can say about scripted curricula is that they might be better than an inexperienced teacher; mixed and zero effect results, however, show that they are not better than a highly competent teacher. There is ample evidence across decades of studies that the single biggest factor in student success is not any particular program but rather the competence of the teacher. More than one study has shown that students who have access to highly qualified teachers achieve a higher rate of success, and test score improvement differs substantially for students with different teachers in the same school and grade.8 Here’s more support for this point: An economic study*** from 2010 looked at the overall effectiveness of teachers based on district test data.9 They used this data to estimate standard deviations of effectiveness for teachers – i.e. those in the lower percentiles were less effective and those in the higher percentiles were more effective — and then looked at the achievement of the students assigned to each teacher. In math, having a teacher at the 75th percentile rather than the 25 percentile of the quality distribution meant a learning gain of 0.2 standard deviations in a single year, or moving a student in the 50th percentile to the 58th percentile – a big gain. Their finding was that teacher quality was an important determinant of achievement. A different economic study from Harvard in 2011 found that just having a highly qualified Kindergarten teacher (competent, experienced) had a profound positive effect on students’ future academic achievement, likelihood of attending higher education, owning a home, being employed, relationship stability, health, and future earnings – to the tune of an additional $300,000 in income over the student’s lifetime. Although neither study could specify the qualities of such teachers, both confirmed that having highly qualified teachers positively and profoundly impacts student achievement and not just in school.
In the next post, we’ll examine how scripted programs affect students.
_______________________
*This would be like Starbucks paying for a study that finds a lot of health benefits to drinking coffee. Researchers are unlikely to torpedo the source of their funding and that directly affects their findings.
**David Steiner, Executive Director of the Johns Hopkins Institute for Education Policy. He was instrumental in helping Texas implement “high quality curriculum,” a term that is in no way proven to be true. Texas is a huge market for educational materials and it’s likely that Steiner has profited greatly from this service, something that should inspire caution toward his assertions.
***Economic studies of education are great because they strip the research down to specific variables that are very different from educational research conducted by educators. At the same time, they are frustrating because they often don’t attempt to show why a phenomenon is occurring, only that it is.
___________________________________________
1 Fitz, J.A. & Nikolaidis, A.C. (2020). A democratic critique of scripted curriculum. Journal of Curriculum Studies, 52(2), 195-213.
2 Stockard, J. (2010). Promoting reading achievement and countering the “fourth-grade slump”: The impact of direct instruction on reading achievement in fifth grade. Journal of Education for Students Placed at Risk, 15(3), 218–240.
3 Ryder, R.J., Burton, J. L., & Silberg, A. (2006). Longitudinal study of direct instruction effects from first through third grades. Journal of Educational Research, 99(3), 179-192.
4 Borman, G. D., Dowling, N. M., & Schenck, C. (2008). A multisite cluster randomized field trial of open court reading. Educational Evaluation and Policy Analysis, 30(4), 389–407.
5Vaden-Kiernan, M., Borman, G., Caverly, S., Bell, N., Sullivan, K., Ruiz de Castilla, V., & Hughes Jones, D. (2018). Findings from a multiyear scale-up effectiveness trial of open court reading. Intervention, Evaluation, and Policy Studies, 11(1), 109–132.
6 Institute of Education Sciences. (2020). What works clearinghouse: Literacy. https://ies.ed.gov/ncee/wwc/FWW/Results?filters=,Literacy
7 McIntyre, E., Rightmyer, E. C., & Petrosko, J. P. (2008). Scripted and non-scripted reading instructional models: Effects on the phonics and reading achievement of first-grade struggling readers. Reading and Writing Quarterly, 24(4), 377–407.
8 Darling-Hammond, Linda. (1999). Teacher Quality and Student Achievement: A Review of State Policy Evidence. Education Policy Analysis Archives. Vol 8.
AND
López-Martín, E., Gutiérrez-de-Rozas, B., González-Benito, A. M., and Expósito-Casas, E. (2023). Why do teachers matter? A meta-analytic review of how teacher characteristics and competencies affect students’ academic achievement. International Journal of Educational Research 120, 102199.
AND
Sanfo, J. B. M., and Malgoubri, I. (2023). Teaching quality and student learning achievements in Ethiopian primary education: how effective is instructional quality in closing socioeconomic learning achievement inequalities?. International Journal of Educational Development, 99:102759. doi: 10.1016/j.ijedudev.2023.102759
9Hanushek, E. A., and Rivkin, S. G. (2010). Generalizations about using value added measures of teacher quality. American Economic Review 100, 267–271. doi: 10.1257/aer.100.2.267