In Part 2, we identified three problems in education that arose out of the Covid 19 pandemic:
- A lot of teachers who left teaching because of the criticism and lack of support,
- A desire on the part of parents – and then state legislatures — to more strictly control what was going on in classrooms; and,
- Significant learning loss in students that desperately needed to be addressed.
We discussed the first two, but the third problem — whether these scripted curricula are effective at improving learning or addressing learning loss – needs more scrutiny. If they improve learning effectively and consistently for all students, we should be willing, as educators, to embrace their use. However, although improvement is one of the central marketing claims of scripted programs, research on their effectiveness is harder to pin down.
The “Big Tent” Effect of Englemann’s Direct Instruction
One of the things that muddies the waters is that Englemann’s original Direct Instruction (DI) model was effective, probably because of the alignment we discussed in Part 1. That effectiveness is frequently conflated with all scripted commercial programs, giving them some borrowed glamour that they are perhaps not entitled to, and that’s problematic. We can’t assume that the success of DI extends to other scripted programs. Direct Instruction was a research intervention for low-income preschoolers that specifically identified objectives and built in prerequisite skills to achieve them, providing alignment between the written, taught, and tested curricula at a time when that alignment generally didn’t exist. Today’s profit-driven programs vary significantly in quality and efficacy because of some key differences:
- Many programs don’t identify specific, measurable learning objectives with clearly specified mastery demonstrations; they plug in the state standards instead and standards don’t function well as objectives.
- Most – nearly all – programs are not aligned to the demands of any state test.
- Companies selling these programs want them to be applicable to more than just one state; it’s virtually impossible to find a commercially designed program that doesn’t assert its alignment to the state standards wherever it’s being marketed.
Given these factors, there are some very good reasons to regard all claims around alignment with caution. State standards aren’t uniform, even in states using the Common Core. Even if they were – and they are not – they aren’t suitable as objectives because standards don’t specify how mastery should be demonstrated, tests do. State tests are even less uniform than standards, with big differences in content, cognitive demand, and contexts in use, so alignment of any one program to the demands of multiple state tests is even less probable. CMSi almost never sees this kind of alignment in commercial programs. We sometimes see attempted test alignment in so-called test-prep materials, but these are uniformly of low cognitive demand and utilize the least engaging contexts. Effective alignment – the kind that produces improved learning – is theoretical unless someone in the district sits down and vets the program to confirm it is there.
Even then, the program might not do what district leaders hope it will. If it doesn’t offer practice in the content, contexts, and cognitive demands of the state test and exceed all of those, kids are unlikely to be either successful on the test or prepared for the real world. Why exceed? Because a state test can only measure a tiny portion of the total learning that occurs in classrooms. And frequently the contexts (how mastery is demonstrated) on a state test are not very engaging or even particularly cognitively demanding because they are subject to the constraints involved in grading them. Those constraints tend to push test contexts toward things that can be machine-graded. Confining all learning to those contexts and cognitive demands suppresses student engagement and leaves out all the critical thinking skills expressed through long-term projects, long-form reading, and extended writing that are both interesting and challenging for students. Such skills are important for preparing kids for adulthood – and in adulthood, the state test does not matter at all.
Direct Instruction vs. Other Scripted Programs
There don’t appear to be many studies offering comparison between DI and its imitators. I was able to find one study from 2010 that compared schools in the Baltimore City Public School district that were using Englemann’s Direct Instruction to schools using Open Court and schools using a variety of other reading curricula – meaning, they constructed lessons using parts of several programs rather than sticking strictly to a single program. They found that students in schools using DI had significantly greater gains in comprehension (21% gain between 1st and 5th grade) than those from schools using either Open Court or a mix of other curricula. Interestingly, the schools using a variety of curricula also had significantly better gains (14.9% gain) than those using Open Court alone (5% gain).* In vocabulary,** DI schools gained significantly more (18.8%) than schools using a variety of curricula (3.7%) and far more than those using Open Court where students actually lost ground (-7%). If students using a scripted program lose learning, that should be an enormous red flag for the district because in a scripted program the problem must lie with the program itself. That’s just one study, but it was a large and pretty robust one and strongly suggests that conflating DI’s efficacy with that of its imitators is a mistake.***
In the next post, we’re going to examine what the research shows about other scripted programs.
__________________________________________________________
*It seems likely that the schools using a mix of programs were using them as programs should be used – as resources to help them get kids to master a specific, measurable objective that is independent of the program in use. They pick and choose the best tools to meet their students’ needs rather than trust that the program will do it for them.
**This is no small thing; vocabulary is a huge predictor of academic success.
***I feel bound to note that the DI schools were also using materials from DI that were meant to supplement the reading instruction. These were designed to develop oral language and other foundational skills for literacy in grades K-1): students received 30 minutes of instruction daily using Language for Learning, Language for Thinking, and Reasoning and Writing. The researchers noted that the DI schools extended the DI curriculum through fifth grade. That curriculum “includes instruction in U.S. history, writing, reasoning, and spelling. The reading programs also include extensive general knowledge material related to the sciences and social studies. In general, the DI curriculum is designed to provide students with background knowledge and underlying schemata that can help them address and understand content throughout the academic curriculum.” (emphasis mine). The supplemental program is certainly related to literacy; I can’t decide if this is a confounder or not in the context of the whole study.