In Part 1 we looked at some hard data that offered reasons why teachers use Teachers Pay Teachers (TPT) and other similar sites to supplement existing curriculum. Reasons ranged from trying to increase diversity and inclusion to trying to make curriculum more engaging for students to trying to fill holes left by district resources. In Part 2, we’re going to look at whether it’s a good idea for teachers to be trying to do these things using resources from online warehouses.
image via kidsconnect.com
The question of whether teachers should be using online resource marketplaces like Teachers Pay Teachers (TPT) to supplement existing curriculum needs to be looked at from the perspective of the various goals the teacher has for the material. We identified several in Part 1: to increase diversity and inclusion, to provide more engaging activities, to address gaps in the current curriculum with regard to alignment to state standards or tests. So let’s take each of these in turn.
To Increase Diversity and Inclusion? No.
Beyond the obvious (content creators on TPT are overwhelmingly white and female so not particularly diverse themselves) one study reported by the Kappan went a step further and took a sample of the most downloaded 11th grade American history activities from TPT to evaluate against diversity and inclusion criteria. What they found was that 75% of activities were rated either poor or moderate in their ability to offer diverse perspectives – they lacked nuance, connection to underrepresented cultures or contexts, and authentic representation of diverse people and communities. Only 25% were rated either good (15%) or excellent (10%) in these areas. Activities rated poor or moderate defaulted to white experiences in given historical periods or events, either ignoring other groups or misrepresenting their experiences. The fact that these were the most downloaded items for 11th grade American history is significant because many teachers use popularity of activities as a measure of quality; beyond that it may say something about what teachers are seeking out, or it may say something about what the teachers or the mostly not-diverse content creators view as important in a historical context.
This lack of diversity is not confined to U.S. History. A Fordham Institute study in 2019 rated 300 ELA activities from several online warehouse sites (including TPT) and found that 68% did not include diverse authors or address culturally diverse topics. TPT activities by themselves were even worse: 79% offered no inclusion of diverse perspectives. Lack of diversity appears to be a problem for online warehouses across content areas.*
To Increase Engagement? Also no.
The Fordham study found that the ELA activities they evaluated weren’t particularly engaging. Among TPT activities, 40% were rated mediocre or very unengaging. A further 53% were adequate – which presumably isn’t better than most textbooks — leaving just 7% of activities that were considered exceptionally interesting or engaging. That’s not much; a teacher’s odds of getting something really engaging are less than 1 in 10. And since students’ ability to see themselves in the curriculum is a major way of increasing engagement, the stark lack of diverse perspectives noted above reduces the likelihood of widely engaging material even further.
To Improve Alignment? Unlikely.
Alignment as defined by the study was grade level appropriateness and connection to state standards. The Fordham study found that the texts (and this is just a measure of text quality, not of overall quality) used in the ELA activities they evaluated were generally high quality and required students to use textual evidence, but the researchers’ measure of text quality didn’t specifically report out grade level appropriateness because it was a combination of three separate indicators. When that specific measure was separately reported, only 25% of texts utilized grade level content and reading levels (as measured by Lexile)**. In short, the texts were good, but teachers have only a 1 in 4 chance of getting something at the appropriate, expected reading level.
Next, the Fordham study looked at whether the activity aligned to state or local standards.*** Curricular activities (and every single textbook out there) are usually at some pains to stress their alignment to state standards. All teachers everywhere are concerned with whether the activity fulfills the mandate of teaching the standards. But – and this is true of textbooks as well – saying something is aligned doesn’t make it so. The Fordham study found that most materials reviewed were weakly aligned or not aligned at all (64% total). Teachers have about a 1 in 3 chance of getting something aligned, at least in high school ELA.
To Improve Existing Curriculum Quality? Really unlikely.
Even more interesting, when the Fordham study broke ELA activities out by online source, Teachers Pay Teachers texts in use achieved an exceptional rating only 30% of the time, while the other two sources’ texts (ReadWriteThink and Share My Lesson) were rated exceptional at least 50% of the time. ReadWriteThink and Share My Lesson activities were rated very low or mediocre 14% and 11% of the time respectively, while TPT activities were rated very low or mediocre 28% of the time. In ELA, teachers using TPT activities have roughly equal odds (about 1 in 3) of getting something really good, just meh, or really substandard. If the goal here is to make a bad curriculum better, the odds overwhelmingly indicate that TPT is not likely to achieve that end.
The Fordham study noted that all the activities they rated were generally free of errors and attractively designed. This doesn’t tally with CMSi’s experience with TPT activities, though by far the greatest proportion of what we see are activities designed for elementary rather than high school as in the study. In elementary pieces, we have seen grammar and usage errors, spelling mistakes, and even factual errors. We’ve also seen confusing wording of tasks and insufficient directions for completion across content areas. There’s no way for me to quantify this other than to say that the high school sample in the study seems somewhat different than the body of elementary samples we’ve seen.
When the Fordham study looked at overall quality, the picture shifts significantly. Overall quality for all online sources was overwhelmingly rated very poor or mediocre, ranging from 55% (ReadWriteThink) to 72% (TpT) of activities. These are not great stats if the goal is to shore up an already mediocre curriculum.
image via fliphtml5.com
The cognitive demand of activities and their ability to build content knowledge for students are also important measures of quality. Ideally, students would think critically about the content in that particular curricular area and build important knowledge (historical, content-specific, etc.). What the Fordham study found was that 58% of units across all online sources built student content knowledge weakly or not at all. In their words: “These units tend to be devoid of historical or literary content, focusing instead on skill building, simple recall, or personal interpretation without emphasizing any particular content (e.g., a unit on Romeo and Juliet without any reference to Elizabethan England or a unit on The Great Gatsby without any reference to the Roaring Twenties).” I will note that the Fordham study’s analysis of cognitive demand is problematic; based on my own experience rating student activities against cognitive taxonomies, I don’t agree with the categorizations they arrived at for the highest levels (DOK 3 and DOK 4). For example, the activity they described as DOK 3 I would have rated a DOK 4, so inter-rater reliability may be an issue.**** Also, the way they reported results is extremely confusing so I will confine my reporting to this: nearly half of all activities failed to reach DOK 3 or 4. This means high school teachers have roughly a 50/50 chance of getting something requiring higher order thinking. Based on what I’ve seen in elementary activities from online sources, I think the chances of higher order thinking and increased content knowledge are even lower — probably closer to Fordham’s engagement results of about 1 in 10, though that is purely a guess.
To Get Extra Support for Instruction? Nope.
For teachers seeking to improve differentiation, the Fordham study found that diverse learners weren’t supported by the materials. In fact, 86% of activities across all sources had no support for diverse learners at all (ELL, GT, IEP, etc.). More than that, there was little direction to teachers in how units and activities were to be implemented and even less direction on how to score assessments (i.e. rubrics, mastery definitions, etc.).
So, Should Teachers Use Online Warehouse Resources?
Many teachers indicated that they used online warehouse sources because it was expedient to do so; it saved them time and energy and allowed them to adapt something to their purposes rather than reinvent the wheel. Some teachers truly are in districts without curriculum — or in districts without appropriate grade level resources for their students — so finding materials somewhere is critical. The study conclusions, however, indicated that while there may be some quality material on Teachers Pay Teachers and other online warehouses, assuming that everything teachers find is of high caliber would be a mistake.
Any resource being considered needs to be carefully vetted across a range of metrics (quality, grade level appropriateness, alignment, content knowledge, cognitive demand, diversity and inclusion) to ensure it is suitable for use in district classrooms. Part of this vetting has to include what the activity is doing — building background knowledge, providing a step in a larger mastery process, practicing mastery performance tasks, etc. — so that teachers understand that completion of an activity doesn’t necessarily mean a child has mastered a standard. In cases where teachers are using online sources because the district doesn’t have specific curriculum or because the existing curriculum is poor or outdated or difficult to use, then the task of curriculum development needs to be undertaken as soon as is practicable and a range of aligned, appropriate resources should be identified and provided to help teachers meet the needs of their diverse learners. Relying on online warehouse resources to solve existing curriculum problems or take the place of a district-developed curriculum isn’t going to cut it.
*CMSi has noted multiple instances of elementary activities from TPT and similar that were (to put it mildly) culturally insensitive. One example was a packet on the Trail of Tears illustrated with cartoon figures of smiling Indian children. Another was an MLK Day activity with cartoon children who all appeared to be white and that encouraged students to write down something they dream about doing or having, which is not quite what Dr. King was getting at in his I Have a Dream speech.
**The authors point out – rightly – that lexile is not as important for literary texts, which often measure low on such scales even when they are considered high school level. Authors such as Ernest Hemingway used short, simple sentences with basic vocabulary as a matter of style, but no one would do Hemingway in 5th grade where the Lexile places his writing. The greater import of lexiles is in informational texts, which follow a more linear progression of difficulty as students move up the grades. The higher the grade level, the more complex and advanced the informational text. Experience with such texts is especially important for college readiness. Also, it should be noted that using reading selections below level to teach literary concepts is sometimes ok — it allows the student to focus on the concepts rather than on comprehending the text.
***I’m going to point out here that alignment to state standards is not as important as alignment to high stakes tests in use. Alignment to the test (which is not the same as teaching to the test) gives students the opportunity to practice what they need to know (in multiple dimensions) to be successful on the test well before the testing date. Alignment to state standards only indicates whether teachers are covering content – not whether it’s the most critical content or the right means of demonstrating mastery. An activity where students read a story and identify the protagonist’s character traits isn’t enough preparation when the task on the state test is to read two stories and write an essay comparing and contrasting the character traits of the protagonists from each. Contexts and cognitive demand are critical to informing how the content is taught and what mastery looks like.
****This is at least partially because DOK is just not specific enough in its descriptions and this leads to a lot of overlap and incorrect typing of activities. This is a major reason why CMSi uses New Bloom’s Taxonomy instead.
If your district would like help writing curriculum to better support instruction and improve learning for all students, Contact Us! CMSi offers an intensive, multi-phase curriculum development training that will help build quality, aligned, units that offer extensive support to teachers for planning and delivering instruction.