{"id":3407,"date":"2024-04-15T04:04:39","date_gmt":"2024-04-15T09:04:39","guid":{"rendered":"https:\/\/curriculumsolutions.net\/?p=3407"},"modified":"2024-05-01T09:18:02","modified_gmt":"2024-05-01T14:18:02","slug":"your-robot-instructor-will-see-you-now","status":"publish","type":"post","link":"https:\/\/curriculumsolutions.net\/blog\/2024\/04\/15\/your-robot-instructor-will-see-you-now\/","title":{"rendered":"Your Robot Instructor Will See You Now"},"content":{"rendered":"
\"\"

image via Yale Daily News<\/em><\/p><\/div>\n

Last week was STAAR Test week in Texas and the first time all public school students in the state took the newly updated and revised STAAR test.\u00a0 \u00a0This test represents a major overhaul in the content, context, and cognitive demand of the STAAR test, from an instrument that was almost entirely multiple choice with one generic writing task in ELA\u00a0 to an assessment which is much more rigorous and includes literature-dependent writing of an entirely different order.* The new STAAR also includes more open-ended items — questions where a child must construct a response in their own words rather than select a response from a list of possibilities.\u00a0 The Texas Education Agency (TEA), which oversaw the development of the new test, says there are approximately six to seven times as many constructed response items on the new test.\u00a0 More open-ended student writing means there’s an increased need for scorers to evaluate and mark those responses.\u00a0 Sometimes, decisions are like pulling a thread on a sweater. One change leads to another, which leads to another, and before long there’s a pile of yarn on the floor.\u00a0 In the case of the revised STAAR, these decisions culminated in an announcement from the TEA that it is rolling out an “automated scoring engine” for constructed response items.\u00a0 That’s a somewhat roundabout way of saying that constructed responses will be graded by AI — artificial intelligence.<\/p>\n

The TEA rather buried the lead here by focusing first and foremost on the cost savings.\u00a0 Instead of the 6000 human scorers needed to grade STAAR tests, they will be able to manage with fewer than 2000, saving the state $15 million. That’s no small thing. When questioned, the TEA revealed that they’d been using hybrid scoring since December 2023 (a combination of AI scoring and human scoring)**.\u00a0 \u00a0Mentioning that this type of scoring has already been in use is supposed to be reassuring. An explanation of how it all works is also meant to soothe worries about its efficacy.<\/p>\n

The AI system — which works like ChatGPT — was trained on 3,000 student responses that were first graded by two sets of human scorers.\u00a0 From this, the AI system is supposed to learn the characteristics of good, average, and poor responses and replicate that scoring on its own. Then, AI is supposed to evaluate its own work and assign a confidence level to the score it gave. If it gives itself a rating of “low confidence,” the response is flagged to be reviewed by a human scorer.\u00a0 It’s also supposed to flag responses it doesn’t understand.\u00a0 And just to be sure, scores are reviewed by TEA test administrators and a random sample of responses goes to a human scorer.\u00a0 \u00a0TEA calls this a “robust quality control” process.<\/p>\n

\"\"

image<\/em> via Dallas Morning News<\/em><\/p><\/div>\n

But there are hints that it might not be robust enough.\u00a0 At least one district whose students took the STAAR in December reported far more zeros on constructed responses.\u00a0 In the December testing, the state overall saw a sharp increase in zeros.\u00a0 The TEA\u00a0 says this is because new scoring guidelines allow student work to receive a zero if it is incoherent or provides no evidence for assertions.<\/p>\n

And this is where we run into some intersectionality. By that, I mean that the AI scoring is being implemented at the same time as the state-wide rollout of the new test. This concurrence is going to make parsing the issues much more difficult. For example:<\/p>\n