The testing expert hired by the state to determine the impact of computer problems that plagued the online ISTEP+ school tests said on Monday he was surprised by his findings that the computer glitches had no negative impact on student test scores.
Richard Hill, co-founder of the National Center for Improvement of Educational Assessment, told the state’s Education Commission that student scores actually improved this year, despite repeated computer problems that pushed about 80,000 students offline while they were taking the high-stakes assessment test.
Hill said that the vast majority of students who took the test scored as well as they would have had the interruptions never happened.
It wasn’t news that many commission members expected, nor news that was particularly welcomed by Indiana Superintendent of Public Instruction Glenda Ritz, who hired Hill in June and paid him $53,000 to do the study.
Ritz has been critical of the weight given to ISTEP tests; they play a key role in teacher evaluation and compensation and in the A-to-F rating system that gives a grade to every school in the state. Schools that consistently score poorly are at risk of being taken over by the state, while schools that score well are in line for more state funding.
“I was expecting to find several points lost at all grade levels,” Hill said, after the commission meeting. Instead, after analyzing ISTEP tests taken by all 495,000 students in grades 3 through 8 this spring, he found just the opposite: an across-the-board increase in test scores that almost mirror the increases of the past four years.
“I don’t know what the reaction is going to be,” Hill said of his analysis. “I’m sure it’s going to be one of general disbelief.”
It seemed that way among some commission members who grilled Hill on Monday about how he did his analysis and speculated that some students may have actually benefitted by having their tests interrupted, giving them more time to figure out test answers.
Hill rejected that notion and said he analyzed the data in many different ways to find some flaw in the test results. He found, for example, that students who were bumped offline for a few minutes didn’t fare much differently than students whose tests were postponed for a day.
“I cut the data a lot of different ways,” Hill said. “When you keep coming back to it, it’s clear: Student scores rose.”
Hill’s conclusions run counter to what Ritz had anticipated when she ordered the ISTEP analysis in May, after computer problems with the testing company, CTB-McGraw Hill, brought the ISTEP tests to halt in late April. The company later apologized for the problems, which it attributed to lack of memory space on its servers.
Ritz and other educators feared student test scores would be negatively impacted by the repeated interruptions and delays in the testing.
Officials in some school districts had called on Ritz to throw out the ISTEP test results. While she had stopped short of that, she did tell local districts in June that they could minimize the weight of the ISTEP results in determining teacher evaluations and pay. On Monday, she said she was sticking by that decision.
“I have spent the last several months talking with Hoosiers about the impact these interruptions had in the classroom,” Ritz said. “Although Dr. Hill’s report found that the statewide average score was not affected by the interruptions, there is no doubt that thousands of Hoosier students were affected.”
That decision by Ritz, a Democrat who’s been critical of the use of ISTEP scores in teacher evaluations, didn’t sit well with Republican House Education Committee Chairman Bob Behning of Indianapolis.
Behning has previously said that Ritz should have waited until the ISTEP analysis was completed. “Now it appears she misshot,” Behning said.
Individual ISTEP results have yet to be distributed to students and schools. Ritz said the DOE is now processing ISTEP student results, which will be available online to parents and students by late August.
In the meantime, the DOE is in negotiations with the testing company, CTB-McGraw Hill, to cover the cost of Hill’s analysis and to pay additional penalties for allegedly violating its contract with the state.
In his analysis, Hill credited students for recovering quickly from whatever stress the computer problems may have caused. He repeated that sentiment after Monday’s meeting, saying the ISTEP test may cause more stress for adults than students.
“They are high stakes for the adults,” Hill said. “But the kids don’t have same level of stakes and so don’t perceive it that same way. So when there is a glitch in the system, the adults tend to react more strongly to it than the kids do.”
Maureen Hayden covers the Statehouse for CNHI, the Tribune-Star’s parent company. She can be reached at email@example.com.
To access the interactive map (above, right) click here.