Study: Error Rate Of Test-Based Teacher Evaluations High, But Might Still Be More Accurate

wwarby / Flickr

The error rates of test score-based teacher evaluations is "quite high" according to a new report. But the evaluation may still be better than traditional performance measures.

A new study says error rates for teacher evaluations based on student test scores is “quite high,” but that the evaluations may still be more accurate than traditional measures.

The study, by Dan Goldhaber of the University of Washington-Bothell and Susanna Loeb, of Stanford University, might be cited as evidence as Florida’s largest teacher’s union challenges the state’s evaluation law in court.

But evaluations based on test scores may still be more accurate, the study argues, than traditional reviews based on certifications, years experience or observations.

“More accurate measures of teacher effectiveness can lead to better decisions by school and district leaders,” the authors write. “But the actions we take based on those measures can have both beneficial and harmful consequences. Clearly, we need a better understanding of the consequences of different systems on teacher development, teacher collaboration, and the desirability of teaching as a profession.”

The study also notes teachers are more likely to incorrectly earn a poor rating than to incorrectly earn a positive rating. That’s because most teachers earned positive rating under traditional reviews.

The study authors also examined the case law regarding teacher evaluations. Here’s their analysis:

Finally, how will the system handle legal challenges to the use of student growth measures for high-stakes teacher evaluation?  There are a number of relevant federal, state, and local laws that bear on this question.^[29]  Precedent suggests that courts generally will review the employment-related decisions of districts and schools (including as they relate to evaluation systems) with a significant degree of deference (EducationCounsel, 2012).^[30]  This deference likely extends to the judgment that educators should be evaluated in part on the basis of the academic gains of their students. Courts are also more likely to defer to districts when the state or district can demonstrate that the evaluation system in question was thoughtfully designed and consistent with sound educational principles.  A state or district can strengthen its position further by showing that it sought out and considered viewpoints by multiple stakeholders when it designed the system.^[31]

Additionally, courts and other arbiters are likely to recognize that there is more at stake than the employment interests of the aggrieved educator. There is, mainly, students’ right to a quality education, acknowledged by state constitutions. At least one federal court has opined that an evaluation system’s false positive, in which students continue to be instructed by an ineffective teacher who was classified as the opposite, “involves greater social costs” than a false negative.  Research suggests that previous evaluation systems did a poor job of distinguishing among levels of effectiveness and that the indicators used were not always correlated to strong instructional practice. Courts will likely recognize that value-added measures are intended to improve those systems.

The 2011 Student Success Act requires school districts develop teacher evaluations. At least 40 percent of those evaluations must be based on student gains on state standardized tests.

Next year, school districts must start paying teachers based on the results of those evaluations.

For more on how the evaluations work — and the criticisms — click here.