Ohio

Eye on Education

Grading the Teachers: How Ohio is Measuring Teacher Quality by the Numbers

Forest Park Middle School teacher Maria Plecnik said for her own sanity, she had to leave the teaching profession.

Lynn Ischay / The Plain Dealer

Forest Park Middle School teacher Maria Plecnik said for her own sanity, she plans to leave the teaching profession.

Maria Plecnik is the kind of teacher who gets chills in a 90-degree classroom when she connects with students during the first week of school. She’s the kind who brags about seeing their test scores go up or turning a kid who was always trouble into an “A” student.

In her seven years at Euclid’s Forest Park Middle School, her principal always told her she was doing a good job.

Teaching was her dream job. But this year, her dream faded.

Ohio is introducing a new way of grading teachers, one based on student test scores and whether students learn as much as expected in a given year. Under that system, Plecnik received the lowest rating, “Least Effective.” She cried when she found out.


GTT logo-resize
This series about valued-added, a new way that Ohio is using to measure whether teachers provide a year’s worth of learning to their students, is the result of a partnership between The Cleveland Plain Dealer and StateImpact Ohio. StateImpact reporters Molly Bloom and Ida Lieszkovszky worked with Plain Dealer reporter Patrick O’Donnell and Plain Dealer data analysis editor Rich Exner to produce these stories.

Multimedia

Data

“Once I saw it in black and white, my heart broke,” she said.

Plecnik’s experience puts her in the middle of one of the biggest transformations to hit the classroom in the past 50 years.

Teacher evaluations, once based largely on a brief classroom visit from a principal every few years, will change dramatically next school year.

Ohio is one of 32 states experimenting with more rigorous, data-driven systems of grading teachers. In Ohio’s case, teachers earn one of five ratings that are a key part of their overall “final grades.” Those grades range from “Most Effective” at the top end of the scale to the “Least Effective” rating Plecnik received.

The rating is based on a statistical measure called “value-added.”

Most teachers — about 70 percent — fall into the middle, average categories. The state started calculating value-added for some teachers in 2010-11 as part of a pilot program and is phasing in the ratings.

What’s riding on these grades varies depending on how school districts choose to use them. But more than a teacher’s pride is at stake. The ratings could eventually become part of decisions about how much teachers are paid, what classes they teach and, if a district has to lay off teachers, their place on the list of who stays and who goes.

Many policymakers view this data-driven approach to sizing up teacher performance as crucial to weeding out bad teachers and rewarding good ones. But some teachers see the measure as a flawed attempt at quantifying something that isn’t easily quantifiable.

The scores offer a new view into what’s going on inside Ohio classrooms. StateImpact Ohio and The Plain Dealer obtained the value-added scores and ratings of about 16,000 teachers across Ohio in reporting this series. 

Managing by the numbers

Policymakers love value-added because it picks up on two trends in education policy.

The first is the push to quantify every aspect of how schools operate and hold schools accountable for showing student progress through test scores.

The second revolves around research suggesting that having a great teacher is the most important factor in determining whether students succeed in school. That research suggests it’s crucial to find ways to tell great teachers apart from less-than-great teachers.

Teacher Maria Plecnik received high marks from her principal, low marks for her students' test scores. Read excerpts from her evaluation.

Teacher Maria Plecnik received high marks from her principal, low marks for her students' test scores. Read excerpts from her evaluation.

That’s not something that most current teacher evaluation systems do well. A 2009 study of a diverse group of districts in four states, including Ohio, found that less than 1 percent of teachers received unsatisfactory ratings, even in chronically low-performing schools.

With better measures of teacher quality in place, supporters say, the best teachers can be encouraged to continue teaching by offering them bigger raises, recognition and extra perks like more planning time. The worst teachers can be given intensive training. And if they don’t improve, they can be encouraged to leave the teaching profession through the stigma of receiving low marks, or be fired.

Plus, value-added is relatively cheap and easy to put in place compared with other school improvement efforts like those that involve hiring additional staff.

Value-added ratings can be one objective measure of how good a teacher really is, said Mary Peters of Battelle for Kids, which has helped implement Ohio’s teacher value-added ratings.

“Not just because they raise the most money and are the most popular in their building and parents love them and bake them cookies. But because quantitative data points to the fact that they are a great teacher,” she said. 

Longtime teachers divided on ratings

Toledo teacher Emily Brown has received state recognition as a top teacher, in part based on her value-added scores.

Ida Lieszkovszky / StateImpact Ohio

Toledo teacher Emily Brown has received state recognition as a top teacher, in part based on her value-added scores.

Many teachers have a more skeptical view of value-added.

Olivia Carter has taught in Toledo for 28 years. When she got her rating from her principal last year in an envelope, she took it home and put it in her “important papers” pile. The sealed envelope is still there.

The information inside wouldn’t tell her anything new, she said.

“Being in the classroom, I know what I do. I can see it in the children every day,” she said.

To Carter, value-added, like standardized tests and state school ratings, is just another measure being imposed on Ohio teachers under the banner of “reform.”

But Emily Brown, a colleague of Carter’s at Toledo’s Grove Patterson Elementary Academy, sees value-added in a more positive way.

Brown said she taught for 13 years in the Toledo public schools before her bosses told her she was doing a really good job. Last summer, she got a text from her school’s principal. Brown had received the highest state rating, Most Effective.

“It’s nice to get that pat on the back,” Brown said.

“Being in the classroom, I know what I do. I can see it in the children every day.”

–Teacher Olivia Carter

And for Katie Zielke, a math teacher at Columbus’ Johnson Park Middle School, receiving a low value-added rating was like a painful wakeup call.

Zielke had been teaching for 20 years and never heard any complaints from principals or colleagues. But two years ago, she learned that she had received the lowest possible value-added grade. The low rating was a surprise to her and to her principal.

“I kind of went, ‘Wow, what I was doing was not right,” Zielke said. “I was doing what I thought was best, but obviously I have to do something different.” 

Scoring expectations versus outcomes

Value-added scores are about masses of cold, hard data. Specifically, the data gathered from state standardized tests.

Value-added scores are calculated by comparing how much students in each teacher’s class learned during the course of the year with how much they’re expected to learn.

How much they learn basically means how much their scores on state tests changed year to year. How much they’re expected to learn is determined by looking at how much other students at similar achievement levels have learned from year to year.

The better students do in comparison to their peers, the better their teacher’s score. That differential is assumed to be the teacher’s doing — the “value” he or she adds in the classroom. The Ohio Department of Education translates those scores into one of the five ratings.

16CGVALUE A.indd

Those ratings are still something of an experiment. Only reading and math teachers in grades four to eight get value-added ratings now. But the state is exploring how to expand value-added to other grades and subjects.

Among some teachers, there’s confusion about how these measures are calculated and what they mean.

“We just know they have to do better than they did last year,” Beachwood fourth-grade teacher Alesha Trudell said.

Some of the confusion may be due to a lack of transparency around the value-added model.

The details of how the scores are calculated aren’t public. The Ohio Education Department will pay a North Carolina-based company, SAS Institute Inc., $2.3 million this year to do value-added calculations for teachers and schools. The company has released some information on its value-added model but declined to release key details about how Ohio teachers’ value-added scores are calculated.

The Education Department doesn’t have a copy of the full model and data rules either.

The department’s top research official, Matt Cohen, acknowledged that he can’t explain the details of exactly how Ohio’s value-added model works. He said that’s not a problem.

“It’s not important for me to be able to be the expert,” he said. “I rely on the expertise of people who have been involved in the field.” 

Fluctuations, other issues spark fairness concerns



Katie Zielke's principal called her a "great" teacher, but under value-added she received the state's lowest rating.

Molly Bloom / StateImpact Ohio

Katie Zielke, pictured here, has been called a "great" teacher by her principal, but under value-added she received the state's lowest rating.

But many teachers believe Ohio’s value-added model is essentially unfair. They say it doesn’t account for forces that are out of their control. They also echo a common complaint about standardized tests: that too much is riding on these exams.

“It’s hard for me to think that my evaluation and possibly some day my pay could be in a 13-year-old’s hands who might be falling asleep during the test or might have other things on their mind,” said Zielke, the Columbus middle school teacher.

A StateImpact/Plain Dealer analysis of initial state data suggests that teachers with high value-added ratings are more likely to work in schools with fewer poor students: A top-rated teacher is almost twice as likely to work at a school where most students are not from low-income families as in a school where most students are from low-income families.

“It’s hard for me to think that my evaluation and possibly some day my pay could be in a 13-year-old’s hands who might be falling asleep during the test or might have other things on their mind.”

-Teacher Katie Zielke

Ohio Department of Education officials say that’s probably because students in those schools with mostly poor students aren’t learning as much as students in other schools.

Teachers say they’ve seen their value-added scores drop when they’ve had larger classes. Or classes with more students who have special needs. Or more students who are struggling to read.

Teachers who switch from one grade to another are more likely to see their value-added ratings change than teachers who teach the same grade year after year, the StateImpact/Plain Dealer analysis shows. But their ratings went down at about the same rate as teachers who taught the same grade level from one year to the next and saw their ratings change.

In general, a teacher’s rating is unlikely to swing from one extreme to another year to year: About 3 percent of teachers who received the state’s top rating in 2010-11 received the state’s lowest rating in 2011-12.

Critics say that even less drastic year-to-year swings make the measure unreliable. Supporters say performance can change from year to year, some likening it to sports statistics like a batting average in baseball — measures for which year-to-year change is considered normal.

Some research has suggested that value-added ratings tend to agree with ratings from principals. But even that research showed that value-added ratings and principal ratings don’t line up exactly.

Johnson Park Middle School Principal Kevin Kinne says a value-added score can't capture everything a teacher does.

Molly Bloom / StateImpact Ohio

Johnson Park Middle School Principal Kevin Kinne says a value-added score can't capture everything a teacher does.

After all, said Kevin Kinne, principal of Zielke’s school in Columbus, value-added doesn’t take into account some of the most important things teachers are supposed to do — like care for students.

“It doesn’t tell about the kid that you drove home and saved from getting beat up by whatever situation because you cared enough to sit down and talk to them,” Kinne said. “And it doesn’t tell about the kid who came to school with some crazy parent situation and you were able to calm them and help them make it through the day. It doesn’t tell you any of that.”

In Ohio, value-added ratings and other growth measures will count for about half of a teacher’s final rating under the new evaluation system that will soon be in place in all school districts. Classroom observations and measures of professionalism make up the other half.

Officials at teachers unions say value-added ratings can be valuable tools but shouldn’t be used as the main factor in making decisions about teachers’ pay or hiring and firing.

Still, value-added ratings for teachers could be useful if many of the problems with value-added were fixed, said Arizona State University professor Audrey Amrein-Beardsley, who has studied value-added measures.

If value-added was shown to align with other measures of teacher quality, be independent of factors outside of teachers’ control and produce relatively consistent ratings from year to year, the model could be a good thing, she said.

“But in the current place we are, we are not ready for prime time,” she said. 

Low score pushes one out the door

Teacher Maria Plecnik's supervisor wrote she "truly cares...about her students' development."

Lynn Ischay / The Plain Dealer

Teacher Maria Plecnik's supervisor wrote she "truly cares...about her students' development."

Stephon Fletcher started this year in Maria Plecnik’s classroom as a kid who would mouth off to teachers and do as much work as he felt like — which usually wasn’t much, he says.

He’s different now, he says, because of how and what Plecnik taught him. She showed that she respected him, and he in turn learned to respect her. His grades moved from mostly D’s and F’s to mostly A’s and B’s. And he’s at work on a book, tentatively titled, “Life and Why Things Happen the Way They Do.”

Stephon says the idea of Plecnik being an ineffective teacher is “outrageous.”

But Plecnik is through. She’s quitting her job at the end of this school year to go back to school and train to be a counselor — in the community, not in schools.

Plecnik was already frustrated by the focus on testing, mandatory meetings and piles of paperwork. She developed medical problems from the stress of her job, she said. But receiving the news that despite her hard work and the praise of her students and peers the state thought she was Least Effective pushed her out the door.

“That’s when I said I can’t do it anymore,” she said. “For my own sanity, I had to leave.” 


What do you think makes a good teacher? Join the conversation on Twitter at #teachereval #ohedchat.


StateImpact Ohio reporter Ida Lieszkovszky, Plain Dealer reporter Patrick O’Donnell and Plain Dealer data analysis editor Rich Exner contributed to this story.

Comments

  • Ed4all

    This series barely scratches the surface of the new teacher evaluation system, and is missing many discussions of key issues that need to be addressed. The legislature has mandated the new evaluation system at a time of great change and where ODE is building the plane as they are flying it. The state has committed to transitioning to the new PARCC evaluation system instead of our current OAA/OGT assessments. These new tests will begin to be implemented in 2014 and are still in development. How they will be administered, in what subjects, and using what standards is still in draft form, and the transition documents are rough.

    How will we be rating the thousands of teachers that teach areas other than reading and math, ate all grade levels? The current answer that I’ve been given is that districts (teachers) will be responsible for developing their own tests that will be used as measures, and there is no plan in place to audit these exams for content, reliability, or adherence to standards. Another unfunded mandate that districts can’t afford to managed and have a limited timeline to execute. How is this fair? Some teachers will be evaluated (and paid) based on high stakes measures that are developed by companies experienced in developing assessments, and others will be evaluated on tests that they or colleagues (often with only one undergraduate course in assessment development) have developed themselves.
    These are important issues that deserve mention in the scope of your “investigation,” in addition to the obvious flaws and concerns in the value-added formula itself, as well as the bias and problems experienced in the pilot year of the OTES program.

  • Ohio Teacher

    I will be losing four planning periods a month to sit in mandatory meetings and discuss data rather than focusing on student needs. When they realized that No Child Left Behind was a joke, they baited and switched us with Race to the Top. When did education become a race instead of an individualized process?

    • OrvsKid

      No Child Left Behind left EVERY child behind. Every program since has become successively worse. This started because politicians who have not spent an appreciable amount of time in a classroom since they were students themselves decided they would convince the voting public that they could improve the schools that didn’t really need improving….until the politicians got involved. Test scores in the US appear to be lower because we have to test everybody, including the learning disabled kids. In other countries they pick and choose which students are tested that elevate their scores. This is common knowledge. Also, Norway, which is the highest scoring country, doesn’t start their children until they are 7, and much better prepared as well as more mature. Until the American public wakes up and sees that testing is NOT the only way to asses a child, and that they are being manipulated by politicians that are more interested in their votes than their children, all schools will continue to decline, teachers will leave the profession, and they will be replaced by those who will take the abuse because they can do nothing else. I, and many other teachers who are forced to retire early will be leaving in approximately 722 days. But who’s counting.

  • Rep Voter

    Only 505 days till this Republican in Cincinnati votes for a Democratic Governor!!!

    • Enough Politics

      This Republican in Columbus will be voting for a Democratic Governor just like you!

      Here’s a thought- why don’t we evaluate the governor on how he eliminates poverty, drugs, and crime? There must be a test for that! How about letting all the top brass in the Governor’s office take the OGT? Scores could be posted in the newspapers.

  • pdawg20

    This is so flawed it’s not worth discussing. VAM just doesn’t have supporting evidence.

  • Melissa Schreiber-Peck

    I have to say this is sad. I don’t agree one bit that the teachers should be blamed for children’s scores being low. I do however agree with part of the grading scales on the schools, being left on the teachers classroom performance. Many parent treat city schools as if they are nothing but a daycare for there children. education doesn’t just come from schools, it comes from parents that care about their children excelling in the education. Teachers do their part during the day, but what about evenings and weekends. If your child is doing badly in a subject , instead of them playing outside all weekend how about parent do some tutoring at home. In my opinion it takes the care of all the adults present in that students life to help them achieve. Also it depends on how bad the child wants to do better as well. Teachers can teach till they are blue in the face, but if we have these smarty pants kids coming in thinking they are going to act as they do on the streets, we have a problem. The teacher are there to teach, they are not there for teaching these children how to act, that job is on the parents. So I think that we should give these teachers a break, many are trying as hard as they can.

  • Jill Thomley
About StateImpact

StateImpact seeks to inform and engage local communities with broadcast and online news focused on how state government decisions affect your lives.
Learn More »

Economy
Education