Inside a ‘Scoring Center’ in the Standardized Testing Industry

We speak to the author of Making the Grades about how standardized tests scorers grade essays.

Some educators have always been skeptical of standardized testing.

But in Florida, educators got another reason to question the accuracy of our test after the Florida Board of Education lowered the passing score of the Florida Comprehensive Assessment Test writing exam so more students would pass it.

A former employee with Pearson, the company that created and scores the FCAT, says the people who score the essays also lower their grading standards to produce the test results states predict their students will get, though Pearson denies the practice.

We spoke to Todd Farley, author of the book Making the Grades: My Misadventures in the Standardized Testing Industry.

Farley was a test scorer with Pearson from 1994-2000 and he trained scorers and managed project in the standardized testing industry until 2008.

And he tells us what it was like to score standardized tests.

Q: Do you think standardized tests produce accurate results?

A: No, the conclusion of my book, what I wrote was, I think what the standardized testing industry produces are numbers. So I’m never surprised to hear numbers went up or went down, because that’s all we did when I was in the business. There were innumerable instances when we were scoring, half way through the project you would go, we have a tremendous number of lower half scores and not enough upper half. So I would stand in front of a group and I would go, ‘hey all that stuff I’ve been telling you for two weeks, lets just forget that and let’s give more upper half scores.’ And all the scorers would moan and complain and I would think it was a scam and they would think it was a scam and then we would do it. Because every one of us was in there to get paid.

Q: But Pearson did give evaluations. There were tests for the scorers that you guys had to take in order to be considered qualified to be a scorer, correct?

A: Absolutely, qualifying tests, you bet. You didn’t have to do it for everything, you didn’t have to do it for the reading test necessarily or science test, but for the writing test you almost always did and I always tell the story, the first writing project I ever worked on, I was just a scorer, I wanted to work on this project. There were two qualifying tests to take. I failed them both and was fired. I would say half of the hundred people in the room were fired and we walked out in shame. Literally in shame. And then the next morning, at 7:30 am they called me back and gave me the job again. So what they had done was just lower the qualifying score. So suddenly all of us who had failed had passed. Because they hadn’t had enough people to finish the project and they needed to get them done, so there, they solved that problem. Seemingly the same thing as your state has done this year with the students.

Q: And again, you were an employee with Pearson, you no longer are. I spoke to Adam Gaber who is with Pearson now and he said in an email that “the scores of the Florida writing exam are accurate, valid and reliable.” The Florida Department of Education agrees. Let’s talk a little about how tests are graded. The multiple choice test are scored electronically so there’s probably not too much room for error there. But what about the open ended questions? The essays. Is there a standardized way to grade writing?

A: There are lots and lots of variables that decide scores other than just writing ability. Again as a human, you get there at 8:30 in the morning, maybe your coffee hasn’t kicked in, maybe you’re in a bad mood. You aren’t quite as generous as you are 5 minutes before lunch. Or maybe you’re more generous than you would be at 4:25 when your eyes are twirling around your head because you’ve read 250 essays. Secondly, it matters when in the project it gets scored. Because like I said, every project begins with the best intentions. And we’re all trying to do the best job. And then three weeks later someone is screaming at us: “give more higher level scores. So you might be giving threes to things you would be giving twos, two weeks before.

Pearson spokesperson Adam Gaber said in an email that it is “completely untrue” the company lowered the qualifying test scores for their scorers.

“All potential scorers undertake a very rigorous qualifying and interviewing process,” Gaber wrote.

Pearson: How Florida test scorers are trained

“Once scorers are hired, based on the hiring guidelines outlined by FDOE, scorers attend 3 to 4 days of training, practice scoring, and qualification,” Gaber wrote.

He says scorers first grade practice tests, which are then reviewed by Pearson.

“Before scorers can move to live scoring, they need to qualify on the item. The high qualification standards and criteria set by FDOE must be met before a scorer can score a live student response.”