Five Questions to Ask About the Dayton Daily News/Atlanta Journal-Constitution Cheating Investigation

C. Carlstead / Flickr

A Dayton Daily News/Atlanta Journal-Constitution national investigation into cheating on state standardized tests suggests that some Ohio schools may have cheated on state standardized tests, but stops far short of saying that cheating is common in Ohio school districts or in districts nationwide.

The Atlanta Journal-Constitution says they found a pattern of suspicious test scores in about 200 school districts nationwide. That’s less than 2 percent of all districts. And the Dayton Daily News is holding off identifying schools or districts with suspicious scores until they can do more analysis.

Using statistical analysis to Identify suspected cheating is complicated, to say the least. So we’ve put together the five big questions you should be asking as you read these stories or look at the Atlanta Journal-Constitution’s online database.

1. How did the newspapers identify suspected cheating?

Their analysis looked only at reading and math tests in grades three through eight. It started by comparing the average score for a group of students at each school in a subject area in one year (say, third grade math) to that same “class” of students at the same school the following year (say, fourth grade math). Using results from the entire state, they calculated the probability of that class’s average score changing that much from year to year. They decided that classes with scores changes with probabilities of less than 5 percent were unusual. Then they looked for “improbable clusters of unusual score changes” within districts. (Here’s the full explanation from the Atlanta Journal-Constitution.)

The Atlanta Journal-Constitution had earlier identified a similar pattern of swings in test scores in the Atlanta school district. Eventually state investigators confirmed widespread cheating was taking place.

However, the Ohio Department of Education doesn’t agree with this way of looking at the test scores. Spokesperson Patrick Gallaway says the 5-percent threshold the newspapers used is lower than the one that the state education department uses in its analysis of test data. And the Dayton Daily News has not yet looked at other evidence — like erased answers or reports of teachers or administrators acting inappropriately — that could support allegations of cheating in Ohio schools. Reporter Ken McCall says they plan to investigate further in follow-up stories.

2. How common is cheating in Ohio schools?

The Dayton Daily News found more than 2,600 “improbable changes”– both up and down — in Ohio schools between 2005-2011. They say it’s “unlikely all of them can be explained away by the quality of instruction, demographics, or changes in mobility and class size.” The changes were not distributed evenly around state: McCall says many of the suspicious test scores were clustered in a smaller number of districts.

3. Were there any differences between the rates at which charter schools saw suspicious test score changes compared to traditional public schools?

The Atlanta Journal-Constitution says that suspicious test score changes were twice as likely to appear in charter schools nationally as traditional public schools. McCall says Ohio charter schools also saw a higher rate of improbable score changes than traditional public schools. But the newspapers have not yet publicly shared data to back up those claims. The Atlanta Journal-Constitution’s online database of school districts with improbable score changes lists only traditional public school districts.

4. Which districts are cheating?

Hard to say. The newspapers’ analysis only identified school districts with schools that saw an improbable number of improbable changes in the test scores of groups of students from year to year. Those changes could be due to factors like good teaching or bad teaching or students entering or leaving a school. They don’t necessarily mean that staff or students at a school cheated.

The Dayton Daily News hasn’t identified any particular districts yet. “We’re going to take a closer look before we name any names,” McCall says.

The Ohio Department of Education does not intend to investigate the flagged schools.

The Atlanta Journal-Constitution has an online database that shows the traditional public school districts they identified as having suspicious changes in their test scores. It identifies over 100 Ohio school districts with improbable test score changes. But as we noted above, that database doesn’t include charter schools. And it seems to have some blind spots in identifying some Ohio school districts, listing them only as ”/N.” (We’ve emailed the Atlanta Journal-Constitution team about the database and will update this once we hear back from them.)

5. How will switching to online tests change things?

In 2014, Ohio schools will start using online tests instead of the current paper-and-pencil models. The Ohio Department of Education says the switch will make “the issue of erasing answers on computer-graded bubble tests… moot.”

But cheating isn’t limited to students or staff erasing answers. It can also be a teacher providing additional information to students, or students sharing information among themselves, or any number of other activities that could still take place, even after the switch to online tests.