The NAEP Mystifies Spellings

Another failed Spellings test: People love to misstate about public schools. Consider Margaret Spellings’ op-ed column in this morning’s Post.

Spellings, Bush’s education secretary, was waxing about her own genius again. This is a miserable passage:

SPELLINGS (5/4/09): It's no accident that the United States has had nine straight years of increasing scores for elementary school students. In the decades before No Child Left Behind was signed into law in 2002 and the state reforms that led to it, taxpayers spent hundreds of billions of dollars on education and hoped for the best. Since No Child Left Behind, we have expected results. The law required that every student in grades three through eight be assessed annually in reading and math, that those results be disaggregated and that the information be provided to educators and parents. And that is exactly the age group for which we are seeing results. Consider: In the 10 years since 1999, reading scores for 9-year-olds have risen eight points; in the nearly three decades before that, scores rose only four points. In the past 10 years, math scores have increased 11 points, while in the nearly three decades prior, scores rose only 13 points.

For the record, it’s clear that Spellings is referring, in all particulars, to scores from “long-term trend” assessment conducted by the National Assessment of Educational Progress (the NAEP). For the reading scores in question, click here, then click ahead to page 9. For the math scores in question, click down to page 29.

Those are the score to which Spellings refers. But good God. The passage above is just awful.

For starters, Spellings claims “nine straight years of increasing scores for elementary school students.” Scores have gone up in the period in question, but uh-oh! The NAEP doesn’t test every year! In its long-term trend assessment, the NAEP tested 9-year-olds (and 13-year-olds; and 17-year-olds) in 1999, 2004 and 2008. There is thus no way to know if scores increased for “nine straight years.” In six of the nine years in question, no scores existed.

Spellings’ ineptitude spirals from there. Minor point: Few people will know what she means when she says that No Child Left Behind requires that test results “be disaggregated.” (Does the Post have editors?) But consider the problem with the way she describes the past decade’s score gains:

She starts by saying that we have expected results like this “since No Child Left Behind.” Immediately, she starts citing test results which predate the famous program! No Child Left Behind was signed into law in January 2002. It thus had no effect on the school year ending in June 2002, or on those which preceded it; its requirements were implemented somewhat gradually over the next year or two. Despite these obvious facts, Spellings seems to give the law credit for changes in test scores dating back to the 1998-1999 school year. Among 9-year-olds, scores bumped way up in the period between 1999 and 2004, substantially more than in the period from 2004 to 2008. Presumably, No Child Left Behind would have had relatively little effect on scores in that first five-year period. But Spellings attributes all the gains in the period since 1999 to the effects of No Child Left Behind. And of course, her Post editor lets her. (We’ll guess about motive tomorrow.)

But then, Spellings has always been good at one main thing: Inflating the greatness of her own program. Her technical skills almost always seem weak. Let’s consider another problem with her analysis of these data: Unless we’re mistaken, Spellings is actually understating the progress made by 9-years-olds from 1999 to 2008. As we’ve noted, a change in procedures created a bit of statistical complexity during this period. (This involves the inclusion of more kids who have disabilities or who are “English language learners.”) We’ll defer to those who may understand this program’s reporting regime better than we do. But if we’re not mistaken, Spellings understates when she says that 9-year-olds bumped up eight points in reading during this period. If we’re right, the greatness she grants herself in one way she takes away here, in another.

These NAEP data are very important. Spellings’ account of the data is clownish. Clearly, significant progress was recorded by 9-year-olds from 1999 to 2008—but this progress almost surely started before No Child Left Behind took effect. Indeed, it looks to us like the progress may have been a bit more pronounced before Spellings’ law took effect. But testing doesn’t occur every year. For that reason, it’s very hard to nail down claims like this using these limited data.

Spellings plays a lot of games in this piece, making things look very bright. But then, Obama played puzzling games last month, painting the opposite picture (see THE DAILY HOWLER, 3/11/09). People love to misstate about public schools. Question: Does anyone think this topic deserves to be reported with care?

h/t The Daily Howler