The Philadelphia Story
by Paul Peterson
February 23, 2007
Judging by test-score results, Philadelphia schools have been enjoying a renaissance over the past four years. Thirty-two percent of Philadelphia's fifth-graders achieved proficiency on the Pennsylvania reading test in 2006, as compared to just 21% in 2002; math proficiency rose to 41% from 19%. At the eighth-grade level, there were 20 percentage point uptics in both reading and math.
Nevertheless, the respected RAND Corporation, in a study expected to influence the report of the Accountability Review Council for the Philadelphia schools that will be released later today, claims these positive results -- which came after a series of reforms including the engagement of private firms in the management of 45 schools -- are in fact deceptive and misleading.
In a press release early this month, announcing results from its latest study, RAND claims to have shown that "the four-year gains . . . were generally on a par with gains from similar low-achieving schools in the rest of the state." According to Jolley Christman, one of the report's authors, Philadelphia's "investment in private management of schools has not paid the expected dividends."
Needless to say, the study attracted wide attention. It also thrilled school choice critics, including Henry Levin of Columbia Teachers College, who said of the experiment in private management, "It's a romance that just hasn't worked. . . . They don't seem to be doing better, and they cost more."
These strong conclusions are not well supported by the RAND study, however. The widely quoted press release overstates what is actually in the report; the comparisons within the study itself are inexact; and vital information on school spending and other school characteristics is missing from the report.
First, a few words about the larger context. In 2002, fighting off fierce opposition, Pennsylvania placed Philadelphia's schools in the hands of a School Reform Commission, which then hired as superintendent the same Paul Vallas who had helped lift student achievement in Chicago. Among other innovations, Mr. Vallas audaciously turned 45 of the lowest performing schools over to private providers.
But reform did not come without a prolonged political struggle. Numerous compromises were struck, including concessions to unions that allowed many of the old regulations to remain in effect. Meanwhile, the unions kept a low public profile while apparently doing their best to undermine the reforms, especially by encouraging experienced teachers to leave the 45 schools assigned to private providers. In the first year, teacher transfer rates more than doubled at many of these schools. Despite this, test scores rose in most of the privately managed schools.
Now RAND says Philadelphia could have done equally well by doing nothing.
To prove that claim, one might expect RAND to have identified other big cities with extremely low-performing schools making equally striking gains -- but without launching reforms such as those undertaken in Philadelphia. Instead, RAND compared Philadelphia to other schools in Pennsylvania -- despite RAND's own admission (buried deep in its report) that "Philadelphia is unique among Pennsylvania school districts: Other schools in the state are only imperfectly comparable to Philadelphia schools."
In any event, RAND says that gains in Philadelphia "were generally on a par" with gains in "similar" schools elsewhere. But the report never identifies those "similar" schools -- not their names, not the characteristics of the students attending them, not their test performance back in 2002 before reforms began. RAND only tells us that the comparison schools were in the bottom 25% of all the state's schools, a very large category that contains a wide range of schools.
Still, Philadelphia schools do not look all that bad. While not making much progress the first year, they are estimated to have outperformed the comparison schools in the second and third years. For example, Philadelphia eighth-graders, in year three, are estimated to have gained eight to 10 points more than those in the comparison schools. In year four Philadelphia's edge slips. That could be a temporary blip; RAND instead concludes that Philadelphia's gains remained only at "par." Ordinarily, researchers make use of all available estimates, not just the most recent one, to draw their conclusions.
In its second analysis of schools within Philadelphia, RAND finds privately operated schools to have done no better -- but also no worse -- than would be expected from that same school, given citywide trends. In the words of the report's italicized conclusion: "We find little evidence in terms of academic outcomes that would support the additional resources for the private managers."
Given the restrictions placed on the private providers, such a finding would not necessarily be surprising. But RAND's way of reaching its conclusion is flawed; the research looked only at those students in certain grades who were tested on multiple occasions, a small, non-random percentage of all students. No information is provided on their representativeness.
More puzzling still is RAND's decision to expect these schools to exceed the overall citywide gains in test score performance. In so doing, RAND seemingly ignored the fact that the 45 schools assigned to the private managers were among the city's most challenging, scoring, in 2002, one to two years below the citywide average. A more suitable analysis would have expected them to gain more than students at comparably low-performing schools.
Finally, RAND questions the additional cost of private management, and says that the privately managed schools gained additional resources that the average school did not. But nowhere does RAND provide evidence on overall per-pupil spending at any Philadelphia school, making it impossible to tell whether the privately managed schools actually received more additional resources than other city schools.
It is more likely that per-pupil expenditure at the privately managed schools dropped when experienced, highly paid teachers left the schools at a much higher rate than before. RAND needs to show that that did not happen, or was offset by large increments in other kinds of expenditures, if it is to draw the conclusion that the privately managed schools had "additional resources."
Just how sensitive results are to RAND's specific methodology is unknown. But if education policy research is going to make a solid contribution to school reform, it needs to be both objective and keep pace with rising standards in educational research. In RAND's case, a forward step would be to make its Philadelphia data available to other qualified researchers, something it has so far been unwilling to do.