The Effect of Survivorship Bias on the Calculation of Charter School Graduation Rates
Charter schools market themselves as being superior to traditional public schools largely on the basis of student performance indicators such as standardized test scores and high school graduation rates. With respect to graduation rates, they usually offer a simple comparison of their rates with something like the statewide average rate for all public schools. You can see why. Charter school graduation rates are nearly always higher than the rates for other schools. According to the California Department of Education (CDE), the statewide graduation rate for all public schools in 2016-17 was 82.7 percent. Nearly all charter schools I reviewed exceeded that rate.
However, graduation rates don’t tell the whole story, as I learned from an examination of charter schools that promote themselves as being “college prep” and/or have been identified as being high performing, in large part on the basis of their high graduation rates. As it turns out, the calculation of charter school graduation rates suffers from “survivorship bias,” which makes them unreliable indicators of school performance.
In his fascinating book, How Not to Be Wrong: The Power of Mathematical Thinking, Jordan Ellenberg provides an illustration of survivorship bias at work. He recounts the story of Abraham Wald, a mathematician at the Statistical Research Group (basically a military think tank) during World War II. At that time, the U. S. military was confronted with the problem of how much protective armor to put on fighter planes and what parts of the plane needed the most protection. More armor made the plane less vulnerable to enemy fire, but also made the plane heavier, less fuel efficient, and less maneuverable. It increased defensive capability at the expense of offensive capability. On the other hand, less armor allowed a plane to be lighter and more maneuverable—thus increasing offensive capability—but made it more vulnerable to enemy fire. The challenge was to find the amount of protective armor that provided the optimal balance between a plane’s offensive and defensive capabilities and determine where on the plane the armor should be placed. This problem was assigned to Abraham Wald to solve.
The military provided Wald with data showing where planes had sustained enemy fire. When planes returned from combat, they were covered with bullet holes, which, on average were distributed on planes as follows:
Section of the Plane Bullet Holes per Square Inch
Engine 1.11
Fuselage 1.73
Fuel system 1.55
Rest of the plane 1.80
Using these data, the military asked Wald to figure out the optimal distribution of armor, assuming that the least amount of armor would cover the part of the plane that appears to sustain the least damage—the engine. Wald came back with the opposite conclusion—the engine should receive the most protection. This is because the data showing the distribution of bullet holes was based on planes that returned to base and did not include the planes that were shot down and did not make it back. Assuming that all parts of the plane were equally likely to be struck by bullets, Wald wondered why the distribution of bullet holes on the returning planes were unevenly distributed. His realization was that the returning planes were not representative of all of the planes that left the field that day, because they did not include the planes that did not return. Those planes must have had bullet holes, too, but where were they? As Ellenberg puts it:
Wald’s insight was to simply ask: where are the missing holes? The ones that would have been all over the engine casing, if the damage had been spread equally all over the plane? Wald was pretty sure he knew. The missing holes were on the missing planes. The reason planes were coming back with fewer hits to the engine is that planes that got hit in the engine weren’t coming back.
Wald’s conclusion was that the armor should not go where the holes were on the returning planes, but where they were not. Basing an analysis on only the planes that returned to the base and not all of the planes that left the base is an example of survivorship bias.
The calculation of charter school graduation rates provides another example. The 12thgraders in a charter school are the “survivors” from the total population of 9thgraders that were enrolled in the school four years before. The greater the gap between 12thgrade enrollment and 9thgrade enrollment four years before, the greater the effect of survivorship bias, and the less valid it is to use 12thgrade performance as an indicator of a school’s effectiveness. And yet charter schools do exactly that.
Data from the California Department of Education (CDE) show that the number of students that charter schools graduate from 12thgrade is far below the number of 9thgraders the same schools enrolled four years earlier. While normal student mobility may account for some of this, the data show that charter high schools systematically fail to retain and graduate as many 9thgraders as traditional high schools. The result of this survivorship bias renders charter high school graduation rates all but meaningless as a measure of school quality.
How Graduation Rates are Computed
I use the adjusted 4-year cohort graduation rates for 2016-17 provided by the CDE. This rate is the percentage of 9thgraders who graduate at the end of 12thgrade in four years. To account for student mobility, the 9thgrade cohort is adjusted for students who transfer in and out during the four-year period. So, to calculate the graduation rate for an individual school, the number of students who graduate at the end of 12thgrade is divided by the number of students who were enrolled in the school four years earlier (in 9thgrade) plus students who transferred in, minus students who transferred out at any time during that four-year period. With this methodology, students who drop out from the charter school are retained in the denominator. Dropouts cause the graduation rate to be less than 100%. But a student who transfers out of the charter school and then drops out from a different school prior to graduation has no effect on the charter school’s graduation rate, even if they don’t transfer out until grade 12. Those students count against the graduation rate of the school that they transfer to.
High School Retention Rates
To get a measure of possible survivorship bias I look at the retention rate for this same cohort, which is calculated by dividing the number of students enrolled in 12thgrade in a school by the number of students enrolled in 9thgrade in the same school four years later. In this case, “retention rate” does not denote the retention of the same students during the four-year period; rather, it denotes the retention of the same number of students. According to data provided by the CDE, the statewide retention rate for all public schools in 2016-17 was 98.5 percent. This means that the number of students enrolled in all public schools in 12thgrade in 2016-17 was almost equal to the number of students who were enrolled in 9thgrade four years earlier. The loss of 1.5 percent of 9thgrade enrollment is due mostly to the students who drop out. Students who transfer from one public school to another public school (including charter schools) do not have an impact of the calculation of the statewide retention rate.
The Charter Schools in this Analysis
Rather than use a randomly selected sample of charter schools, I focus on charter schools that specifically promote themselves as being college preparatory and/or that have been identified as high performing schools, as least partly on the basis of their graduation rates. These are schools that point to their graduation rates as evidence of their superiority. Among the schools in this study, three are KIPP schools, and seven are Aspire schools. Most of these schools have “college preparatory” or “university preparatory” in their names, signifying their focus on preparing their students for graduation and college admission.
According to its website, KIPP offers “an excellent college-prep education” that provides “personalize[d] learning based on a student’s needs, skills, and interests.” Similarly, the Aspire website states that, “our purpose is to prepare our…students for success in college, career, and life” and that Aspire schools have a “clear focus on College for Certain.”
I also looked at four Los Angeles Partnership Schools high schools. The Partnership is a collaboration between the Los Angeles Unified School District, the City of Los Angeles, and other public and private partners to “transform schools and revolutionize school systems.” It claims to have raised graduation rates since 2008 from 36 percent to 81 percent.
Finally, I include charter high schools that have been identified by either the US News & World Report or Great Schools.Org as being among the best public schools in the nation based, in large part, on their graduation rates.
Graduation Rates
Table 1 shows the graduation rates of the selected charter schools. It shows that all but five of the schools have a graduate rate that exceeds the statewide rate for all schools of 82.7 percent. And 17 of the 29 schools have graduation rates of 90 percent or higher, including two at 100 percent and one at 99.3 percent. These are, indeed, impressive figures and—by themselves—suggest a high level of performance. All things being equal, one would expect the charter school graduation rate to mirror the statewide average, with about half of the schools exceeding the average and half falling below it. The fact that 82 percent of these charter schools exceed the statewide average graduation is a strong indication that all things are not equal, and that there is a systematic reason for this. Charter school advocates have taken this “systematic reason” to be a superior education provided by charter schools. Under this line of reasoning, a charter school education is an intervention that, when applied to a population of students that resembles students statewide, produces superior results.
But we cannot look at graduation rates out of context, and when we look at the broader picture, we are forced to question whether charter school 12thgraders do, in fact, resemble their traditional school counterparts. Specifically, we must take a deeper look at the four-year cohort from which these graduation rates are computed. Table 1 shows that, on a statewide basis, the cohort graduating in 2016-17 had a retention rate of 98.5 percent. In other words, the 12thgrade enrollment for all California public schools in 2016-17 was 98.5 percent of the 9thgrade enrollment four years earlier, in 2013-14. By contrast, the retention rate for the charter schools in this analysis ranges from a low of 56.5 percent to a high of 97.1 percent. None exceed the statewide average. Assuming a normal distribution around the average of 98.5 percent, about half, or 14 of the schools in this analysis would be expected to be higher, and the other half lower. Instead, all of the charter schools are below the statewide average, and 24 of the 28 are far below the average, at less that 90 percent. This finding is too consistent across schools to be random.
The charter schools in this analysis have been selected because of their presumed superiority and their focus on college and university preparation. They claim to provide instruction that is student centered and equitable. Aspire Public Schools, for example, claim to be “committed to providing equitable opportunities for our students, families, and teammates. We use an equity lens to examine our policies, practices, and systems at Aspire to strive for all groupsto increase access and benefit from our work” [emphasis added]. Similarly, KIPP schools “blend small-group instruction and technology in creative ways to personalize learning and keep children encouraged, engaged, and continuously learning” [emphasis added]. The Stockton Collegiate International Secondary School “provides a multi-cultural, student-centered environment.” One would expect that schools with these welcoming, student-centered environments designed to serve all students and keep them engaged would be better at retaining them—at least as good as the statewide average—especially from among a population of students that affirmatively chose to attend those schools in the first place. Instead, these schools show a remarkable failure to retain students.
Under the methodology for computing the four-year cohort graduation rate, students who transfer away from a charter school for any reason are excluded from the denominator. Accordingly, when we combine the fact that charter school retention rates are consistently below the state average with the fact that charter school graduation rates are consistently above the state average, we are forced to that conclude that something systematic is at work. Specifically, the charter school 9thgraders who do not persist in their schools through 12thgrade must be among those least likely to graduate. Attrition among low performers would naturally increase a school’s graduation rate as well as improve scores on standardized tests. Charter schools may succeed not necessarily by providing a superior education, but by ridding themselves of lower performing students.
California law prohibits charter schools from “limiting enrollment access” of “academically low-achieving pupils.” However, nothing in the law requires charter schools to retain low-achieving pupils after they have been admitted and enrolled, and charter schools are not prohibited from expelling students for academic reasons. They may be expelled, counseled out, or simply discouraged from persisting at any time. Whatever the reason, or however accomplished, the pairing of a consistently higher-than-average graduation rate with a consistently lower-than-average retention rate leads to the inescapable conclusion that it is the low-performing students that charter schools fail to retain. Again, this finding is too consistent across schools to be random and strongly suggests that the attrition of low performing students from charter schools is part of the “magic sauce” that makes them appear to be better than traditional schools.
Policy Implications
Policy makers often look at performance indicators such as standardized test scores and graduation rates in isolation and conclude that charter schools provide a superior education. This leads to the further conclusion that the creation of more charter schools should be encouraged as a way to improve opportunities for more students. In fact, this would be consistent with the intent of the Legislature, as expressed in the California Education Code, that charter schools “Increase learning opportunities for all pupils…” However, the legislative intent goes on to specify that the increased learning opportunities should be “with special emphasis on expanded learning experiences for pupils who are identified as academically low achieving [emphasis added].” The evidence presented by this analysis suggests that charter schools, at least at the high school level, actually do the opposite: they weed out low performing students.
Because of this culling out process, a high graduation rate is not an indication that a charter school does a superior job of serving all of the students it initially enrolls. One could reasonably question whether a high graduation rate is a measure of the value that the schools brings to the students or of the value that the remaining students bring to the school.
The first lesson for policy makers is to recognize the effect of survivorship bias and look beyond performance indicators that exclude all students that charter schools enroll in order to understand better the reason why a particular school or program achieves exceptionally good (or poor) results. Performance indicators may measure results, but they do not explain how results are achieved. If good results are achieved by weeding out low performing students, it would be poor public policy to promote the expansion of those practices.
The second implication is the need to get a better understanding of the reasons for the high charter school attrition rates. Without surveying students who transfer out of charter schools, this is not easy to do. To collect data on charter school students who may have been expelled for academic reasons, the Legislature could fund the dropout report required by Section 48070.6 of the Education Code. Although the first report was due August 1, 2011, it has never been produced due to lack of funding. The law also requires that, when data are available, the report include “behavioral data by school and school district, including suspensions and expulsions.” The Legislature could specify that the suspension and expulsion data include the reasons for and outcomes of those actions.
California law also requires that “If a pupil is expelled or leaves the charter school without graduating or completing the school year for any reason, the charter school shall notify the superintendent of the school district of the pupil’s last known address within 30 days, and shall, upon request, provide that school district with a copy of the cumulative record of the pupil, including report cards or a transcript of grades, and health information.” This requirement could be expanded to report also to the school’s chartering authority, if it is different from the district of residence, and to include in the report the reason(s) for expulsions or transfers out. In addition, the report should also require verification that the student has actually enrolled in another school. This would prevent a charter school from erroneously reporting a dropout as a transfer. The Legislature could make these reports a condition of reauthorization and allow authorizing agencies to take this information into account when considering reauthorization.
| SCHOOL | 2016-17 GRADUATION RATE | DIFFERENCE FROM STATE AVERAGE | 2016-17 RETENTION RATE | DIFFERENCE FROM STATE AVERAGE |
| KIPP SCHOOLS | | | | |
| San Francisco College Prep | No data | n/a | 69.4% | -29.4% |
| King Collegiate High School* | 93.1% | 10.4% | 78.6% | -20.2% |
| San Jose Collegiate* | 88.0% | 5.3% | 77.7% | -21.1% |
| ASPIRE SCHOOLS | | | | |
| Golden State College Prep | 90.0% | 7.3% | 86.4% | -12.4% |
| Lionel Wilson College Prep | 81.5% | -1.2% | 77.4% | -21.4% |
| E. Palo Alto Phoenix Academy | 69.4% | -13.3% | 69.8% | -29.0% |
| Benjamin Holt College Prep | 100% | 17.3% | 81.2% | -17.6% |
| Langston Hughes College Academy | 93.9% | 11.2% | 74.3% | -24.5% |
| Vanguard College Prep Academy | 85.0% | 2.3% | 63.9% | -34.9% |
| Ollin Univ. Prep Academy | 92.5% | 9.8% | 91.3% | -7.5% |
| USNEWS TOP SCHOOLS | | | | |
| Stockton Collegiate International Academy | No data | n/a | 56.5% | -42.7% |
| Univ. Prep. Academy Ctr. | 93.0% | 10.3% | 68.8% | -30.0% |
| Preuss School YCSD | 95.3% | 12.6% | 87.4% | -11.4% |
| Animo Leadership High | 93.8% | 11.1% | 80.6% | -18.2% |
| Summit Prep. Charter High | 94.4% | 12.0% | 64.1% | -34.8% |
| Leadership Public School, Richmond | 94.1% | 11.4% | 94.3% | -4.5% |
| Oakland Charter High | No data | n/a | 97.1% | -1.7% |
| GREATSCHOOLS.ORG | | | | |
| New West Charter, LA | 91.2% | 8.5% | 84.1% | -14.7% |
| Bright Star, LA | 89.9% | 7.2% | 73.9% | -24.9% |
| High Tech High, San Diego | 99.3% | 16.6% | 87.3% | -11.5% |
| University High, Fresno | 97.3% | 14.6% | 86.2% | -12.6% |
| Stockton Early College Academy | 100% | 17.3% | 77.1% | -21.7% |
| University Prep Academy, San Jose | 93.0% | 10.3% | 96.9% | -1.9% |
| Gateway HS, San Francisco | 94.3% | 11.6% | 85.3% | -13.5% |
| LA PARTNERSHIP SCHOOLS | | | | |
| David Starr Jordan HS | 66.0% | -16.7% | 87.9% | -10.9% |
| Felicitas & Gonzale Mendez HS | 90.0.% | 7.3% | 75.9% | -22.9% |
| Santee Education Complex | 81.3% | -1.4% | 62.9% | -35.9% |
| Theodore Roosevelt HS | 78.4% | -4.3% | 82.6% | -16.2% |
| STATEWIDE (ALL CHARTER AND TRADITIONAL SCHOOLS) | 82.7% | | 98.5% | |
*Also on the USNews Top Schools list
