Pulling the Curtain Back on CCSA

Leaked documents received by the blogger Michael Kohlhaas (a fictitious name, check it out) at michaelkohlhaas.org provides an interesting insight into the long-term goals of the California Charter School Association (CCSA).  At its executive summit that took place last October, CCSA considered several changes to its “Strategic Snapshot,” including a proposed change to its vision statement.  Documents prepared for that meeting show that the current vision is “Increasing student learning by growing [sic: “growing” is an adjective, not a verb] the number of families choosing high quality charter schools so that no child is denied the right to a great public education.”  The focus of this statement is exclusively on increasing the number of charter schools.

The proposed new vision is “Providing absolutely every young person a great public education by growing the number of families choosing high quality charter public schools and encouraging all public schools to become more charter-like” [emphasis added].  

The CCSA documents acknowledge that, up to now, it has been “silent on what is supposed to happen to the rest of public education [and] we have not been able to agree about what we want the public education system to evolve into.” But now that they’re getting “blowback” from people “who assume that the charter school movement is one big replacement strategy” they need a vision statement to allay those concerns.  In other words, the purpose of the change is to quell concerns within the traditional school community that the long-term goal of organizations like CCSA is some sort of hostile takeover of all public schools. Quoting again from the document, “By showing that we believe that all public schools can evolve to become more charter-like, we are signaling that we believe the end state we are moving to is one where, yes, all public schools will be charter schools or charter-like schools, but that we believe a part of the equation is that many existing public schools will be able to remake themselves so that they may play an important role in the future of public education.”  

The phrase, “we believe that all public schools can evolve to become more charter-like” is both condescending and arrogant.  It assumes a superiority for charter schools that does not stand up under close scrutiny and ignores the troubling level of corruption that distinguishes charter schools from traditional schools.  In addition, nowhere does CCSA explain what it means to be “charter-like.” This is a curious omission given the centrality of this concept to its new vision statement.  Let me take a stab at it.

At the classroom level, there is little, if anything, to distinguish between a charter school and a traditional school.  (Of course, I’m excluding virtual schools and charters that are organized as home schools).  Both address the same content standards with similar curricula and instructional materials. Both employ certificated teachers. And both are subject to the same state testing requirements.  

But apart from the classroom there are some important differences.  For example, a “charter-like school” is a school that:

  • Is only indirectly accountable to a publicly elected governing board for management and is instead governed by a self-appointed group of individuals.  
  • Is beyond the reach of a publicly elected governing board to take any corrective action for academic or financial failures short of the “nuclear option” of closing the school through non-renewal or revocation of the charter.
  • Can expel a student for poor academic performance.
  • Does not have to follow statutory due process procedures when suspending or expelling a student.
  • Allows teachers to teach classes for which they are not credentialed.
  • Is not required to participate in PERS or STRS.
  • Is exempt from the Field Act (earthquake safety requirements).
  • Has (on average) a higher rate of suspensions/expulsions.

There may be other differences that don’t come to mind right now.  But the point is this is a troubling vision for the future of public education.  It will be interesting to see how CCSA responds to this disclosure. 

Public Funds for Private Fun

How to Go to Disneyland on the Public Dime

On July 24, the Los Angeles Times ran an article that detailed how home schools that operate as charter schools use taxpayer dollars to pay for trips to places like Disneyland, Medieval Times, and SeaWorld as well as private horseback riding lessons and other extracurricular activities.  Public funds have even been used to purchase family memberships at the San Diego Zoo.  Some charter schools provide home school families as much as $3,200 per year for these purposes.  You can read the article here, and Diane Ravitch also covered it here.

Some expenses, like ice skating classes, or acting classes (!) qualify as physical education, while trips to Disneyland or SeaWorld are considered field trips.  Generally, home school families can use public funds to make purchases from a list of charter school-approved materials and activities.  Vendors typically must have their products and services approved by the charter school to have them on the list.  In this sense, the public funds are like script that can only be spent at the “company store.” 

At first glance this would seem to be a clear violation of the prohibition against making gifts of public funds that is contained in Article XVI, Section 6 of the California Constitution, which prohibits “the making of any gift, of any public money or thing of value to any individual, municipal or other corporation whatever…”  This prohibition applies to all units of government in California, including school districts.

However, the courts have determined that a “thing of value” may be provided to an individual if the private benefit is incidental to a public purpose.  As far back as 1940, the court, in County of Alameda v. Janssen, cited several prior court cases in stating that, “It is well settled that, in determining whether an appropriation of public funds or property is to be considered a gift, the primary question is whether the funds are to be used for a ‘public’ or a ‘private’ purpose.  If they are for a ‘public purpose,’ they are not a gift within the meaning of section 31 of article IV [now Section 6 of Article XVI].”  Presumably, home charter schools have determined that the private benefit (in the form of personal entertainment) of a ride through Pirates of the Caribbean is incidental to the value the public receives from it.  Sure, that may sound ridiculous, but how else would it be legal?

The same court also ruled that, “The determination of what constitutes a public purpose is primarily a matter for legislative discretion.”  The term “legislative” refers to the action of the legislative body of any government entity (including school district governing boards) and not just the state legislature.  In other words, the courts have granted substantial discretion to legislative bodies (including school district governing boards) in determining whether a private benefit is incidental to a public purpose.

The California Education Code is silent on this issue.  The California Department of Education, however, specifically prohibits the use of public funds for the cost of admission for students or staff to amusement/theme parks or other similar social events, but that prohibition applies only to the California Partnership Academies.

The ruling that “the determination of what constitutes a public purpose is primarily a matter for legislative discretion” has, in every case, been made with reference to a publicly-elected legislative body.  Charter school governing boards, by contrast, are non-elected and self-appointed private bodies with jurisdiction over public funds.  The question of whether this same deference should be granted to charter school governing bodies, therefore, may be ripe for a court challenge.  

But legal action would not be necessary if the Legislature exercised its authority to prohibit—or at least reduce—the inappropriate use of public funds by home charter schools.  The courts have made it clear that the Legislature has the authority to strictly prohibit the use of public funds for specific out-of-school activities, such as admission to amusement parks, etc.  

Alternatively, if the Legislature prefers to allow for a case-by-case approach, it could require the governing body of a charter school’s authorizer to approve each such expenditure at a public meeting, as a non-consent action item.  At least this would bring a publicly-elected legislative body back into the decision-making process and provide for public scrutiny.  In any event, the longer these questionable expenditures are allowed to continue and as the number of families benefiting from this public largesse continues, the harder it will be politically to stop or curtail this practice. 

Is this Ironic or What?

There’s a movement afoot in Tennessee to establish stronger central control over its charter schools

In 2012, Tennessee decided that the best way to improve persistently low performing schools was to turn them over to charter school operators.  To this end, the state created the Achievement School District (ASD) and assigned the low performing schools to it.  The charter schools in the ASD are under the control of 11 charter management organizations.  The ASD has a superintendent, but this is an oversite position, with very little control over the operation of the schools.  

Predictably, the ASD schools have not performed any better than the other low performing schools in the state.  Although the state has the authority to replace a CMO whose schools are not improving, that has not yet happened.

To address these issues, the current ASD superintendent, Sharon Griffin (she’s been in the job for about a year), wants more centralized control over the schools. (Griffin also serves as Assistant Commissioner for the Tennessee Department of Education’s Office of School Turnaround.)  As explained by Chalkbeat  here, that is ruffling some feathers.  The charter school operators complain that they were given the autonomy to run the schools as they see fit, and they want to keep it that way.  But that’s not working, according to Griffin, who came to the ASD from Memphis, where “her no-nonsense leadership was credited with turning around other schools.”  She hinted that she may be looking at assuming the authority over “hiring practices, instructional practices, culture, and climate” if the schools are not “yielding the results we desire for our children.”   Griffin has the support of Penny Schwinn, Tennessee’s Education Commissioner (and Broad Academy Graduate) in this effort.

So, there you have it.  Due to their poor academic performance, Tennessee is on the verge of establishing centralized control over charter schools, which have championed local (meaning school site) control and decision making as the path to improvement.

The Chalkbeat story is a good read.

Reed Hastings Wades into Missouri Politics

“Those are my principles. And if you don’t like them, I have others”—Groucho Marx

California readers will be interested in this item, which Diane Ravitch posted on her blog today.  It discusses the $143,000 in political contributions that Reed Hastings gave to 74 Missouri Republicans, including 73 Republicans who voted for a controversial abortion ban bill and the Republican Governor Mike Parson, who signed the bill last month. As explained here, Hastings said that his contributions were specifically in support of a bill, HR 581, which would have expanded the jurisdictions in which a charter school could be established (charter schools in Missouri are currently limited to specific cities and districts).  The bill drew bi-partisan opposition and, after a filibuster, was dropped from the calendar without a vote.

But I have to wonder—why would Reed Hastings, a self-described progressive Democrat, be so interested in charter school expansion in another state that he maxes out on political contributions to anti-choice Republicans?  Especially on behalf of a bill that had bi-partisan opposition and was unlikely to pass.  The mystery deepens when we consider that the best thing that can be said about charter schools in Missouri is that they have been less than stellar.  According to thiseditorialin The Missourian, the expansion of charter schools “should be rejected unless legislators agree to improve accountability and oversight of charter schools — and unless lawmakers allocate additional funds for public schools.”  So how is it that support for an unpopular bill to allow the expansion of mediocre charter schools in another state can trump one’s core progressive principles?

Well, as CEO of Netflix, Hastings has other principles.  While he has made personal contributions to Missouri Republicans, it turns out that his company, Netflix, has also recently hired a lobbying firm to work in Missouri, as reported here.  Perhaps not-so-coincidentally, legislation in that state to extend video service provider fees to non-cable companies like Netflix died without a hearing.  

Do Charter Schools Have a Negative Financial Impact on Their Host Districts?

If you ask this of almost any district that hosts a charter school, the answer would be “yes.”  A report by MGT of America, which was commissioned by the United Teachers of Los Angeles, found that charter schools cost LAUSD almost $600 million annually.  To be fair, that report is disputed by the California Charter Schools Association here.

But if you ask Paul Bruno, who authored a recent PACE report, “Charter Competition and District Finances: Evidence from California,” the answer is “no.”  He writes that the fiscal stresses resulting from the presence of charter schools in California is less than that found in other states, because “California’s policy context shields districts to a large degree from fiscal strain.”  Many districts in California would be surprised to hear this.

This conclusion is based on an analysis of the impact of charter schools on per student spending (measured by spending per unit of average daily attendance, or ADA) in what is presumably, but not specifically stated, the host district.  That impact is measured by using a regression equation, in which per student expenditures is the dependent variable and one of the independent variables is charter school enrollment as a percentage of district enrollment.  The data are collected “at least once between the 2003-2004 and 2014-2015 fiscal years.”  

The author deserves credit for a technically sound analysis.  But that technical virtuosity is misapplied.  Production functions and other forms of mathematical modeling of relationships are only as good as their underlying assumptions.  In this case, the assumption is that the financial impact of charter schools on their host districts would be manifested in changes in per student spending.  In fact, that is the hypothesis of this paper.  But that hypothesis doesn’t make sense in the context of California’s school finance system.

In California, school districts and charter schools are funded on the basis of a base grant for all students (which is the same amount for all students within a grade span), and the base grant is increased by a supplemental grant and concentration factor, which are awarded on the basis of a district’s or charter school’s enrollment of students who are English learners, low income, or in foster care. In short, California’s districts and charter schools receive a specific amount of revenue per student, and they must live within that amount.  

The funding that each district receives per student is totally unrelated to the number of charter schools in its boundaries or the size of the enrollment in those schools.  Since spending per student is directly related to funding per student, and because the presence of charter schools has no impact on funding per student, there is no reason to believe that the presence of charter schools has any effect on per student spending.  The report uses sophisticated analytical tools to confirm what we already know—per student spending is driven by per student funding.  Period.

So where should we look for a fiscal impact?  Here’s a thought experiment:  consider a district with 100 students, which is funded at the rate of $10,000 per student. That school receives a total of $1 million in revenue and, after setting aside 3% for a reserve, spends a total of $970,000 (or $9,700 per student).  Now let’s say one of those students transfers to a charter school.  The district loses $10,000 in revenue, but its costs are reduced by only a fraction of that revenue loss.  Let’s be generous and say its costs go down $3,000.   Now the district is spending a total of $967,000, or $9,768 per student.  That looks like a $68 per student windfall, right?  But wait, don’t forget that the district also has lost $10,000 in revenue, which is $101 per the remaining students.  When a district loses students, the marginal savings are smaller than the reduction in funding. In this case, the district is $33 per student in the hole.

A district must adjust to this by deficit spending, reducing spending, or some of both.  It is in these areas that the financial consequences of the presence of charter schools will be felt.  We should be asking:  What are the consequences of deficit spending?  How long is it sustainable?  Where is sending reduced?  Which programs or services are affected?  What is the impact on students?  This report does not raise these questions.

Through deficit spending and/or budget cuts, the district will eventually restore its per student spending to what it otherwise would have been in order to live within its per student revenue.  The presence or absence of a charter school has no effect on this dynamic.

The report acknowledges that “per-pupil aggregate fund balances at the end of the fiscal year are significantly negatively associated with local charter school enrollment shares. This suggests that even when California districts are able to reduce their expenditures in the face of charter school competition this may be insufficient to completely offset contemporaneous declines in revenue.”  

My point exactly!  But it gets tossed aside in the next paragraph, in which the author states, “I do not find evidence that districts facing charter competition are financing their expenditures with debt….This suggests that any strain indicated by districts’ declining fund balances is not so severe as to force districts to take on additional debt.”  In truth, this suggests nothing more than that districts are obeying the legal prohibition against borrowing to pay for current expenses of education.  (Districts engage in short-term borrowing in order to manage cash flow, but such loans are repayable within the fiscal year and do not constitute deficit financing of current expenses.)

The author also argues that the 3% fee that districts may charge charter schools “may reduce, or even reverse, the fiscal strain districts might otherwise experience from charter school expansion.”  Far from being a money-maker, most districts report that the costs of oversight (especially the level of oversight that the public deserves) exceeds the revenue from the 3% fee.  This possibility is implicitly dismissed without comment.

Finally, the author refers to charter schools that are “affiliated” with a school district.  This term is not defined.  It could mean either the district in which the charter school is located (let’s call it the host district) or the authorizing district, but the explanation is less that clear:  “I treat any charter school not formally affiliated with a TPS district as though it is affiliated with the nearest TPS district.”  Maybe this refers to charter school that is “affiliated” with (authorized by?) a non-district entity, such as a county office of education, and the “nearest” district is the one in which the charter school is located, but this is not clear.

The distinction between a host district and an authorizing district is considered unimportant by the author, because “this reflects technicalities of oversight and administration, not differences of enrollment policy.”  This is not true, because, for the time period during which data for this report were collected, the host district and authorizing district were not necessarily the same.  Many districts authorized charter schools that were located outside of their own boundaries.  This allowed them to collect the oversight fee (and perform little actual oversight) while the host district experienced the effects of declining enrollment.  

The U.S. News “Best High Schools” ranking for charter schools continues to fall into the survivorship bias trap

This is a quick update to my post of April 25.  Just yesterday the U. S. News released its newest list of the best high schools in the United States.  The list includes a separate ranking for charter schools, which are ranked on the basis of several indicators.

The indicator with the highest weight (30%), is “College Readiness,” which is the proportion of 12thgraders who took and passed at least one AP or IB exam.  This indicator applies only to those students who enrolled in 9th grade or later AND were retained through 12thgrade.  These are the “survivors.”  All students who left the school prior to the administration of these exams in 12thgrade–and there were many–are excluded from the calculation.  You won’t find a purer example of survivorship bias.

Charter schools are also ranked on “Math and Reading Proficiency” and “Math and Reading Performance,” which are both weighted at 20%.  Proficiency is the aggregated scores on state math and reading assessments that students may be required to pass for graduation.  This does not apply to California, because state graduation exams are not required.  It is not known how the 20% weight that would have been applied to this criterion is redistributed among the others.

Math and Reading Performance is how scores on state assessments compare to U.S. News’ expectations, given the student demographics of the school.  (They apparently use some kind of “value added” calculation, but I don’t have the space to get into that here.) In California, these assessments are administered late in 11thgrade, so would measure only those students who did not transfer out before the time of administration.  More survivorship bias.

The graduation rate is weighted at 10% and is defined as the proportion of entering 9thgraders who graduated four years later in 2018.  This is a curious way of putting it, because it implies that U. S. News tracks ALL 9thgraders who enroll in a charter school, including those who leave the school prior to 12thgrade.  They don’t.  That would require them to track outcomes for every student who transfers out of a charter school.  Actually, their numbers reflect the graduation rate for only the survivors.

The four-year cohort graduation rate for all California public schools in 2018 was 83%, and all of the graduation rates of the charter schools ranked reported by U. S. News exceed that rate.  At the same time, the statewide retention rate for the same cohort was 99%, and NONE of the ranked charter schools exceeded that rate.  The school with the highest graduation rate, at 100%, is Stockton Unified Early College Academy, which also has the third lowest retention rate:  77%.  Over all, the retention rates ranged from 58% (Stockton Collegiate International Academy) to 98% (Leadership Public Schools in Hayward), but only four exceeded 90% while six were below 80%. The charter school that was ranked #1 in California and #12 nationally—Preuss School USD—had a retention rate of only 82% and a graduation rate of 95%.


How the University Arkansas Measures Charter School Effectiveness and Return on Investment

(You’ve got to read it to believe it)

One problem with easy access to lots of data and a computer is that—in the wrong hands—it can result in some pretty ridiculous “research.”  Especially when the research is more for the purpose of promoting an agenda than advancing knowledge.  Case in point: A Good Investment: The Updated Productivity of Public charter Schools in Eight U. S. Cities, recently released by the Walton-funded School Choice Demonstration Project at the University of Arkansas.  The authors are Corey DeAngelis, an education policy analyst at the Cato Institute Center for Education Freedom; Patrick Wolf, a professor of education policy and 21stCentury Endowed Chair in School Choice at the University of Arkansas (endowed by Walton); Larry Maloney, president of Aspire Consulting (information technology); and Jay May, a senior consultant for EduAnalytics, LLC (they provide data based solutions to what ails K-12 education).  The eight cities reviewed are Atlanta, Boston, Denver, Houston, Indianapolis, New York City, San Antonio, and Washington, D.C.

This paper has been reported in newspapers around the country like the New York Postand the Washington Examiner.  The tone of the reporting is exemplified by the headline in the Anchorage Daily Planet: “Case Closed:  Charter Schools Deliver More Education.”  As if, in social science research, the case is ever closed. Never mind that the report examines only two subject areas (reading and math) in one grade (8th) and uses highly suspect methodology at that.    

The report compares charter schools in each city’s metropolitan area (not just the city districts) with the traditional public schools (TPS) in the same area on the basis of two factors:  cost-effectiveness and return on investment (ROI).    The authors conclude that, “On average, for the students in our cities, public charter schools are 40 percent more cost-effective and produce a 53 percent larger ROI than TPS.”  This is a pretty startling finding, but does it hold up under even casual scrutiny?  The answer is “no” according to Peter Green, the blogger at Curmudgucation.org.  He has posted an excellent critiqueof this report, but there are some problems he doesn’t get into that further discredit it. 

To begin with, the estimates of cost-effectiveness and ROI both depend on the definition of cost effectiveness the authors use, which is simply the average standardized test scores divided by average revenue per student:

The numerator and denominator in this formula are both beset with problems that relegate the resulting estimates of cost-effectiveness and ROI to the category of junk science.  First, the numerator consists of only 8thgrade reading and math NAEP scores.  Peter Green rightly questions whether standardized test scores in only two academic areas in only one grade really tell us everything we need to know about school quality for an entire school or district.   But let’s play devil’s advocate and say they do.  Even with this stipulation, the scores still have to provide a valid point of comparison between the two segments.  In other words, either the demographics of the test taking populations in the two segments must be identical, or the test scores must be adjusted to account for the differences.  The authors of this report do not adjust the scores for demographic differences, and they apparently believe that the two populations are identical for comparison purposes.  I say “apparently,” because they don’t raise the issue and make no effort to convince the reader that they are.  There are solid reasons for assuming they aren’t.

First, to be demographically identical, the student subgroups within each population must be proportionately the same.  We know, however, that charter schools typically enroll fewer students with disabilities (SD) and English learners (EL), which are historically low scoring subgroups.  Enrolling proportionately fewer of these students would result in higher scores for the charter schools as compared to TPS.  The authors do acknowledge that charter schools enroll fewer EL and SD students, but say “those enrollment gaps failed to explain the revenuedifferences between the public school sectors in every city except Boston (emphasis added).”  Huh?  What about test scoredifferences?  No mention of that.

But even if they did enroll the same proportion of SD and EL students, this still is no guarantee that these students in the charter schools actually took the NAEP.  Here’s what NAEP itself has to say on the subject:

Some SD and ELL students may be able to participate in NAEP, with or without accommodations. Others are excluded from NAEP assessments because they cannot participate with allowable accommodations. The percentage of SD and ELL students who are excluded from the NAEP assessments varies both from one jurisdiction to another and within a jurisdiction over time (emphasis added).  

Charter schools are jurisdictions that are independent of the districts in which they are located. They can make their own decisions about testing accommodations (if any) and which SD and EL students they exclude from the test.  It’s a stretch to believe that the decisions they make are identical to the decisions their host districts make.  And because charter schools like to point to standardized test scores as evidence of their superiority, they have an incentive to exclude low-scoring students from the NAEP.  For these reasons, we cannot accept as an article of faith—as the authors would seem to have us do—that the demographics of either the school enrollment or the test taking populations are identical or even close enough to make a valid comparison in each of the either cities.

The denominator used in this study—total revenue per student—has equally serious problems.  First of all, it uses total revenue, not just revenue applied to 8thgrade reading and math instruction.  So, it compares the total revenue of a charter school containing an 8thgrade (usually a middle school, but sometimes a K-8 elementary school) with that of city districts, which are K-12.  Since grade 9-12 secondary schools receive a higher level of per student funding than the lower grades, this alone inflates the TPS denominator relative to the charter schools.

But there’s more.  The revenue for the TPS includes funding for preschool and adult education!  The authors do this with a straight face and make no effort to try to explain how preschool and adult education revenue can possibly be related to 8thgrade test scores.  Instead, they literally state that, with this methodology, “We are able to connect funding to student outcomes,” completely ignoring the disconnect between the funding they include and the outcomes they purport to measure.

Even within a 6-8 or 7-8 middle school, different schools will spend different amounts on 8thgrade math and reading, depending on different needs and priorities.  The only valid denominator for the author’s purposes is revenue actually spent on 8thgrade math and reading instruction.  Instead of attempting to do this, the author’s go to the opposite extreme of including funding for everything, including the kitchen sink.

I don’t know what’s worse: what this report says about what passes for educational research these days or about the gullibility of too many newspapers and their education journalists. 

True Facts; False Narrative

The Effect of Survivorship Bias on the Calculation of Charter School Graduation Rates

Charter schools market themselves as being superior to traditional public schools largely on the basis of student performance indicators such as standardized test scores and high school graduation rates.  With respect to graduation rates, they usually offer a simple comparison of their rates with something like the statewide average rate for all public schools.  You can see why.  Charter school graduation rates are nearly always higher than the rates for other schools. According to the California Department of Education (CDE), the statewide graduation rate for all public schools in 2016-17 was 82.7 percent.  Nearly all charter schools I reviewed exceeded that rate.

However, graduation rates don’t tell the whole story, as I learned from an examination of charter schools that promote themselves as being “college prep” and/or have been identified as being high performing, in large part on the basis of their high graduation rates.  As it turns out, the calculation of charter school graduation rates suffers from “survivorship bias,” which makes them unreliable indicators of school performance.

In his fascinating book, How Not to Be Wrong:  The Power of Mathematical Thinking, Jordan Ellenberg provides an illustration of survivorship bias at work.  He recounts the story of Abraham Wald, a mathematician at the Statistical Research Group (basically a military think tank) during World War II.  At that time, the U. S. military was confronted with the problem of how much protective armor to put on fighter planes and what parts of the plane needed the most protection.  More armor made the plane less vulnerable to enemy fire, but also made the plane heavier, less fuel efficient, and less maneuverable.  It increased defensive capability at the expense of offensive capability.  On the other hand, less armor allowed a plane to be lighter and more maneuverable—thus increasing offensive capability—but made it more vulnerable to enemy fire. The challenge was to find the amount of protective armor that provided the optimal balance between a plane’s offensive and defensive capabilities and determine where on the plane the armor should be placed.  This problem was assigned to Abraham Wald to solve.  

The military provided Wald with data showing where planes had sustained enemy fire.  When planes returned from combat, they were covered with bullet holes, which, on average were distributed on planes as follows:

               Section of the Plane                    Bullet Holes per Square Inch

                Engine                                               1.11

                Fuselage                                           1.73

                 Fuel system                                      1.55

                 Rest of the plane                             1.80

Using these data, the military asked Wald to figure out the optimal distribution of armor, assuming that the least amount of armor would cover the part of the plane that appears to sustain the least damage—the engine. Wald came back with the opposite conclusion—the engine should receive the most protection.  This is because the data showing the distribution of bullet holes was based on planes that returned to base and did not include the planes that were shot down and did not make it back.  Assuming that all parts of the plane were equally likely to be struck by bullets, Wald wondered why the distribution of bullet holes on the returning planes were unevenly distributed.  His realization was that the returning planes were not representative of all of the planes that left the field that day, because they did not include the planes that did not return.  Those planes must have had bullet holes, too, but where were they?  As Ellenberg puts it:

Wald’s insight was to simply ask:  where are the missing holes?  The ones that would have been all over the engine casing, if the damage had been spread equally all over the plane?  Wald was pretty sure he knew.  The missing holes were on the missing planes.  The reason planes were coming back with fewer hits to the engine is that planes that got hit in the engine weren’t coming back.

Wald’s conclusion was that the armor should not go where the holes were on the returning planes, but where they were not.  Basing an analysis on only the planes that returned to the base and not all of the planes that left the base is an example of survivorship bias.  

The calculation of charter school graduation rates provides another example.  The 12thgraders in a charter school are the “survivors” from the total population of 9thgraders that were enrolled in the school four years before.  The greater the gap between 12thgrade enrollment and 9thgrade enrollment four years before, the greater the effect of survivorship bias, and the less valid it is to use 12thgrade performance as an indicator of a school’s effectiveness.  And yet charter schools do exactly that.

Data from the California Department of Education (CDE) show that the number of students that charter schools graduate from 12thgrade is far below the number of 9thgraders the same schools enrolled four years earlier.  While normal student mobility may account for some of this, the data show that charter high schools systematically fail to retain and graduate as many 9thgraders as traditional high schools.  The result of this survivorship bias renders charter high school graduation rates all but meaningless as a measure of school quality.

How Graduation Rates are Computed

I use the adjusted 4-year cohort graduation rates for 2016-17 provided by the CDE.  This rate is the percentage of 9thgraders who graduate at the end of 12thgrade in four years.  To account for student mobility, the 9thgrade cohort is adjusted for students who transfer in and out during the four-year period.   So, to calculate the graduation rate for an individual school, the number of students who graduate at the end of 12thgrade is divided by the number of students who were enrolled in the school four years earlier (in 9thgrade) plus students who transferred in, minus students who transferred out at any time during that four-year period.  With this methodology, students who drop out from the charter school are retained in the denominator.  Dropouts cause the graduation rate to be less than 100%. But a student who transfers out of the charter school and then drops out from a different school prior to graduation has no effect on the charter school’s graduation rate, even if they don’t transfer out until grade 12.  Those students count against the graduation rate of the school that they transfer to.

High School Retention Rates

To get a measure of possible survivorship bias I look at the retention rate for this same cohort, which is calculated by dividing the number of students enrolled in 12thgrade in a school by the number of students enrolled in 9thgrade in the same school four years later.  In this case, “retention rate” does not denote the retention of the same students during the four-year period; rather, it denotes the retention of the same number of students.  According to data provided by the CDE, the statewide retention rate for all public schools in 2016-17 was 98.5 percent.  This means that the number of students enrolled in all public schools in 12thgrade in 2016-17 was almost equal to the number of students who were enrolled in 9thgrade four years earlier. The loss of 1.5 percent of 9thgrade enrollment is due mostly to the students who drop out.  Students who transfer from one public school to another public school (including charter schools) do not have an impact of the calculation of the statewide retention rate.  

The Charter Schools in this Analysis

Rather than use a randomly selected sample of charter schools, I focus on charter schools that specifically promote themselves as being college preparatory and/or that have been identified as high performing schools, as least partly on the basis of their graduation rates.  These are schools that point to their graduation rates as evidence of their superiority.  Among the schools in this study, three are KIPP schools, and seven are Aspire schools.  Most of these schools have “college preparatory” or “university preparatory” in their names, signifying their focus on preparing their students for graduation and college admission.  

According to its website, KIPP offers “an excellent college-prep education” that provides “personalize[d] learning based on a student’s needs, skills, and interests.”  Similarly, the Aspire website states that, “our purpose is to prepare our…students for success in college, career, and life” and that Aspire schools have a “clear focus on College for Certain.”

I also looked at four Los Angeles Partnership Schools high schools.  The Partnership is a collaboration between the Los Angeles Unified School District, the City of Los Angeles, and other public and private partners to “transform schools and revolutionize school systems.”  It claims to have raised graduation rates since 2008 from 36 percent to 81 percent.  

Finally, I include charter high schools that have been identified by either the US News & World Report or Great Schools.Org as being among the best public schools in the nation based, in large part, on their graduation rates.

Graduation Rates

Table 1 shows the graduation rates of the selected charter schools.  It shows that all but five of the schools have a graduate rate that exceeds the statewide rate for all schools of 82.7 percent.  And 17 of the 29 schools have graduation rates of 90 percent or higher, including two at 100 percent and one at 99.3 percent.  These are, indeed, impressive figures and—by themselves—suggest a high level of performance.  All things being equal, one would expect the charter school graduation rate to mirror the statewide average, with about half of the schools exceeding the average and half falling below it.  The fact that 82 percent of these charter schools exceed the statewide average graduation is a strong indication that all things are not equal, and that there is a systematic reason for this.  Charter school advocates have taken this “systematic reason” to be a superior education provided by charter schools.  Under this line of reasoning, a charter school education is an intervention that, when applied to a population of students that resembles students statewide, produces superior results.

But we cannot look at graduation rates out of context, and when we look at the broader picture, we are forced to question whether charter school 12thgraders do, in fact, resemble their traditional school counterparts.  Specifically, we must take a deeper look at the four-year cohort from which these graduation rates are computed.  Table 1 shows that, on a statewide basis, the cohort graduating in 2016-17 had a retention rate of 98.5 percent.  In other words, the 12thgrade enrollment for all California public schools in 2016-17 was 98.5 percent of the 9thgrade enrollment four years earlier, in 2013-14.  By contrast, the retention rate for the charter schools in this analysis ranges from a low of 56.5 percent to a high of 97.1 percent.  None exceed the statewide average.  Assuming a normal distribution around the average of 98.5 percent, about half, or 14 of the schools in this analysis would be expected to be higher, and the other half lower.  Instead, all of the charter schools are below the statewide average, and 24 of the 28 are far below the average, at less that 90 percent.  This finding is too consistent across schools to be random.  

The charter schools in this analysis have been selected because of their presumed superiority and their focus on college and university preparation.  They claim to provide instruction that is student centered and equitable.  Aspire Public Schools, for example, claim to be “committed to providing equitable opportunities for our students, families, and teammates.  We use an equity lens to examine our policies, practices, and systems at Aspire to strive for all groupsto increase access and benefit from our work” [emphasis added].  Similarly, KIPP schools “blend small-group instruction and technology in creative ways to personalize learning and keep children encouraged, engaged, and continuously learning” [emphasis added].  The Stockton Collegiate International Secondary School “provides a multi-cultural, student-centered environment.”  One would expect that schools with these welcoming, student-centered environments designed to serve all students and keep them engaged would be better at retaining them—at least as good as the statewide average—especially from among a population of students that affirmatively chose to attend those schools in the first place.  Instead, these schools show a remarkable failure to retain students.

Under the methodology for computing the four-year cohort graduation rate, students who transfer away from a charter school for any reason are excluded from the denominator.  Accordingly, when we combine the fact that charter school retention rates are consistently below the state average with the fact that charter school graduation rates are consistently above the state average, we are forced to that conclude that something systematic is at work. Specifically, the charter school 9thgraders who do not persist in their schools through 12thgrade must be among those least likely to graduate.  Attrition among low performers would naturally increase a school’s graduation rate as well as improve scores on standardized tests. Charter schools may succeed not necessarily by providing a superior education, but by ridding themselves of lower performing students.

California law prohibits charter schools from “limiting enrollment access” of “academically low-achieving pupils.” However, nothing in the law requires charter schools to retain low-achieving pupils after they have been admitted and enrolled, and charter schools are not prohibited from expelling students for academic reasons.  They may be expelled, counseled out, or simply discouraged from persisting at any time. Whatever the reason, or however accomplished, the pairing of a consistently higher-than-average graduation rate with a consistently lower-than-average retention rate leads to the inescapable conclusion that it is the low-performing students that charter schools fail to retain.  Again, this finding is too consistent across schools to be random and strongly suggests that the attrition of low performing students from charter schools is part of the “magic sauce” that makes them appear to be better than traditional schools.

Policy Implications

Policy makers often look at performance indicators such as standardized test scores and graduation rates in isolation and conclude that charter schools provide a superior education.  This leads to the further conclusion that the creation of more charter schools should be encouraged as a way to improve opportunities for more students.  In fact, this would be consistent with the intent of the Legislature, as expressed in the California Education Code, that charter schools “Increase learning opportunities for all pupils…” However, the legislative intent goes on to specify that the increased learning opportunities should be “with special emphasis on expanded learning experiences for pupils who are identified as academically low achieving [emphasis added].”   The evidence presented by this analysis suggests that charter schools, at least at the high school level, actually do the opposite:  they weed out low performing students.  

Because of this culling out process, a high graduation rate is not an indication that a charter school does a superior job of serving all of the students it initially enrolls.  One could reasonably question whether a high graduation rate is a measure of the value that the schools brings to the students or of the value that the remaining students bring to the school.  

The first lesson for policy makers is to recognize the effect of survivorship bias and look beyond performance indicators that exclude all students that charter schools enroll in order to understand better the reason why a particular school or program achieves exceptionally good (or poor) results.  Performance indicators may measure results, but they do not explain how results are achieved.  If good results are achieved by weeding out low performing students, it would be poor public policy to promote the expansion of those practices.

The second implication is the need to get a better understanding of the reasons for the high charter school attrition rates. Without surveying students who transfer out of charter schools, this is not easy to do.  To collect data on charter school students who may have been expelled for academic reasons, the Legislature could fund the dropout report required by Section 48070.6 of the Education Code.  Although the first report was due August 1, 2011, it has never been produced due to lack of funding.  The law also requires that, when data are available, the report include “behavioral data by school and school district, including suspensions and expulsions.”  The Legislature could specify that the suspension and expulsion data include the reasons for and outcomes of those actions.

California law also requires that “If a pupil is expelled or leaves the charter school without graduating or completing the school year for any reason, the charter school shall notify the superintendent of the school district of the pupil’s last known address within 30 days, and shall, upon request, provide that school district with a copy of the cumulative record of the pupil, including report cards or a transcript of grades, and health information.” This requirement could be expanded to report also to the school’s chartering authority, if it is different from the district of residence, and to include in the report the reason(s) for expulsions or transfers out.  In addition, the report should also require verification that the student has actually enrolled in another school.  This would prevent a charter school from erroneously reporting a dropout as a transfer.  The Legislature could make these reports a condition of reauthorization and allow authorizing agencies to take this information into account when considering reauthorization.

SCHOOL2016-17 GRADUATION RATEDIFFERENCE FROM STATE AVERAGE2016-17 RETENTION RATEDIFFERENCE FROM STATE AVERAGE
KIPP SCHOOLS
San Francisco College PrepNo datan/a69.4%-29.4%
King Collegiate High School*93.1%10.4%78.6%-20.2%
San Jose Collegiate*88.0%5.3%77.7%-21.1%
ASPIRE SCHOOLS
Golden State College Prep90.0%7.3%86.4%-12.4%
Lionel Wilson College Prep81.5%-1.2%77.4%-21.4%
E. Palo Alto Phoenix Academy69.4%-13.3%69.8%-29.0%
Benjamin Holt College Prep100%17.3%81.2%-17.6%
Langston Hughes College Academy93.9%11.2%74.3%-24.5%
Vanguard College Prep Academy85.0%2.3%63.9%-34.9%
Ollin Univ. Prep Academy92.5%9.8%91.3%-7.5%
USNEWS TOP SCHOOLS
Stockton Collegiate International AcademyNo datan/a56.5%-42.7%
Univ. Prep. Academy Ctr.93.0%10.3%68.8%-30.0%
Preuss School YCSD95.3%12.6%87.4%-11.4%
Animo Leadership High93.8%11.1%80.6%-18.2%
Summit Prep. Charter High94.4%12.0%64.1%-34.8%
Leadership Public School, Richmond94.1%11.4%94.3%-4.5%
Oakland Charter HighNo datan/a97.1%-1.7%
GREATSCHOOLS.ORG
New West Charter, LA91.2%8.5%84.1%-14.7%
Bright Star, LA89.9%7.2%73.9%-24.9%
High Tech High, San Diego99.3%16.6%87.3%-11.5%
University High, Fresno97.3%14.6%86.2%-12.6%
Stockton Early College Academy100%17.3%77.1%-21.7%
University Prep Academy, San Jose93.0%10.3%96.9%-1.9%
Gateway HS, San Francisco94.3%11.6%85.3%-13.5%
LA PARTNERSHIP SCHOOLS
David Starr Jordan HS66.0%-16.7%87.9%-10.9%
Felicitas & Gonzale Mendez HS90.0.%7.3%75.9%-22.9%
Santee Education Complex81.3%-1.4%62.9%-35.9%
Theodore Roosevelt HS78.4%-4.3%82.6%-16.2%
STATEWIDE (ALL CHARTER AND TRADITIONAL SCHOOLS)82.7%98.5%

*Also on the USNews Top Schools list