The Secrets to Charter School Success in Newark: Comments on the NJ CREDO Report

Today, with much fanfare, we finally got our New Jersey Charter School Report. The unsurprising findings of that report are that charter schools in Newark in particular seem to be providing students with greater average annual achievement gains than those of similar (matched) students attending district schools. Elsewhere around the state charter schools are pretty much average.

Link to report: http://credo.stanford.edu/pdfs/nj_state_report_2012_FINAL11272012.pdf

So then, the big question is, what exactly is behind the apparent success of Newark Charter schools – or at least some of them enough to influence the analysis as a whole – that makes them successful? Further, and perhaps more importantly, is there something about these schools that makes them successful that can be replicated?

The General Model

Allow me to start by pointing out that the CREDO study uses its usual approach  – a reasonable one given data and system constraints, of identifying matched sets of students from feeder schools (or areas) who end up in district schools and in charter schools. CREDO then compares (estimates) the year to year test score gains of students in the charter and district schools.

The CREDO approach, while reasonable, simply can’t sort out which component of student achievement gain is created by “school factors” (such as teacher quality, length of day/year, etc.) and which factors are largely a function of concentrating non-low income, non-ell, non-disabled females in charter schools while concentrating the “others” in district schools.

School Effect = Controllable School Factors + Peer Group & Other Factors

In other words, we simply don’t know what component of the effect has to do with school quality issues that might be replicated and what component has to do with clustering kids together in a more advantaged peer group. Yes, the study controls for the students’ individual characteristics, but no, it cannot sort out whether the clustering of students with more or less advantaged peers affects their outcomes (which it certainly does). Lottery-based studies suffer the same problem, when lotteried in and lotteried out students end up in very different peer contexts. Yes, the sorting mechanism is random, but the placement is not. The peer selection effect may be exacerbated by selective attrition (shedding weaker and/or disruptive students over time). And Newark’s highest flying charter schools certainly have some issues with attrition.

Given my numerous previous posts, I would suggest Figure 1 as the general model of the secrets of Newark Charter School success.

Figure 1. The General Model

Put simply, while resource use – additional time, compensation, etc. – may be part of the puzzle – the scalable part – the strong sorting patterns of students into charter and district schools clearly play some role – a substantial role – and one that constrains our ability to use “chartering” as a broad-based public policy solution.

One Part Segregation

Let’s start by taking a look at the most recent available data on the segregation of students by disability status, free lunch status, gender and language proficiency. Now, the CREDO report is careful to point out that charter school enrollments match the demographics of their feeder schools – and uses this finding as an indication that therefore charter schools aren’t cream-skimming. That’s all well and good…. EXCEPT … that for some (actually many) reason, charter schools themselves end up having far fewer of the lowest income students. See Figure 2.

Figure 2. % Free Lunch

Now, one technical quibble I have with the CREDO report is that it relies on the free/reduced priced lunch indicator to identify economic disadvantage (and then sloppily throughout refers to this as “poverty”). I have shown on numerous previous occasions that Newark charters tend to serve larger shares of the less poor children and smaller shares of the poorer children. So, it is quite likely that the CREDO matched groups of students actually include disproportionate shares of “reduced lunch” children for charters and “free lunch” children sorted into district schools. This is a non-trivial difference! [gaps between free lunch and reduced lunch students tend to be comparable to gaps between reduced lunch and non-qualified students.]

Here are the other sorting issues:

Figure 3. % ELL/LEP

Figure 4. % Female

 

Figure 5 shows that not only do charter schools in Newark tend to serve far fewer children with disabilities, they especially serve few or no students with more severe disabilities. In fact, they serve mainly students with Specific Learning Disabilities and Speech Language Impairment. Given the data in Table 5, it is actually quite humorous – if not strangely disturbing – that the CREDO study attempted to parse the relative effectiveness of district and charter schools at producing outcomes for children with disabilities using only a single broad classification [Student matching was based on a single classification, creating the possibility that children with speech language impairment in charters were being compared with children with mental retardation and autistic children in district schools. It is likely that most students who took the assessments were those with less severe disabilities in both cases.].

Figure 5. Special Education Distributions

Here are some related findings from (and links to) previous posts

Newark Charter Effects on NPS School Enrollments

New Jersey Charter School Special Education

Newark Charter School Attrition Rates

Here are just a few visuals of how the free lunch shares and female student test-taker shares relate to general education proficiency rates on 8^th grade math. Both are relatively strong determinants of cross-school proficiency. And both with respect to gender balance and free lunch balance, Newark Charter schools are substantively different from their district school counterparts.

Figure 6: 8^th Grade Math & % Free Lunch

Figure 7: 8^th Grade Math & % Female

 

Now, these are performance level differences, which are not the same as the gain measures estimated in the CREDO study. But, I’ve chosen the 8^th grade scores because that is when the charter scores tend to pull away from the district school scores (that is, these are the score levels at the tail end of achieving greater gains). But, the contexts of the gains for charter students are so substantially different from the contexts of achievement gains for district school students that scalability is highly questionable.

As I’ve said before – There just aren’t enough non-disabled, non-poor, fluent English speaking females in Newark to fully replicate district-wide the successes of the city’s highest flying charters.

One Part Compensation

Now, I’ve also written many posts which address the resource advantages and some resource allocation issues for high flying New York City charter schools, which a) also promote substantial student population segregation and b) have been shown in numerous studies to yield positive achievement gains.

I do not intend to imply by my above critique that peer group effect is necessarily the ONLY effect driving Newark Charter’s supposed success. The problem is that because high flying Newark Charters in particular serve such uncommon student populations we can never really sort out the peer group versus school quality effects.

It is certainly reasonable to assume that the additional time and effort spent with these students in some schools – even though they are a more advantaged (less disadvantaged) group – makes a difference.  No excuses charters in Newark like those in New York City tend to provide longer school days and longer school years, and importantly, they compensate their teachers for the additional time & effort. Here’s a simple chart of the average teacher compensation for early career teachers in NPS and Newark Charters. NPS teachers catch back up in later years, but as I’ve pointed out in numerous previous posts, a handful of Newark charters have adopted the reasonable (smart) competitive strategy of leveraging higher salaries and salary growth at the front end to improve teacher retention and recruitment.

Figure 9: Newark Teacher Compensation

Below is a more precise comparison that teases out the differences that aren’t so apparent in Figure 9. For Figure 10, I have used 3 years of data on teachers to estimate a regression model of teacher salaries as a function of experience, degree level and data year.

Some of Newark’s “high flying charters” [North Star, Gray, TEAM] tend to substantially outpace salaries of NPS teachers over the first ten years of a teacher career. Few of these schools have any teachers with more years of experience than 10. Other Newark charter schools maintain at least relatively competitive salaries with NPS.

Now, a critical point here is that as I’ve shown above, teaching in many of these schools comes with the perk of working with a much more advantaged student population. As such,  it is conceivable that even a comparable wage provides recruitment advantage – given the student population difference. Clearly, a higher wage provides a significant recruitment advantage – though in the case of the highest paying school(s), the elevated salary comes with substantial additional obligations.

Figure 10. Modeled Teacher Salary Variation by Experience

Closing Thoughts

So, when all is said and done, this new “charter school” report like many that have come before it leaves us sadly unfulfilled, at least with respect to its potential to provide important policy insights. Most cynically, one might argue the main finding of the report is simply that cream-skimming works – generates a solid peer effect that provides important academic advantages to a few – and serving a few is better than serving none at all (assuming the latter is really the alternative?). Keep it up!  Don’t worry ’bout the rest of those kids who get shuffled off into district schools. Quite honestly, given the huge, persistent differences in student populations between high flying Newark charters and districts schools, and given the relatively consistency of research on peer group effects, it would be shocking if the CREDO report had not found that Newark charters outperform district schools.

While it is likely that there exists some strategies employed by some charters (as well as some strategies employed by some district schools) that are working quite well – THE CREDO REPORT PROVIDES ABSOLUTELY NO INSIGHTS IN THIS REGARD.  It’s a classic “charter v. district” comparison – where it is assumed that “chartering” represents one set of educational/programmatic strategies and “districting” represents another – when in fact, neither is true (see the scatter of dots in my plots above to see the variations in each group!).