Note: This is not a “Study.” This is just a summary of NJDOE Report Card Data, which can be found here:http://education.state.nj.us/rc/
I made a few more graphs for fun today, pursuing the question of whether the shares of children scoring only “partially proficient” over time are changing in New Jersey Charter Schools at any different rate from the shares scoring partially proficient in traditional public school districts. Note, of course, that partially proficient is a nice way of saying – failed the test. Each graph below includes only General Education students, as I have previously shown that NJ Charters serve very few if any special education students and including these students substantially changes performance levels for non-Charters. Again, it is most relevant to compare visually the Charter schools to schools in district factor groups A and B, because Charter schools do tend to serve children from these districts – primarily A. But, as the graphs show, Charters have continued to perform similarly to schools in DFG A (less well in Grade 4).
"R" indicates Charter Schools"R" indicates Charter"R" Indicates Charter Schools
About a week ago, I commented that charter school average performance was not much, if any different from the average performance of the poorest urban public schools. This is admittedly an oversimplified comparison, but not one I would have made had I believed it to be deceptive, which it is not – given the available data on New Jersey schools.
Here, I will walk through a more complicated though still imperfect analysis of elementary school performance in host districts and in charter schools based on data from 2004 to 2006 (data I had already compiled for related work). First, let’s begin with some descriptive characteristics of the charter schools and schools of similar grade level (elementary in this case) in their host districts based largely on school reports data from those years.
The table below shows that the data set includes 28 charter schools per year and 173 host district schools of same grade level. The charters serve about 1,000 tested students and the host district schools about 11,000 tested students. While the free/reduced lunch share is roughly the same between the two, the free lunch share is higher in the host district schools (these are the poorer students). These differences vary by host district and charters. Newark Charters, for example, are on average (though not all) relatively high poverty.
Note that the average free lunch share in DFG A schools, used in my previous comparison, is 63% (much higher than charters or their hosts on average).
Also higher in the host district schools are the share of children who are LEP/ELL and who are classified as having disabilities. But, the host district schools do have higher total certified salaries per pupil (compiled from state database on personnel salaries).
Three year average scale scores are also listed, for the 2004 to 2006 period.
But, the big question is what happens when you throw this all into the mix of a statistical model to evaluate whether charters outperform host district schools, controlling for the fact that they have less needy populations, but fewer resources to work with? Again, this is a simple school level model, which does not account for individual children’s relative gains in charters (treatment effect) compared to otherwise similar children not in charters but in host district schools. It would be wonderful to be able to conduct such analyses in NJ.
This school level model includes a dummy variable for each district that is a host district, such that charter performance in the model is measured against performance of the host district of that charter. The model includes only host districts and their respective charters. The overall charter effect is essentially the average of differences between charters and hosts, across hosts (and their respective charters).
What we see in this model is that charters, on average, are no different from their hosts on the combined math and language scale scores for NJASK from 2004 to 2006. While the statewide model of the same data shows a strong effect of cumulative salaries per pupil on outcomes, the model within host districts of charters does not – an interesting point to explore. But, other factors play out quite logically – with each student need factor statistically significantly depressing scale scores.
So, what does this more complicated, but still not complicated enough analysis tell us? It tells us that average charter school performance from 2004 to 2006 on elementary assessments is no different from that of average performance in other poor urban schools – specifically the host districts of those charters. It just says this in a more complicated way. Sometimes simple averages – when not deceptive – can be sufficient.
One factor that could turn the findings in favor of charters (as treatment effect) would be if the average starting performance level of charter students, compared to otherwise similar host school students, is lower than that of host school students – which could occur if there is a tendency for parents to look to charters when their children are under-performing. This appears to be the case in the Missouri data in the CREDO study noted below. But, this is unlikely to create a substantial effect.
Again, this is just playing with the numbers, albeit a more rigorous play than my previous posts – leading to the same conclusions.
For more thorough discussions of charter school research, see:
Check out specifically, the original NYC Hoxby study, and critique of it, and the CREDO 16 state study and RAND 8 state study. Exercise caution in linking any specific findings to the New Jersey context.
Getting back to the original point of my blog, this post is simply about introducing to the public discourse some actual data on NJ charter school spending. Back when I wrote my textbook on school finance, I found that DC charter schools were having to rely on private contributions to the tune of 14% of their annual operating expenses. One can obtain such information from IRS non-profit tax filings (IRS 990). I did a quick run of New Jersey Charter School IRS 990 filings for 2008, reflecting revenues and expenditures for 2007. I simply combined their tax filing information with their total expenditure information – which does include expenses for facilities.
What is most striking but not surprising is the degree of disparity among charter schools, driven substantially by differences in private fund raising. Also important to note is that many of these schools spend well over the assumed $11,000 to $12,000 per pupil constantly spun by the media these days. I’ve not yet aligned the performance data with these new financial data, as I need to return to my actual research agenda (this particular analysis is a part of ongoing research).
You might say, hey, if the public only has to subsidize $11k to $12k and private contributors pick up the rest, it’s still a bargain for taxpayers, right? Perhaps – but the necessity to rely on $3k to $5k of private contributions for each charter child educated then seriously limits the potential expansion of charter schools.
Note from my previous posts and work on private schools, I have also shown that private independent day schools spend well above the average public expenditure. New Jersey private independent schools spent in 2007, an average of $25k to $30k per pupil (day schools only) with some exceeding $30k, also based on IRS 990 data.
My previous research on staffing in charter schools (based on undergraduate college selectivity of teachers) has shown that charters in some states attempt to staff their schools in ways similar to elite private academies – the private independent schools.
There is at least anecdotal evidence that some New Jersey Charter schools wish also to emulate elite private schools. For example, Ethical Community Charter School is founded by individuals previously associated with the Ethical Culture Schools of New York City, including the Fieldston School, a school where I taught for 5 years. An absolutely amazing school, which, by the way, spends well over $30,000 per child per year (even tuition is higher than that). I would argue that it will be quite difficult to emulate the ECFS schools of NYC on a mere $11k to $12k and that substantial private fundraising will be required. But private fundraising shouldn’t be required.
Good schools cost money! Sometimes a lot of money. Good education is expensive, which is not to say that all expensive education is good. My point here is that we are not going to solve our “urban education” problems on the cheap ($11k to $12k per kid), or necessarily any cheaper than what we’re spending currently. Any attempt to do so is likely to cause more harm than good.
[for those hanging on to anecdotal information about private religious school tuition as their basis for assuming good schooling can be done dirt cheap – about $3,500 per kid- please read http://www.epicpolicy.org/files/PB-Baker-PvtFinance.pdf]
Bob Bowdon, of Cartel fame and E-3 make the claim that New Jersey’s poor urban districts are scamming the public and taxpayers by having overstated graduation rates. About half of poor district kids pass the HSPA test, but 85% graduate. Their brilliant solution to this problem, as I’ve noted previously, is to give kids the choice to attend charters – on the argument that charters are less likely to do such scamming? So, here are some fun numbers.
First, the percent proficient or higher on HSPA MATH Assessments by district factor group for 2008:
So, what we have here is that Charters (DFG R) actually had the lowest rate of kids proficient or higher on HSPA (matching my graph on previous posts, but lower here because only math is included). Yep, even lower than the poorest urban publics (DFG A). Yes, this is an average – among general ed test-takers – and averages conceal the highs… but they similarly conceal the lows.
Now, here are graduation rates for the schools by DFG:
Wait one second. How can charters have a 97% graduation rate if only about half of the kids pass HSPA? Where’s the scam here? I thought you said that the differential between HSPA proficiency and graduation rates was supposed to be indicative of a scam? And that charters were the solution to the scam? But where is that differential bigger? Charters are lower on HSPA proficiency by a few points and are 12% higher on graduation rate? Now I’m really confused.
Okay – I’m not trying to pick on charter schools here. You guys are mostly working your butts off for a great cause, and quite honestly I don’t hear these completely absurd arguments coming from the charter leaders and teachers themselves. But the supposed “advocacy” out there on your behalf is deeply problematic. Quite honestly, if someone was out there advertising so poorly for my cause, I’d be a little concerned… or perhaps outraged.
Note to Non-Jersey readers about my casual use of Jersey terminology – DFG. In New Jersey, district factor groups or DFGs are a classification scheme that has been used for decades to characterize socio-economic features of public school districts. DFG A districts are generally poor urban districts, but many NJ poor urban districts are relatively small in total enrollment (a cluster of poor urban neighborhoods segregated from their more affluent neighbors). DFG I and J districts are affluent suburban districts. Charters are labeled “R.”
With little doubt, Robert Treat Academy in Newark is one of those charter schools that is doing well by common outcome measures and likely by even more important measures than state tests. What we know about are the tests. And even if one controls for a variety of factors about student populations, Treat’s test scores are pretty darn good.
Here’s a figure from a model I re-ran the other day (based on older work), using a variety of school, student population and community factors to control for expected differences in student outcomes. Schools above the line are those that outperformed expectations and those below the line fell below expectations. Charters are in red, and again, there are roughly equal numbers of traditional publics above and below the red line and charters above and below the red horizontal line. Treat is one of those above the line.
So the argument goes, Treat is producing these test scores with much less money, and therefore we should be able to do the same, with similarly less money across poor urban settings by emulating the Treat model.
I addressed in a previous post how charter schools receive less through the state aid formula than traditional public districts. Again, this should shift somewhat over time, but charters will remain relatively disadvantaged. Using Robert Treat’s IRS 990 for 2007 expenditures (instead of their NJDOE reporting of their expenditure of public charter funding only), Treat shows expenditures per pupil in 2007 around $12,600. I’m still not sure I’ve captured the full expenditure here, because Treat’s IRS 990s show unusually low levels of private contribution for a successful charter school.
That aside, is the Treat miracle replicable across Newark? Or, is Treat different in substantive ways that can’t be spread throughout the system. Here are a few numbers that raise concern.
First, as I noted on a previous post, Robert Treat’s student body is only 3.8% special education in a district with an average of 18.1%. This is from the special education classification data from NJDOE. In the enrollment files, Treat reports 0%. At 100% additional average expenditure per special education pupil, matching district demographics would raise Treat’s expected spending to $14,868 (1.18 x 12,600 in 2007).
Second, while Robert Treat does show about 62.4% students qualifying for free (130% poverty level) and reduced (185% poverty level) lunch, the free lunch share is about 42.9%. That is, Treat’s free or reduced share is boosted by the share of children who are more well off among the less well off. Note that the model I used above used Free & Reduced shares, not Free alone or the ratio between them.
By contrast, Newark Public Schools in total has 82% free or reduced and 71% free lunch alone.
Treat also reports less than 1% limited English proficient students while Newark City schools report 8.7%.
It’s one thing for me to try to control for these differences in estimating who does and does not “beat” odds, but yet another to take a model that has been successful under certain circumstances and apply it widely under very different circumstances, at the same cost.
It’s all well and good to cite other studies from other cities and states that show that charter schools on average aren’t “cream-skimming,” (where most of those comparisons are based either on student’s initial performance or on free + reduced shares) but the reality in this case is that Treat Academy is producing its current level of outcomes at its current price tag with a substantively different student population – most notably the absence of children with disabilities. Again, they’re doing well, and even in models I’ve run controlling for some of these things, they still stand out and should be applauded for their efforts and results.
But, given the demography of the entire student population of Newark in particular, replicating this model may prove difficult. Adding more schools that serve fewer of the poorest children and few or no children with disabilities may be significantly problematic for those schools which then serve the larger shares of both.
Note: The information below is not a comprehensive research study on the relative effectiveness of New Jersey Charter Schools. Rather, it is a quick summary of average proficiency rates for charters compared to other New Jersey schools by socio-economic strata. Unfortunately, New Jersey Charter schools were not part of two major recent multi-state analyses of charter school effectiveness, which can be found here, along with reviews & critiques of those studies. http://www.epicpolicy.org/think-tank/reviews These studies also found mixed results, with charters in some states slightly outperforming their public school counterparts, in other states performing comparably and in others performing less well. I have put together this post merely to stimulate conversation on how NJ charter schools are doing and perhaps encourage additional more thorough research.
In my original post on NJ Charter School performance, Charter schools appeared to be performing somewhere between performance levels of DFG A and DFG B traditional publics. Here’s one of the graphs to that effect.
Note that the charter line – R – falls between the DFG A (poorest traditional publics) and DFG B lines. But, this analysis includes all tested students. While I expected that children with disabilties were underrepresented in Charter schools, I had no idea just how under represented until I took a look, here: https://schoolfinance101.com/wp-content/uploads/2009/11/charter-special-ed.jpg
For example, Robert Treat Academy has 3.8% and North Star Academy 7.8% children with disabilities in a district that has 18.1% in 2007. These are higher than many, which actually serve 0%.
So, correcting for this problem by looking only at General Education students, the graph above becomes the graph below:
In this graph, the Charter line maps almost precisely with that of the DFG A line. That is, the slightly higher performance in the first graph is almost entirely a function of the fact that NJ Charters simply don’t serve children with disabilities and don’t have them in their test taking pool. My apologies for this apparently glaring omission.
The biggest change to my analysis however is in the relative probability that a student attends a tested grade level where less than 40% of students are proficient or higher. Making the above correction, leads to the finding that a child in a charter school is 35% more likely than a student in a DFG A traditional school to be in a tested grade level where fewer than 40% of general education students scored proficient or higher.
Here’s the logistic regression, weighted for number of test takers in grade level and on test (general education only), based on the 2008 report card data:
DFG A is the baseline comparison group. An odds ratio of greater than 1.0 indicates a greater likelihood of being in a grade level with fewer than 40% proficient or advanced than in a traditional DFG A school. Only charters have a greater likelihood – and much greater – 36% greater. Likelihoods vary dramatically for the different tests and subject areas. Apparently, 6th grade tests have cut scores aligned such that many more students do poorly on them. I don’t think that it’s just that 6th graders get dumb for a year. Newer tests take some tweaking. Note the dip in previous graphs. Note also that in affluent communities (GH through J), there is statistically no chance of being in a low performing grade level.
Please – take your own stab at this. I’ve been running these quickly. My Stata data are here.
New Update: Here’s my last shot at it for now. I’ve got the odds for charters down to about 25% greater chance than DFG A schools of being in a grade level where fewer than 40% were proficient or higher. Unfortunately, poverty rates among test takers were only calculable at the district level (and for charters) not school and charters with these data (must use the enrollment data for whole school for that). Also, NJDOE continues the habit of not identifying specific locations of charters in their coding system by county. I have a bridge file somewhere, constructed by zip code, but for charters through 2006. May revisit. Anyway, here’s the logistic regression:
Ah the perils of goofing around with data too quickly/on the fly. Fun though.
Here’s a quick snapshot of the percent of children classified as having disabilities in Charter schools and in Traditional Public Schools in Essex County. These figures add some context to the spending deficit figures in my previous post. Yes, Charters receive a reduced operating aid subsidy. Charters are most disadvantaged financially by not receiving support for facilities, and having to draw on operating funds for facility leases, or receive substantial private support. But, this piece – special education populations- cuts the other way. Traditional public school districts have about 14% to 18% children with disabilities, which typically run about 90% to 110% above “average” expenditure (to provide typical – not necessarily adequate or great – special education services). For example, if 16% of children qualified as disabled and required additional per pupil expenditure of 100% each, these students would add 16% total cost onto district operating costs – or $1,920 over $12,000 for an average per pupil cost of $13,920. That is – just to provide average/typical special education services – the per pupil cost in a district with 16% special ed would be 16% above the per pupil cost of a district with 0% special ed. In other words, if a district with 16% special ed spends $13,920 and another with 0% special ed spends $12,000, those spending figures are comparable – not vastly different.
Here are the special ed rates among Essex county districts and charters:
Note: A knowledgeable reader has informed me that the “0” value for Greater Newark Charter is actually “missing data,” for that year and has assured me that Greater Newark Charter does indeed enroll children with disabilities. At some point, I may get around to updating these analyses. Other “0” values may also represent missing data. But, very low, actual reported rates likely do not.
If one watches the trailer clips from the Cartel movie on two highly successful New Jersey charter schools, one might be misled to believe that Charter schools are simply uniformly freakin’ awesome. They can do no wrong. They are clearly the answer to all of our problems in urban schooling in New Jersey. Indeed there is some, if not much solid empirical research literature out there which finds favorable results for charter schools and much which finds that charters on average, are pretty much a break even option.
For an exceptional review of charter school research, I would recommend Robert Bifulco and Katrina Bulkley’s chapter on Charter Schools in the Handbook of Research on Education Finance and Policy. Neither of these scholars are charter school naysayers, yet they conclude:
Research to date provides little evidence that the benefits envisioned in the original conceptions of charter schools – organizational and educational innovation, improved student achievement, and enhanced efficiency – have materialized.”
I am also not a Charter school naysayer, having written in my own previous work that leaders of charter schools seem more likely to recruit or select teachers with stronger academic credentials than traditional public schools in the same state. But, I’m also a realist when I look at data on charter schools, their students and their outcomes.
For starters, let’s look at how New Jersey charter schools begin with a public subsidy disadvantage – which may explain some of the mixed results that follow. Current expenditures from NJDOE annual financial reports through 2005, show charters spending less than many districts, organized by factor group (A being generally poor urban districts, through I & J, being relatively affluent suburbs).
In many parts of the country, Charter schools make up for this difference with private fund raising. In fact, most infrastructure costs are covered by such fund raising especially where states fail to provide any facilities support to charter schools. A few years back, I was able to compile the tax returns of the non-profits that support Washington DC charters to show that they received, on average, 14% of their revenue through private contributions. I ran an extract the other day of New Jersey Charter school IRS 990 forms, but few reported their data. Still working on that.
Now, on to the raw outcomes of charter schools in New Jersey based on 2008 assessments. Again, based on the cherry picking in the Cartel movie, one would think that all charters in NJ are kicking butt like North Star Academy. However, prior Charter research and the logic of deregulation lead to more realistic assumptions that – some do well – some not so well – and on average, there may be little difference (if the system, either the “market” or the accountability system, does not shut down those who do not do well). Under less regulation one would simply expect more dispersion. Higher highs perhaps, but also lower lows.
Here’s a quick run down. I begin with the “averages” by grade level and by district factor group. Here’s the % proficient or advanced by DFG, with Charters labeled “R.”
Charters labeled "R"
Charter schools, most though not all of which serve relatively poor student populations, hang right down there, across grade levels with DFG A and B poor schools – especially at both the beginning and end grades. Charters look little different when viewing only those who score advanced and higher.
Okay, so these are the averages which conceal the really fun and interesting variations and drag down the superstars. Here’s the 3rd grade assessment data for two groups of schools – those in District Factor Group A and Charters. Schools are sorted by poverty. DFG A – Poor traditional publics are Blue Cirlces and Charters are hollow red diamonds.
Red Diamonds are Charters, other are DFG A (Poor)Red Diamonds are Charters, Others are DFG A (poor)Red Diamonds are ChartersRed Diamonds are Charters
In each case above, the schools are sorted by poverty along the horizontal axis and by proficiency rates on the vertical axis. In each case above, charters are represented by the red diamonds and traditional public schools including only those schools in the poorest district factor groups are represented as blue circles.
The bottom line is that Charter school performance varies widely and varies as widely as traditional public school performance in poor districts. What we do not know yet, because of lack of data is whether the successful charter schools are, in part, successful due to their ability to raise substantial additional resources for their schools. It may be the case that the unsuccessful charters – those that do much less well than even the worst traditional publics are suffering from lack of resources.
Sadly, rather than address these real, substantive issues, organizations such as NJ E3 and individuals like Bob Bowdon have decided to pitch a load of baseless propaganda on public audiences that deserve better. In fact, by pitching this schlock that all charters can do no harm (just look at these 2 really awesome ones!), ignorant pundits like Bowdon are arguably compromising the market decisions of parents – leading them to believe that all charters must necessarily be better than all traditional publics. Market decisions must be based on good information regarding product quality. In this case, it would appear that at least some pundits are creating a new “market for lemons” (knowingly marketing bad charters) at the expense of parents and children for purely political gain (or to sell movie tickets and build reputation). This sales pitch may encourage parents to continue choosing low performing charters, sustaining those schools and holding down the charter average – making the case for charters harder to argue.
School Finance 101 
