Category: Charter Schools

Truly Uncommon in Newark…

A while back I wrote a post explaining why I felt that while Robert Treat Academy Charter School in Newark is a fine school, it’s hardly a replicable model for large scale reform in Newark, or elsewhere.  I have continued over time to write about the extent to which Newark Charter schools in particular have engaged in a relatively extreme pattern of cream skimming.  The same is true in Jersey City and Hoboken, but not so in Trenton. But, Trenton also offers us fewer examples of those high-flying charters that we are supposed to view as models for the future of NJ education. When I wrote my earlier post on Treat, I somehow completely bypassed North Star Academy, which I would now argue is even that much less scalable than Robert Treat. That’s not to say that North Star Academy is not a highly successful school for the students that it serves… or at least for those who actually stay there over time.  But rather that Star of the North is yet another example of why the “best” New Jersey charter schools provide a very limited path forward for New Jersey urban school reform. Let’s take a look:

So, here’s where North Star fits in my 8th grade performance comparisons of beating the odds, based on the statistical model I explain in previous posts:

In this figure (ab0ve), we see that North Star certainly beats the odds at 8th grade. Now, we can also already see that North Star has a much lower % free lunch than nearly any other school in Newark, limiting scalability right off the bat. There just aren’t enough non-poor kids in Newark to create many more schools with demography like North Star. Not to mention the complete lack of children with disabilities or limited English language proficiency.

Here’s North Star on the map, in context. Smaller lighter circles are lower % free lunch schools. Most of the charters in this map are… well.. smaller lighter circles (with charters identified with a red asterisk). Not all, however, are as non-representative as North Star.

Now, here’s the part that sets North Star and a few others apart – at first in a seemingly good way…

If we take the 2009 assessments for each grade level, one interesting finding is that the charter schools serving lower grade levels in Newark are generally doing less well than the NPS average (red line). But, those schools that start at grade 5 seem to be picking up a population that right away is doing comparable or better than the NPS average. See, for example, TEAM and Greater Newark (comparable to NPS in their first grade – 5th – served) and, of course, North Star whose students perform well above NPS in their first year – likely not fully a North Star effect, but rather at least partly a selection effect (Lottery or not, it’s a different population than those served in the district).  More strikingly, with each increase in grade level, proficiency rates climb dramatically toward 100% by 8th grade. Either they are simply doing an amazing job of bringing these kids to standards over a 3 year period… or … well… something else.

The figure above looks at 6th, 7th, and 8th graders in the same year. That is, they aren’t the same kids over time doing  better and better. But, even if we looked at 6th graders in one year, 7th graders the next year and 8th graders the following year, we wouldn’t necessarily be looking at the same kids. In fact, one really easy way to make cohort test scores rise is to systematically shed – push out – those students who perform less well each year. Sadly, NJDOE does not provide the individual student data necessary for such tracking. But there are a few other ways to explore this possibility.

First, here are the cohort “attrition rates” based on 3 sequential cohorts for Newark Charter schools:

In this figure, we can see that for the 2009 8th graders, North Star began with 122 5th graders and ended with 101 in 8th. The subsequent cohort also began with 122, and ended with 104. These are sizable attrition rates. Robert Treat, on the other hand, maintains cohorts of about 50 students – non-representative cohorts indeed – but without the same degree of attrition as North Star. Now, a school could maintain cohort size even with attrition if that school were to fill vacant slots with newly lotteried-in students. This, however, is risky to the performance status of the school, if performance status is the main selling point.

Here’s what the cohort attrition looks like when tracked with the state assessment data.

Here, I take two 8th grade cohorts and trace them backwards. I focus on General Test Takers only, and use the ASK Math assessment data in this case. Quick note about those data – Scores across all schools tend to drop in 7th grade due to cut-score placement (not because kids get dumber in 7th grade and wise up again in 8th). The top section of the table looks at the failure rates and number of test takers for the 6th grade in 2005-06, 7th in 2006-07 and 8th in 2007-08. Over this time period, North Star drops 38% of its general test takers. And, cuts the already low failure rate from nearly 12% to 0%. Greater Newark also drops over 30% of test takers in the cohort, and reaps significant reductions in failures (partially proficient) in the process.

The bottom half of the table shows the next cohort in sequence. For this cohort, North Star sheds 21% of test takers between grade 6 and 8, and cuts failure rates nearly in half  – starting low to begin with (starting low in the previous grade level, 5th grade, the entry year for the school). Gray and Greater Newark also shed significant numbers of students and Greater Newark in particular sees significant reductions in share of non(uh… partially)proficient students.

My point here is not that these are bad schools, or that they are necessarily engaging in any particular immoral or unethical activity. But rather, that a significant portion of the apparent success of schools like North Star is a) attributable to the demographically different population they serve to begin with and b) attributable to the patterns of student attrition that occur within cohorts over time.

Again, the parent perspective and public policy perspective are entirely different. From a parent (or child) perspective, one is relatively unconcerned whether the positive school effect is function of selectivity of peer group and attrition, so long as there is a positive effect. But, from a public policy perspective, the model is only useful if the majority of positive effects are not due to peer group selectivity and attrition, but rather to the efficacy and transferability of the educational models, programs and strategies. Given the uncommon student populations served by many Newark charters and even more uncommon attrition patterns among some… not to mention the grossly insufficient data… we simply have no way of knowing whether these schools can provide insights for scalable reforms.

As they presently operate, however, many of the standout schools – with North Star as a shining example – do not represent scalable reforms.


Searching for Superguy in Jersey…

A short while back I did a post called Searching for Superguy in Gotham.  In that post, I tackled the assumption that Superguy was easily identifiable as a hero leader of charter schools – or at least that was one distorted portrayal of Superguy in Waiting for Superman. Now, I should point out here that I really don’t know of anyone actually out there running charter schools who wishes to portray him/herself in this way. So, to be absolutely clear, this post is in no way an attack on those who are out there just trying to do the best job they can for kids in need.

This post IS a criticism of the punditry around charter schools- the notion that charter schools are easy to pick out from the crowd of urban (or other) schools- because they are necessarily, plainly and obviously better. That classic argument that the upper half is better than average!

This was the basis of my Searching for Superguy in Gotham activity. In that activity, I estimated a relatively simple statistical model to determine which schools performed better than expected, given their students and location and which schools performed less well than expected, given their students and location. I had been planning all along to do something similar with New Jersey Charter Schools. Now is that time!!!!!

As I did with New York City charter schools, I have estimated a statistical model of the proficiency rates of each charter school and each other school in the same New Jersey city. In the model, I correct for a) free lunch rates, b) homelessness rates, c) student racial composition (Hispanic and black). AND, I compare each test – grade level and subject – to the same test across all schools. AND, I compare each school to other schools in the same city (by using a “city” dummy variable). I obtained all necessary variables from a) NJ school report cards (outcome measures) and b) NJ enrollment data file (free lunch, race, homelessness) and c) NCES Common Core of data for “city” location of school.

So now, the search for Jersey Superguy begins! Let’s start with 4th Grade Math performance in 2009. This scatterplot includes all schools with ASK4 Math scores in cities where charters existed in 2009. Schools above the red horizontal line are schools that “beat the odds.” That is, they are schools that had proficiency rates that were above the expected proficiency rates for that school, given its students, the test, and the location (city). Schools below the red line are schools that did not meet expectations. So, is superman (mythical super charter school leader) hiding in one of those dots way at the top of the scatter? Is he in a high-flying, high poverty school? Is he in a high-flying low poverty school? Certainly, he could not be down in the lower half of the graph.


CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTERS AND WHICH ARE DISTRICT SCHOOLS

CLICK HERE FOR A CLOSE UP ON NEWARK SCHOOLS OVER AND UNDER THE LINE

NOTE: I’m in the process of fixing a data error that occurs on a few charter schools (affecting merging of data).  These figures still include the merge error, but the overall distributions are not affected. Schools affected include: Environment Community School, Liberty Academy, Hope Academy, International CS of Trenton, Jersey City Community CS and Jersey City Golden Door. I HAVE  NOW EXCLUDED MISMATCHED SCHOOLS.

The source of the error is the NJDOE enrollment file, which, for example identifies Environment Community School as both 80_6232_920 (county, district, school) and as 80_6235_900.  The first of these codes is correct. The second is for Liberty Academy CS (according to the School Report Card and according to NCES data).

Now, let’s take a look at the 8th Grade Math outcomes. Here’s the statewide scatterplot:


Surely superguy must be hangin’ out in one of those high flyin’ dots way at the top of the scatter?

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTERS AND WHICH ARE DISTRICT SCHOOLS

CLICK HERE FOR A CLOSE UP ON NEWARK SCHOOLS OVER AND UNDER THE LINE

As you can see, there are plenty of charters and traditional public schools above the line, and below the line. The point here is by no means to bash charters. Rather, this is about being realistic about charters and more importantly realistic about the difficulty of truly overcoming the odds. It’s not easy and any respectable charter school leader or teacher and any respectable traditional public school leader or teacher will likely confirm that. It’s not about superguy. It’s about hard work and sustained support – be it for charters or for traditional public schools.

As I noted in my previous searching for superguy post:

Yeah… I’d like to be a believer. I don’t mean to be that much of a curmudgeon. I’d like to sit and wait for Superguy – perhaps watch a movie while waiting (gee… what to watch?). But I think it would be a really long wait and we might be better off spending this time, effort and our resources investing in the improvement of the quality of the system as a whole. Yeah, we can still give Superguy a chance to show himself (or herself), but let’s not hold our breath, and let’s do our part on behalf of the masses (not just the few) in the meantime.

TECHNICAL APPENDIX

Here is a link to the model used for generating the over/under performing graphs above

And here is a separate model  in which I test whether Charter schools on average outperform traditional public  schools in the same city. This model shows that they don’t, or at least that their 1 to 3 percentage point edge on proficiency is not statistically significant. But whether charters on average outperform – or don’t – traditional public schools is not the point. The point is that like traditional public schools – they vary – and it’s important for us to get a handle on how and why all schools vary in their successes and failures – charter or not.

Complete slide set here: New Charter Figures Nov 12

BONUS MAPS

Here are some updated maps of the demographics and adjusted performance measures of charter and district schools in Newark.

First, % Free Lunch 2009-10:

Next, a new one, % LEP/ELL – note that the % LEP/ELL for NWK charters is so low, therefore their dots are so small that the star indicating “charter” covers them entirely:

Finally, here are the Beating the Odds figures converted into color coded circles – with large purple circles being high performers – better than expectations – medium size pale dots being relatively average performers – and large yellow dots performing below expectations:

Jersey City % LEP/ELL

Jersey City % Free Lunch

Jersey City Performance Index

Biddle me this? (or Flunkout Nation)

While I suspect few people have read or seen this post by RiShawn Biddle of Dropout Nation, I felt that it was worth mentioning because it presents such egregiously flawed logic coupled with flat-out factually incorrect and unsubstantiated claims. Sadly, this is what we have come to all too often in the current education reform debate. And this isn’t really about RiShawn Biddle as an individual or his blog and tweets, but rather about the propensity to argue important and complex issues in such crude terms and with so little knowledge or understanding of context and history.

In a recent post, Biddle argues that the NAACP is and has been heading up a misguided public policy agenda on behalf of black America (my characterization – perhaps not right on target). Biddle argues that Jealous should a) admit that arguments for more funding and more equal funding are wrong and have proven to fail and b) that instead, Jealous should embrace charter schools as a solution.

Now, this is a strange dichotomy to begin with – either fair and adequate funding or charter schools. It seems from a charter advocacy approach that one would also want fair and adequate funding including substantial funding targeted to high need areas. That is, both, not either/or.

That logical point aside, Biddle then goes on to make his bold points to Jealous with the most absurd claims I’ve read, in well, about a week.

Here’s how Biddle explains the failures of legal challenges over school funding:

The NAACP has taken the wrong approach on school reform for far too long. The continuing dropout factory status of Newark, Kansas City, Mo., and other cities that have benefited from funding equity suits is clear evidence that this approach doesn’t spur any kind of reform.

I’ll set aside the equally absurd claims about integration that follow in the next sentence. But, let’s take a quick look at this claim.

First of all, Kansas City, Mo. never really benefited from a case over school funding. Rather, Kansas City, Mo. – this supposed poster child for failed school funding reform – saw a short term boost in funding while under court ordered desegregation. Funding litigation concurrent (1993) with the desegregation litigation had negligible effect on KCMSD funding. Later funding challenges in Missouri were found in favor of the state, producing no benefit to Kansas City, Missouri. In fact, from about 1995 to present, KCMSD funding has generally slid backwards.

Preston Green and I document the disconnect between desegregation litigation in Kansas City and claims of school funding failures in this article: Urban Legends, Desegregation and School Finance: Did Kansas City Really Prove That Money Doesn’t Matter? (which appeared in the Michigan Journal of Race and Law). Among other things, we note:

Critics cite the statistics the KCMSD spent more than $11,000 per pupil and that $2 billion were spent on the desegregation plan as evidence of exorbitant spending. When taken out of context, these numbers appear huge. However, our analysis reveals that the KCMSD was a very high spending district for no more than five years, or the time in which one cohort of children is able to progress through five grade levels in the district. Further, when adjusted for student needs, the KCMSD’s funding dropped below the metropolitan area average by 1998. This is hardly enough time to erase the generational poverty of the KCMSD or alter the residential structure and demographics of a school district that had been designed to be racially segregated until the 1960s.

Biddle also points to Newark, NJ as providing evidence of the failures of school finance reforms. Yet, New Jersey is among those states we discuss here as having some (albeit limited due to data quality) evidentiary basis for the positive effects of state school finance reforms, including court ordered reforms. In School Finance and Courts: Does Reform Matter, and How Can We Tell?, Kevin Welner and I discuss the many flawed claims about the dreadful failures of attempts to improve equity and adequacy of school funding. We also point out how the Kansas City case does not even fit into this category. Regarding the general, popular claims of the failures of funding reforms, Kevin Welner and I review the basis for those claims and conclude:

We conclude that there is arbitrariness in how research in this area appears to have shaped the perceptions and discourse of policymakers and the public. Methodological complexities and design problems plague finance impact studies. Advocacy research that has received considerable attention in the press and elsewhere has taken shortcuts toward desired conclusions, and this is troubling.

We also review more rigorous peer-reviewed studies and find, on balance, that those studies show positive effects of school finance reforms, both in terms of improving equity in student outcomes and in terms of improving the overall level of student outcomes.

Higher quality research, in contrast, shows that states that implemented significant reforms to the level and/or distribution of funding tend to have significant gains in student outcomes. Moreover, we stress the importance of the specific nature of any given reform: positive outcomes are likely to arise only if the reform is both significant and sustained.

And now for the truly ironic part of Biddle’s claim. So, Biddle’s argument to Jealous is that Jealous should drop all this funding equity and integration crap from NAACP’s past, and focus on charters – expanding access to charters. After all, if Kansas City, Missouri had not wasted all that time arguing in court over money and chasing more equitable funding and instead had spent its time pursuing an aggressive strategy of increasing numbers of charter schools, kids in Kansas City – especially poor, minority kids – would have much better educational opportunities!!!!

Biddle argues (regarding NAACP):

It must embrace the charter school movement: After all, charters have been the leading source of improving access to high quality education for urban black and Latino communities, who would otherwise be forced to attend the dropout factories in their neighborhoods. The success of charter school operators such as KIPP and Uncommon Schools — all of which educate mostly-minority students — can be replicated throughout the nation.

I wouldn’t have even written this response (to a post not really worthy of response) had it not been for the fact that I addressed this very topic the other day. As funding for KCMSD was receding as the district moved toward unitary status, what did happen in KCMSD? The massive expansion of charter schools! Here’s what I wrote the other day, upon release of a very interesting report from Kauffman Foundation regarding educational opportunity in Kansas City and the role of charter schools:

Kansas City is #4 on charter market share, according to the National Alliance report, and rose to that position much earlier in the charter proliferation era than other cities. As a result, by reformy logic, Kansas City should be a hotbed for educational opportunity for school-aged children – after years of previously throwing money down the drain in the Kansas City Missouri Public School District (many of these claims actually being Urban Legend).

In Kansas City, the reality of charter expansion has clashed substantially with the reformy ideology. Arthur Benson in a recent Kansas City Star Op Ed, noted:

Charters have subtle means for selecting or de-selecting students to fit their school’s model. The Kansas City School District keeps its doors open to non-English speakers and all those kids sent back from the charter schools. In spite of those hurdles, Kansas City district schools across the board out-perform charter schools. That is not saying much. We have until recently failed 80 percent of our kids, but most charters fail more.

I was initially curious about Benson’s (a district board member and attorney) claims that charters have done so poorly in Kansas City. Could it really be that the massive expansion of charter schools in Kansas City has done little to improve and may have aided in the erosion of high quality educational opportunities for Kansas City children?

The recent Kauffman Foundation report draws some similar conclusions, and Kauffman Foundation has generally been an advocate for charter schools. The report classifies district and charter schools into groups by performance, with level 4 being the lowest, and level 1 being the only acceptable group.

  • Level I- A school that met or exceeded the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level II- A school that scored between 75 and 99 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level III– A school that scored between 50 and 74 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level IV– A school that scored below 50 percent of the state standard on the MAP Communication

Among other things, the report found that charter operators had avoided opening schools in the neediest neighborhoods. Rather, they set up shop in lower need neighborhoods, potentially exacerbating disparities in opportunities across the city’s zip codes. The report recommended:

A strategy for charter school growth should be developed by Kansas City education leaders. Charter schools should only be approved by DESE if they can demonstrate how they intend to fill a geographic need or a specific void in the communities they intend to serve.

Regarding charter performance more generally, the report noted:

In many communities charter schools are a model that increases students’ access to better public schools, but the majority of charter school students (5,490 or 64.7 percent) are in a Level IV school. Many of Kansas City’s charters have existed for 10 years and are still not able to reach even half of state standard.

Now, I’m not sure I accept their premise that in many communities this actually works – and that it just went awry for some strange reason in Kansas City. That said, the reality in Kansas City, by the authors own acknowledgment is in sharp contrast with the reality the authors believe exists in other cities.

One implication (not tested directly) of this report is that the massive charter school expansion that occurred in Kansas City may have done little or nothing to improve the overall availability or distribution of educational opportunities for children in that city and may have actually made things worse.

So, Mr. Biddle, I urge you to do a little reading. Check a few facts and consider your arguments carefully. Your current arguments FLUNK at even the most basic level.

More importantly, others who come across such bombastic claims like those argued by Biddle should scrutinize those claims carefully. Heck, I hope you scrutinize the stuff in my own posts carefully too. I try to shoot for a reasonably high level of rigor and factual accuracy in these posts and do what I can to cite my claims to respectable sources. Biddle’s blog to Benjamin Jealous is, to me, an example of the worst form of ill-conceived, factually incorrect, contorted reform logic out there. Sadly, there’s way too much of it.

READINGS

Baker, B.D., Welner, K. (2011) School Finance and Courts: Does Reform Matter, and How Can We Tell? Teachers College Record 113 (11) p. –

Green, P.C., Baker, B.D. (2006) Urban Legends, Desegregation and School Finance: Did Kansas City really prove that money doesn’t matter? Michigan Journal of Race and Law 12 (1) 57-105

When reformy ideologies clash…

(note: lots of ideas here that I wanted to start writing about… but not yet well organized or articulated. It will come, with time, I hope.)

Summary of Reformy Ideology

Bluntly stated, the two major components of education reform ideology are as follows:

  • Reformy Ideology #1: Teacher quality is the one single factor that has the greatest effect on a child’s life chances. Get a bad teacher or two in a row, and you’re screwed for life. The “best possible” way to measure teacher quality is by estimating the teacher’s influence on student test scores (value-added). Hiring, retention and dismissal decisions must, that is, MUST be based primarily on this information. This information may be supplemented, but value-added must play the dominant single role.
  • Reformy Ideology #2: Charter schools are the answer to most of the problems of poor urban school districts. Take any poor, failing urban school district, close its dreadfully failing schools and replace them as quickly as possible with charter schools and children in the urban core will have greatly expanded high-quality educational opportunities.

Now, let me do a bit of clarification here. These are the representations of reform ideology at the extremes – but these views are not uncommon in the “reform” community. Let me also clarify that item #2 above isn’t about the broader issue of charter schooling, the origins of charter schooling, purposes of individual charter schools or research on the effectiveness of specific charter school models. Item #2 above is specifically about the argument that large urban districts can and should be replaced with an open market of charter schools – that charter schools should not just be used to try out new and interesting ideas which may be replicated in other schools – charter or not – but rather that charters should become dominant providers of urban education.

In my framing of item #1 above, I do not by any means intend to discredit the importance of high quality teachers. I’m with the “reformers” on that idea. But, it is certainly an overstatement to attribute all of student gains and life chances to teacher quality alone. And, as I have discussed previously on many occasions on this blog, it is very problematic to assume that we presently have sufficient tools for measuring precisely and accurately the true effectiveness of any single teacher.

So, this brings me to recent completely unrelated events and media on education reform issues that raise some interesting points of conflict.

Part I: Ideology Clashing with… Ideology

The first example of the clash of reformy ideologies comes from upstate New York as the state begins the process of implementing the “reforms” that got that state Race to the Top funding. In short, charter school operators really don’t seem to want to be compelled to adopt the first prong of reformy ideology. What? Charters don’t want to be compelled to use student test scores as the primary, or even a major basis for personnel decisions? Blasphemy!

In recent weeks, in casual conversations and at symposia, I’ve actually heard a number of charter school operators raise serious questions about being compelled to adopt Reform Ideology #1 above. Charter operators appreciate their autonomy, and while most do enjoy wider latitude over personnel decisions than large urban school districts that serve as their hosts, most do not necessarily base the majority of their hiring, firing and compensation decisions on student test scores and test scores alone. And most don’t seem very interested in being compelled to do so – adopting a one size fits all evaluation model. Apparently, they are also relatively uninterested in disclosing how they evaluate faculty. Here’s a quote from the Albany Times Union.

Carroll, one of the most prominent education reformers in the state, helped write the state’s original charter laws. He said if the charter schools accepted the money, they would lose their current flexibility in the firing and hiring of teachers. He also said charter schools would be forced to disclose their teacher evaluation process, which is now confidential, and that it could become harder to fire an educator deemed ineffective.

http://www.timesunion.com/default/article/No-to-cash-with-a-catch-714008.php

So then, if expanding charters is a major component of reform, and making sure teachers are evaluated by student test scores is a major component of reform, how can this apparent clash be reconciled? It can’t! It seems hypocritical at best to force public school districts to play by a seriously flawed set of teacher evaluation rules and then let charters off the hook? This is especially true if one of the supposed benefits of charter schools is to experiment with creative strategies that may be emulated by traditional public schools and if traditional public schools are expected to improve by competing with charters on a level playing field. I’m with the charter leaders on this one.

UPDATE: Tom Carrol has clarified his comments here: http://www.nyfera.org/?p=2827, where he attempts to explain that charters are opposed to the teacher evaluation requirements not because charters oppose the ideas behind using data to evaluate teachers, but that charters oppose having the state education department mandate how that data should be used in evaluations:

SED simply has no authority to set thresholds for the use of data in teacher evaluations in charter schools.  Nor do they have the authority to require us to group teachers by four categories, or require such annual evaluations to be “a significant factor” for “promotion, retention, tenure determination and supplemental compensation.”  Nor do they have the authority to require charters to pursue “the removal of teachers and principals receiving two consecutive annual ratings of ‘ineffective’ after receiving supports from improvement plans.”

Carrol’s clarification coupled with his unsubstantiated claim that charters are already doing these things really doesn’t change my point above – that it remains hypocritical to hoist these deeply problematic policies on traditional public schools while letting charters off the hook on the basis that charters should be given the flexibility to experiment with alternative strategies and should be exempt from full disclosure regarding those strategies.

Part II: Ideology Clashing with Research

Now, this one is really not an obvious major clash, but rather a more subtle clash between ideology and research embedded in Eric Hanushek’s WSJ editorial the other day.  Embedded in the editorial were comments/claims that I found at least a little disingenuous.

The point of Hanushek’s OpEd was to explain that there is no war on teachers, and that this is really about getting teacher’s unions to negotiate for reasonable changes in contracts that would allow for more expeditious dismissal of the worst teachers – the bottom 5% or so. I’ll admit that I really haven’t seen Hanushek himself outright attacking teachers in his work of late, especially in his actual research on teacher labor markets. Much of it is very good and very useful stuff. That said, it’s hard to deny that many major public figures – talking head tweeters and bloggers for think tanks, etc. – have actually engaged in an all out attack on teachers, the teaching profession and teachers unions.

Setting that broader issue aside, Hanushek’s Op Ed rubbed me the wrong way because of examples he chose to advance his argument, and the extent to which his examples in the Op Ed clash – and clash quite significantly – with his own best recent research. Hanushek summarized his arguments about the non-war on teachers as follows:

What’s really going on is different. President Obama states that we can’t tolerate bad teachers in classrooms, and he has promoted rewarding the most effective teachers so they stay in the classroom. The Los Angeles Times published data identifying both effective and ineffective teachers. And “Waiting for ‘Superman'” (in which I provide commentary) highlighted exceptional teachers and pointed out that teachers unions don’t focus enough on teacher quality.

This is not a war on teachers en masse. It is recognition of what every parent knows: Some teachers are exceptional, but a small number are dreadful. And if that is the case, we should think of ways to change the balance.

http://online.wsj.com/article_email/SB10001424052748703794104575546502615802206-lMyQjAxMTAwMDEwODExNDgyWj.html

So, part of his claim is that it was unjustified for teachers unions – not teachers mind you – but their unions to object so loudly when the LA Times merely – in the public interest – revealed data which validly (implied above, by absence of disclaimers) identified, labeled and named effective and ineffective teachers.

Wait… isn’t it Eric Hanushek’s own research and writing that highlights problems with using value-added measurement to evaluate teachers where non-random student assignment occurs (which is pretty much anywhere)? For me, it was my familiarity with his work that led me to explore the biases in the LAT model that I’ve written about previously on this blog.  In that same post, I explain why I am more inclined to accept Jesse Rothstein’s concerns over the problems of non-random assignment of students than to brush those concerns aside based on the findings of Kane and Staiger.  Hanushek provides a compelling explanation for why he too places more weight on Rothstein’s findings.

Correct me if I’m wrong, but isn’t it possible that teachers and their union were in an uproar at least partly because the LA Times released highly suspect, potentially error ridden and extremely biased estimates of teacher quality? And that the LA Times misrepresented those estimates to the general public as good, real estimates of actual teacher effectiveness?

Yes, much of Eric Hanushek’s recent writing does advocate for some reasonable use of value-added estimates for determining teacher effectiveness, but he usually does so while giving appropriate attention to the various caveats and while emphasizing that value-added estimates should likely not be a single determining factor. He notes:

Potential problems certainly suggest that statistical estimates of quality based on student achievement in reading and mathematics should not constitute the sole component of any evaluation system.

http://edpro.stanford.edu/hanushek/admin/pages/files/uploads/HanushekRivkin%20AEA2010.CALDER.pdf

Yet, that’s just what the LA Times did, and without even mentioning the caveats!

Isn’t it a bit of an unfair assertion, given Hanushek’s own research and writing on value-added estimates, to claim that LA teachers and their union were completely unjustified in their response to the LA Times?

Part III: Ideology Clashing with Reality

There exists at least one segment of the truly reformy crowd that believes deeply in second major ideology laid out at the beginning of this post – that if we can simply close failing urban schools (the whole district if we have to!) and let charters proliferate, children in the urban core will have many more opportunities to attend truly good schools. Yes, these reformers throw in the caveat that we must let only “good” charters, “high performing” charters start-up in place of the failing urban schools. And when viewing the situation retrospectively, these same reformy types will point out that if we look only at the upper half of the charters, they are doing better than average. Yeah… Yeah… whatever.

One long-term research project that has interested me of late is to look in-depth at those “failing” urban school districts that over the past decade have had the largest shares of their student population shift to charter schools – that is, the largest charter market share districts. Here is link to the Charter Market share report from the National Alliance for Public Charter Schools: http://www.publiccharters.org/Market_Share_09

It would seem that if we adopt the reformy ideology above, that if we identify those districts with the largest charter market shares, those districts should now be models for high quality, equitably distributed educational opportunities. We should eventually see sizeable effects in the achievement and attainment of children growing up in these cities, we should see quality of life increasing dramatically, housing values improving with an influx of families with school-aged children – a variety of interesting, empirically testable hypotheses, which I hope to explore in the future.

In the mean-time, however, we have new and interesting descriptive information from a report from the Ewing and Marion Kauffman Foundation in Kansas City, focused on educational opportunities in Kansas City. Kansas City is #4 on charter market share, according to the National Alliance report, and rose to that position much earlier in the charter proliferation era than other cities. As a result, by reformy logic, Kansas City should be a hotbed for educational opportunity for school-aged children – after years of previously throwing money down the drain in the Kansas City Missouri Public School District (many of these claims actually being Urban Legend).

In Kansas City, the reality of charter expansion has clashed substantially with the reformy ideology. Arthur Benson in a recent Kansas City Star Op Ed, noted:

Charters have subtle means for selecting or de-selecting students to fit their school’s model. The Kansas City School District keeps its doors open to non-English speakers and all those kids sent back from the charter schools. In spite of those hurdles, Kansas City district schools across the board out-perform charter schools. That is not saying much. We have until recently failed 80 percent of our kids, but most charters fail more.

I was initially curious about Benson’s (a district board member and attorney) claims that charters have done so poorly in Kansas City. Could it really be that the massive expansion of charter schools in Kansas City has done little to improve and may have aided in the erosion of high quality educational opportunities for Kansas City children?

The recent Kauffman Foundation report draws some similar conclusions, and Kauffman Foundation has generally been an advocate for charter schools. The report classifies district and charter schools into groups by performance, with level 4 being the lowest, and level 1 being the only acceptable group.

  • Level I- A school that met or exceeded the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level II- A school that scored between 75 and 99 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level III– A school that scored between 50 and 74 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level IV– A school that scored below 50 percent of the state standard on the MAP Communication

Among other things, the report found that charter operators had avoided opening schools in the neediest neighborhoods. Rather, they set up shop in lower need neighborhoods, potentially exacerbating disparities in opportunities across the city’s zip codes. The report recommended:

A strategy for charter school growth should be developed by Kansas City education leaders. Charter schools should only be approved by DESE if they can demonstrate how they intend to fill a geographic need or a specific void in the communities they intend to serve.

Regarding charter performance more generally, the report noted:

In many communities charter schools are a model that increases students’ access to better public schools, but the majority of charter school students (5,490 or 64.7 percent) are in a Level IV school. Many of Kansas City’s charters have existed for 10 years and are still not able to reach even half of state standard.

Now, I’m not sure I accept their premise that in many communities this actually works – and that it just went awry for some strange reason in Kansas City. That said, the reality in Kansas City, by the authors own acknowledgment is in sharp contrast with the reality the authors believe exists in other cities.

One implication (not tested directly) of this report is that the massive charter school expansion that occurred in Kansas City may have done little or nothing to improve the overall availability or distribution of educational opportunities for children in that city and may have actually made things worse.

Isn’t it strange how we hear so little about these things as we look to replicate these models of great reformy success in other cities of comparable scale such as Newark, NJ?

A few thoughts on the unlikely alliance…

Today was the day of the big Oprah-Christie-Booker-Zuckerberg event, which I guess we can all watch around 4pm if we really want to. I’ve been trying to dig up any information I can, without wasting too much time on this, because there are certainly more important things to get to. That said, I do have a few brief comments in response to specific points and issues raised.

In an effort to get a good soundbite, Mayor Booker commented on Oprah that “You can not have a superior democracy with an inferior system of education” a comment that has now been re-tweeted over a hundred times. Here’s the thing. This whole situation is about a philanthropic contribution from a single wealthy individual, which has been described in the media as a contribution that carries with it a stipulation that the Governor grant unprecedented power to the Mayor to control Newark Public Schools. Anyone else seeing the contradiction here? My basic summary points are:

  • We should be concerned and skeptical any time a single individual uses their wealth to buy substantive changes to public policy.
  • Setting aside Booker’s loose use of the term democracy, I have to ask: Is it really democratic to have a single individual pay to alter the very structure of state and local government?

Would that be “democracy hypocrisy?”

Next on my list – the nature of the preferred reforms. We have little specific information on the types of school reforms that Mark Zuckerberg would like to see implemented in Newark or whether he has any specific interest in promoting certain education reforms. Zuckerberg provides some insights in this interview: http://techcrunch.com/2010/09/24/techcrunch-interview-with-mark-zuckerberg-on-100-million-education-donation/. Perhaps the most striking part of the interview is here:

So that – that way Cory is really aligned towards one – like this is his top priority. He just got re-elected by a pretty big margin and it’s his biggest priority. Then, so now – so that’s kind of what we’re doing, I mean, the idea is fund him and basically support him in doing a really comprehensive program to get all these things in place that they need to get done. [DELETE: So we should close down schools that are failing, get a lot of good charter schools and figure out new contracts for teachers so that better teachers can get paid more money, that more for performance as opposed to just based on how long you’ve been there. Have a lot of programs that are after schools that to keep kids healthy and safe and I mean, Newark, isn’t the safest city. So that’s the basic thing. And I mean for…]

I was particularly intrigued by that part in brackets, after DELETE – where Zuckerberg or the interviewer interpreting Zuckerberg seems to be suggesting a strong preference for massive charter expansion, closing public schools, and pushing for teacher contracts tied to student achievement data. The implication across media sources yesterday and today has been that the preference for these specific types of reforms across this seemingly diverse set of individuals – Zuckerberg, Oprah, Christie and Booker – validates the public interest in moving quickly toward accomplishing these public policy objectives. Setting aside the issue that fast-tracking these reforms under these circumstances is built on buying – with a big $$ gift – a change in state and local governance, I offer the following comments regarding this new unlikely alliance and these specific reform strategies:

  • It’s interesting to see such an eclectic cast of characters unify around a set of unproven and ill-conceived school reform strategies to hoist upon the children of Newark.
  • The fact is that major research organizations including the National Research Council, American Education Research Association, National Council on Measurement in Education, American Psychological Association and others have advised strongly against misusing student testing data to evaluate teacher effectiveness and there are many technical and statistical as well as practical reasons for their conclusions. With all due respect, a consensus vote in favor of these flawed policies from our Governor, the Newark Mayor, Oprah and Mark Zuckerberg doesn’t change that.

More Information:https://schoolfinance101.wordpress.com/category/race-to-the-top/value-added-teacher-evaluation/

  • The reality is that two of Newark’s most acclaimed charter schools – Robert Treat and North Star both serve far fewer of the lowest income children than nearby Newark Public Schools (43% to 47% compared to over 70% NPS) and very few children with disabilities (3.8 to 7.8% compared to 18.1% NPS) or limited English skills. It may be ‘working’ for them, but that’s not scalable reform. Eventually someone has to serve all of those other kids.

Data link: https://sites.google.com/site/schoolfinancepolicy/Home/NJCharters.xls?attredirects=0&d=1

Update from the Star Ledger: http://www.nj.com/news/index.ssf/2010/09/facebook_ceo_mark_zuckerberg_s.html

Apparently the “deleted” section has been removed from the interview, but I’m not the only one who saw it!

A few pictures related to my comment on charter school demographics:


And here are the 2009 assessment results for NPS and Newark Charter schools. As you can see, the very low poverty charters do very well. But they just aren’t comparable to NPS schools. Other higher poverty charters, which are still actually much lower poverty (and low or no special ed) than NPS schools, are actually distributed among the NPS schools, regardless of test subject or grade.


If money doesn’t matter…

A) Then why do private independent schools, like those attended by our President’s children (Sidwell Friends in DC), or by Davis Guggenheim’s children (?), spend so much more than nearby traditional public schools?

Davis Guggenheim, producer of Waiting for Superman, frequently explains to the media these days that he feels uneasy that he has made a personal choice drive by his neighborhood school each day to bring his children to a private school. Now, I don’t know which private school his children attend, but I would suspect (though I may be wrong) that it is more likely to be an academically elite, private independent school than to be a conservative Christian or urban Catholic school. As I discuss in this previous report, the spending differences and resulting programmatic resources and teacher characteristics by type of private school are striking: http://epicpolicy.org/publication/private-schooling-US

I would see little problem with Guggenheim’s personal anecdote were it not for one of the central arguments of Superman being that money plays little or no role in fixing public education systems. Instead, tough-minded superintendents like Michelle Rhee, or charter schools are the solution – money or not.

Again, I’ll fess up to the fact that I am a former teacher at and big supporter of Private Independent Schools. Here’s the school in New York City where I used to teach www.ecfs.org, and here’s its page on tuition: http://www.ecfs.org/admission/tuition.aspx. It was then, and I suspect still is an outstanding example of what a school can be! But that outstanding-ness comes at a price!

(approximately $36,000 per year for middle school and up)

The problem with the assertion that “money wouldn’t help public schools anyway” is that many of those pitching the argument seem themselves to favor private schools that spend more – A LOT MORE – per child than the public schools they are criticizing as failing (speak nothing of the fact that the public schools are serving a much more diverse student population).

Here are some comparisons pulled from my 2009 study on private school expenditures.

First, here’s the per pupil spending in 2005-06 for a handful of major labor markets that had sufficient numbers of Private Independent Day Schools for calculating the averages. My original sample of IRS Tax filings covered about 75% of all Private Independent Day Schools (NAIS or NIPSA member schools), so these are not “outlier” schools.

FIGURE 1 (This figure is now the figure from my original report: http://epicpolicy.org/publication/private-schooling-US)

And here are the regional averages, adjusted for regional differences in competitive wages, using the NCES Education Comparable Wage Index.

FIGURE 2 (This figure is now the figure from my original report: http://epicpolicy.org/publication/private-schooling-US)

If money doesn’t matter when it comes to school quality, then why not pick one of those private schools that charges only $6,000 in tuition, and spends $8,000 per pupil? Clearly there is some basis for the decision to send a child to a more expensive private school? There is some “utility” placed on the differences in what those schools have to offer? In the complete report above, I discuss (in painful detail) those differences across private schools, but here, I quickly summarize some of the differences between private independent schools and traditional public school districts.

FIGURE 3 (This figure is now the figure from my original report: http://epicpolicy.org/publication/private-schooling-US)

Private independent schools a) spend a lot more per pupil, b) have much lower pupil to teacher ratios and c) have much higher shares of teachers who attended more competitive colleges. These seem like potentially substantive differences to me. And they are differences that come at a cost.

I am by no means criticizing the choice to provide your own child with a more expensive education. That is a rational choice, when more expensive is coupled with substantive, observable differences in what a school offers. I am criticizing the outright hypocritical argument that money wouldn’t/couldn’t possibly help public schools provide opportunities (breadth of high school course offerings, smaller class sizes) more similar to those of elite private independent day schools, when this argument is made by individuals who prefer private schools that spend double what nearby public schools spend.

Sidebar: I suspect there are few if any private independent day schools out there which currently evaluate their teachers based on student test score gains alone. Please let me know if you know of one? And, I should note that the private independent school where I worked in New York City was actually unionized and had a tenure system in place with a probationary period similar to that of public schools and a salary schedule tied to experience.

B) Then why do venture philanthropists continue to throw money at charter schools while throwing stones at traditional public schools?

Charter school backers like Whitney Tilson love to throw stones at public schools while throwing money at charter schools. Here’s one of his presentations:

http://www.tilsonfunds.com/Personal/TheCriticalNeedforGenuineSchoolReform.pptx

On Slide 13, Whitney Tilson opines that increased spending on public education has yielded no return to outcomes over time, and therefore, by extension, increased spending would not and could not help public schools in the future. Tilson is featured prominently in this New York Times article on affluent fund managers in NYC rallying for charter schools: http://www.nytimes.com/2010/05/10/nyregion/10charter.html?pagewanted=all

So, here we have one of many prominent New York City charter school supporters on the one hand arguing that throwing more money at the public school system could not possibly help that system, but on the other hand, providing substantial financial assistance to charter schools (or at least participating in and promoting groups that engage in such activity)?

A New York City Independent Budget Office report suggested that charter schools housed in public school facilities have comparable public subsidy to traditional NYC public schools, but charter schools not housed in public school facilities have to make up about $2,500 (per pupil) in difference. I will show in a future post, however, that student population differences (charters serving lower need populations) largely erase this differential.

Kim Gittleson points out here, that in 2008-08, NYC Charter schools raised an average of $1,654 per pupil through philanthropy. But, some raised as much as $8,000 per pupil. As a result, some charters – those most favored by venture philanthropists – spend on a per pupil basis much more than traditional NYC public schools (including KIPP schools). I will provide much more detail in this point in a future post.

One might argue that the Venture Philanthropists are trying to spend their way to success – To outspend the public schools in order to beat them! After all, it’s the New York Yankee, George Steinbrenner way? (spoken from the perspective of a Red Sox fan, who spent the last several years in Kansas City, supporting the underdog – low payroll – Royals).

But here’s the disconnect – These same Venture Philanthropists – like Tilson, who are committed to spending whatever it takes on charters in order to prove they can succeed, are arguing that public schools a) don’t need and b) could never use effectively any more money. They are trying to argue that charters are doing more with less, when some are doing more with more, others less with less, and some may be doing more with less, and others are actually doing less with more. Shouldn’t traditional public schools be given similar opportunity to do more with more? And don’t give me that … “we’ve already tried that and it didn’t work” claim. I’ll gladly provide the evidence to refute that one, much of which is in the article at the bottom of this post!

C) Then why do affluent – and/or low poverty – suburban school districts continue in many parts of the country to dramatically outspend their poorer urban neighbors?

Last but not least, why do affluent suburban school districts in many states continue to far outspend poor urban ones? If there is no utility to the additional dollar spent and/or no effect produced by that additional dollar then why spend it?

Here is the overall trend, over time in the relationship between community income and state and local revenues per pupil.

When the red line is above the green horizontal line, there exists a positive relationship between district income and state and local revenue. That is, higher income districts have more state and local revenue per pupil. The red line never drops below the green line. This graph, drawn from this article (http://epaa.asu.edu/ojs/article/view/718) shows that state and local revenues per pupil remain positively associated with income across school districts nationally, after controlling for a variety of factors (see article for full detail). Things improved somewhat in the 1990s, but then leveled off.

FIGURE 4 (from: http://epaa.asu.edu/ojs/article/view/718)

Here are the trends for mid-Atlantic states, where some including New York State improved, but remain strongly associated with income. New Jersey is the only state among these where the relationship between income and revenue is disrupted and ultimately reversed.

FIGURE 5 (from: http://epaa.asu.edu/ojs/article/view/718)

Here are the trends for the New England trend, where New Hampshire school district state and local revenues remain strongly tied to income.

FIGURE 6 (from: http://epaa.asu.edu/ojs/article/view/718)

Here are the trends for the Great Lakes are trend, where Illinois remains among the most regressively funded systems in the nation (along with New York).

FIGURE 7 (from: http://epaa.asu.edu/ojs/article/view/718)

Here’s a specific look at state and local revenues per pupil in New York State districts in the NY metropolitan area, with districts organized by U.S. Census Poverty rates.

FIGURE 8

Is there a reason why Westchester County and Long Island school districts choose to spend so much more than New York City on a per pupil basis? What about those North Shore Chicago area districts?

These communities demand higher expenditures per pupil for their schools on a presumption that the marginal dollar does not go entirely to waste – that there is some value, some return for that dollar, perhaps in the richness of supplemental programs offered or the smaller class sizes – much like the differences in private schools seen above.

Finally, I point you to this recently published article in Teachers College Record, where Kevin Welner and I try to set the record straight on the effectiveness of “reforms” involving state school finance systems. They’re not the “reformy” reforms, but school finance reforms are reforms nonetheless.

Baker, B.D., Welner, K. School Finance and Courts: Does Reform Matter, and How Can We Tell? Teachers College Record

http://www.tcrecord.org/content.asp?contentid=16106

DoReformsMatter.Baker.Welner

Searching for Superguy in Gotham

Who is Superguy? By most popular accounts, Superguy is a figure of mythical proportion (urban legend proportion) capable of swooping down into the poorest of urban neighborhoods of America’s largest cities, gaining immediate access to schooling facilities, rounding up unthinkable private contributions from wealthy philanthropists and quite simply saving the lives of low-income urban school children trapped in bleak, adult-centered, perpetually failing traditional public schools.

Superguy could be found anywhere in the U.S where urban charter schools have proliferated in the past decade – Kansas City, Washington D.C., Chicago, Dallas, Houston, or even more likely, New York City – Gotham itself (yeah… Gotham was a Batman thing, not Superman… but hang in there with me).  I’ve chosen to focus on urban locations here, because who has ever heard of a “rural legend?”

I’ve written on a number of occasions about my general skepticism that Superguy really exists, or that he necessarily exists in the form of an urban charter school operator. My skepticism is based on my own read of the balance of research on charter schools and my own casual analysis  of New York City and New Jersey Charter Schools. New Jersey Charter Schools in particular are pretty average and those that are better than average serve very few of the lowest income children, no special needs children and few or no limited English proficient children. Personally, I’d expect Superguy to be out there fighting for these kids in particular, not just setting up shop in their neighborhood and cream-skimming the less needy among the more needy. But hey, that’s just my notion of what Superguy should be.

For an exceptional review of charter school research, I would recommend Robert Bifulco and Katrina Bulkley’s chapter on Charter Schools in the Handbook of Research on Education Finance and Policy. Neither of these scholars are charter school naysayers, yet they conclude:

Research to date provides little evidence that the benefits envisioned in the original conceptions of charter schools – organizational and educational innovation, improved student achievement, and enhanced efficiency – have materialized.

Of course, the true believers in Superguy (as charter operator) will argue vehemently that the finding that charters, on average, are average does not shake their belief… because the “upper half of charter schools is really good!, better than average, in fact!” Skeptically, I respond – isn’t the upper half of all schools better than average? If so, might Superguy actually be found in any school that’s better than average? But who am I to nitpick?

The most compelling evidence that Superguy exists was provided in Caroline Hoxby’s finding regarding NYC charter schools that:

On average, a student who attended a charter school for all of grades kindergarten through eight would close about 86 percent of the “Scarsdale-Harlem achievement gap” in math and 66 percent of the achievement gap in English.

Who other than Superguy could close the Harlem-Scarsdale gap? However, Stanford University researcher Sean Reardon explains:

Because the report relies on an inappropriate set of statistical models to analyze the data, however, the results presented appear to overstate the cumulative effect of attending a charter school.

Superguy in Gotham is also assumed to have competitive effects, lifting entire neighborhoods wherever he may be present. This evidence is often cited to Marcus Winters’ (Manhattan Institute) finding that:

The analysis reveals that students benefit academically when their public school is exposed to competition from a charter.

But thwarting this Superguy sighting is Wellesley economist Patrick McEwan’s observation that:

The statistical analysis suggests that increasing competition has no statistically significant impact on math test scores, but that it has small positive effects on language scores. The report does not conclusively demonstrate that the results are explained by increasing competitive pressure on public school administrators; they may also be explained by shifting peer quality or declining short-run class sizes in public schools.

Those pesky, curmudgeonly,  academics are at it again… denying the true believers that Superguy comes in the singular form of a New York City charter school operator!

Then there’s the claim that Superguy himself may have been outed in Harlem (is Superguy really Geoffrey Canada?) – as evidenced by Dobbie and Fryer’s studies of the amazing success of Harlem Children’s Zone. But then Russ Whitehurst of Brookings stepped in to rain on this parade, finding the HCZ Promise academy to be relatively average as far as NYC charter schools go.

For several additional curmudgeonly critiques of Superguy sightings, see: http://epicpolicy.org/search/epicpolicy/charter

It is with these contradictory findings in mind that I present the following figures, and we begin our statistical search for Superguy. Now, this is not a really deep, statistically rigorous search for Superguy. The approach here is what some refer to as a “beating the odds” approach (BTO) and is similar to the adjusted performance approach used by Whitehurst in his Brooking critique of HCZ.

It seems that the logical place to start would be New York City, home to the greatest number of Superguy sightings. Let’s begin with a simple flyover of NYC schools, including traditional public schools and charter schools just to get a feel for the demographics of those schools.

Here is the % of children qualifying for Free Lunch across Harlem (and South Bronx) schools. This map does not indicate which schools are charters, but you can click the link below the map to see which ones are.

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS (indicated with an asterisk)

Here is the % of children who are limited in their English language proficiency. Again, this map doesn’t show us which schools are charters, but you can click the link below.

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS (indicated with an asterisk)

Now, here’s our first “Beating the Odds” scatterplot. The predicted performance values expressed on the vertical axis are from a regression equation that accounts for a) limited English proficiency shares, b) free lunch shares, c) mobility shares, d) borough and e) year (includes 2008 & 2009). These graphs look at the adjusted performance levels (not value-added) of NYC traditional public and NYC charter schools (standardized difference between actual and predicted values for 2009). These are illustrative. While outcome levels do go up (are inflated) in 2009, the distributions in these scatterplots don’t change a whole lot if I use 2008 or earlier. (here are the models)

The first Beating the Odds scatterplot looks at the average performance from 2009 (yes, the really inflated test score year) for NYC public schools. Charter schools are not identified in this graph. Schools are displayed from low to high poverty along the horizontal axis. Schools above the red horizontal line are beating the odds, or scoring higher than expected given their location and student population. Schools below the line are, well, not beating the odds. We would, of course, expect our Superguy operated schools to be flying high above the rest and certainly not falling well below… at the bottom of the scatter. So, where is Superguy?

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS

Now, here’s the average of the 4th and 5th grade outcomes. Same deal. A good ol’ BTO analysis (yeah… this isn’t really rigorous stuff, but it is illustrative). Again, charters are not identified in this picture. Yes, there are some high and some low flyers in this graph too. But are all of the high flyers charter schools? Is Superguy really here?

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS

While I don’t think we’ve found Superguy here, we are left with some potential clues about the conditions surrounding Superguy sightings – A) that superguy sightings seem more common in the presence of unexplainable deficits in the shares of children who qualify for Free Lunch, B) that Superguy sightings seem more common in the presence of unexplainable deficits in the shares of children with limited English proficiency. Other than that, it seems that Superguy is equally likely to be hiding in a traditional New York public school as it is that Superguy is secretly disguised as a charter school operator somewhere in Gotham.

Alternatively, there exists the depressing but real possibility that Superguy simply doesn’t exist – at least not in the expected form. That there just isn’t a charter school operator out there who can single-handedly swoop into poor urban neighborhoods and save childrens lives – creating results never seen before with a truly representative population of children. Or, at the very least, not all or even the average charter school operator qualifies as Superguy. Yes, some are better than others. And, some are quite good. But you know what I have to say about that argument (see above).

Yeah… I’d like to be a believer. I don’t mean to be that much of a curmudgeon. I’d like to sit and wait for Superguy – perhaps watch a movie while waiting (gee… what to watch?). But I think it would be a really long wait and we might be better off spending this time, effort and our resources investing in the improvement of the quality of the system as a whole. Yeah, we can still give Superguy a chance to show himself (or herself), but let’s not hold our breath, and let’s do our part on behalf of the masses (not just the few) in the meantime.

More Fun with New Jersey Charter Schools

LINK TO UPDATED SPREADSHEET OF FREE LUNCH AND SPECIAL ED DATA

I love maps. I love GIS software. This is a particularly interesting one related to the shares of children who qualify for free (not free or reduced, but free only, a poorer population) lunch in traditional public schools and in charter schools in Newark. One reason why mapping is useful here is that it is important to compare school demographics with other nearby schools, rather than district average. This map of a portion of Newark pretty much speaks for itself. Click to enlarge the map (to read the free lunch ranges on the key). Clearly, two of the “high performers” among charters – North Star and Robert Treat, have noticeably lower free lunch rates than other schools around them (except for other special schools).(CS indicates Charter School)

Data for this map were acquired from the National Center for Education Statistics Common Core of Data – Public School Universe Survey for 2007-08. These data include latitude and longitude for schools, which may not be perfectly precise. But, they are pretty good overall.

While I’m at it – here are the Jersey City Charters – even more striking differences:

Stepping back a bit to see more charters and taking off the names for clarity, here’s what all of Newark looks like, including some other neighboring towns. Charters have a pink asterisk. Again, smaller circles in lighter shades are lower free lunch schools. There are a few charters that are moderate to higher poverty – similar to many Newark schools (bright green, medium size bubble). However, many charters are the lowest poverty schools to be found. The same is true in the second map below for Jersey City.

Recall from previous posts that Charters are even more different from their neighbors in terms of the numbers of special education and limited English proficient students they serve. Their one saving grace was that they did seem to have relatively high shares of students qualifying for free or reduced price lunch. But, as I have noted in previous posts, they seem, on average to be taking in the less poor among the poor – at least the “model charters” do.  That’s simply not scalable reform. Claims by NJ Charter advocates that these schools are serving the same, high poverty, needy student populations as other schools in their neighborhood are simply wrong – and not supported by any legitimate, fine-grained analysis (and it doesn’t even have to be that fine grained).

Note: One error in other analyses that compare charter school free or reduced lunch rates to district average rates is that those analyses fail to compare by grade level. Few charters in New Jersey are High Schools. High schools on average have lower rates of children qualifying for free/reduced lunch for a variety of reasons – primarily reporting issues. So, if you compare a bunch of elementary schools to a district average which includes high schools, you are likely to show that the elementary schools have higher average free/reduced lunch rate. But it’s not a correct comparison. Charter schools should be compared by grade level to their nearest neighboring and/or sending schools. I’ve not yet run the relevant spatial statistics above.

So, here are a few basic guidelines for future comparisons:

1) compare by relevant grade level because of the way in which subsidized lunch rates shift from elementary to secondary school;

2) while it’s okay to evaluate free and reduced shares, it is also important to slice those shares because children in these categories differ by family background. Looking only at the sum of free and reduced conceals substantial differences in student populations across charters and traditional public schools;

3) compare by location.

========

Here’s some follow-up data on New Jersey Charter School demographics. Here are the comparisons of disability rates and free lunch shares for Newark and Jersey City Public (Traditional Public) schools by grade level and the disability rates and free lunch shares for Charter schools often cited as outperforming the host district. Note that most claims that these schools outperform the host district use all kids’ test scores, not just general student test scores. So, differences in shares of children with disabilities make a huge difference in proficiency rates. I show this in previous posts on this thread (New Jersey Charter Schools).

Special Education

Note: A knowledgeable reader has informed me that the “0” value for Greater Newark Charter is actually “missing data,” for that year and has assured me that Greater Newark Charter does indeed enroll children with disabilities. At some point, I may get around to updating these analyses.

% Free Lunch

Note that the highest flyin’ charters (Treat and North Star) have substantially lower free lunch shares than the host district in addition to having very low special education rates. Only Marion P. Thomas has a relative high free lunch share, but has very few special education students.