Two buzz phrases have been somewhat quietly floating around reformyland of late, for at least a year or so. I suspect that many have not even picked up on these buzz phrases/words.  They are somewhat inner circle concepts in reformyland. The first is the notion of the great relinquisher (a seemingly bizarre contradiction indeed… to be great at surrendering… but I believe that’s the point). The second is the idea that we all must learn to be sector agnostics. That is, we all must stand behind the provision of a system of great schools as logical replacement for existing school systems and that this system of great schools might be provided by any sector – public/government, charter, private non-profit, private for profit. After all, it doesn’t matter how we provide them, as long as they are great schools. Who can argue with that?

Linking these two conceptions, the great relinquishers – primarily public officials perceived as otherwise self-interested bureaucrats – must learn to relinquish their self-interested stronghold on publicly financed schooling to alternative providers.  Among inner circle reformers, these ideas are treated as somehow ground breaking, deep intellectual thoughts about re-envisioning schooling. But in reality, they are anything but.

On Relinquishers & Sector Agnosticism

Some abbreviated backdrop on the relinquisher notion.  I converse (constructively) on occasion via e-mail with Neerav Kingsland who promotes this particular notion. For those who don’t know Neerav, he’s a Yale Law grad who completed a Broad Residency, and is currently CEO for New Schools for New Orleans. Thus, as I interpret it, he derives his core arguments largely on his perception of the (highly debatable) successes of post-Katrina New Orleans.  That in mind, and with all due respect to Neerav, I have grave concerns about what he refers to as the movement toward “relinquishment” or creating a culture of “relinquishers” among current public officials regarding the provision of the public good of schooling (differing substantively from public schooling.)

Neerav introduced the concept of Relinquishers in a letter he wrote to urban (not all, just “urban”) superintendents in Education Week:

Before I begin in full, let me say this: Superintendents, over the years I’ve begun to believe that your identities–how each of you perceives your professional charge–are often misguided. In my experience, most of you view yourselves as system reformers–leaders who can make the current educational system much better. For the sake of the letter, let’s call you, well, Reformers. With great diligence, you fight to make our government-operated system better.

But let me suggest another identity–one whose charge is to return power, in a thoughtful manner, back to parents and educators. Let’s call these types of superintendents Relinquishers. With great diligence, these superintendents attempt to transfer power away from a centralized bureaucracy.

Both Reformers and Relinquishers possess noble aims, but only one group, I think, possesses a sound strategy.

Superintendents, in the rest of this letter I hope to convince you to become Relinquishers. Specifically, I will advocate that you return power to parents and educators through the creation of charter school districts, which are the most politically acceptable mechanisms for empowering educators. (my emphasis)

Let’s start by taking the word “relinquish” literally for a moment. A quick synonym search in Microsoft Word yields: Surrender, Abandon, Renounce, Resign

The implication here is that public officials must “surrender” or “abandon” or “renounce” their schools, handing them over largely to private managers of charter schools (note that Neerav Kingsland has suggested that charter operators are the “politically acceptable” choice, leaving for others including Smarick to consider conventional private schooling and voucher models). Yeah… I get that this is an interesting notion – to suggest that there is some nobility is declaring defeat and handing control over to those who might be able to play a positive role. I get that. But I find this use, and this framing rather disturbing.

This is not to suggest that I don’t believe that many local public school districts, large or small, need work (some, a hell of a lot of work) on how they interact with their local communities and how they balance stakeholder interests (responsiveness to parents/students, etc.). That’s an ongoing concern in any public or private sector business, with differing structural/governance issues involved in public governance. This is also not to suggest that public officials should never look to other sectors for appropriately contracted, sufficiently regulated support. But “relinquishment” is an extreme perversion of this notion, especially when we start considering relinquishment of the system as a whole – Surrendering, abandoning, renouncing any and all role for public governance and centralized public policy.

Now for this notion of “sector agnosticism” – In his book The Urban School System of the Future and in several tweets and blog posts, former deputy commissioner of Education of New Jersey, Andrew Smarick promotes the reformy religion of what he refers to as Sector Agnosticism. A brief explanation is provided in a recent education week post:

Smarick: “Second, we need to have a three-sector accountability system that treats similarly district public schools, charter public schools, and private schools; we must focus on school results, not school operator. I call this “sector agnosticism;” in other words, we shouldn’t care who runs a school as long as it is superb.”

In the 1990s, when this idea arguably first gained some momentum (summarized in Paul Hill’s book Reinventing Public Education), I was actually a pretty big fan of the idea – which consisted primarily of finding ways to employ private contractors through performance contracting to improve urban schools. Heck, my own first conference paper ever was on the issue of private management of public schools, at a time when I thought there might be great hope for such strategies. Unfortunately, the self-interest of the (publicly traded, for profit) private manager (who eventually fell into financial collapse) to extract as much revenue as possible from the urban district (Baltimore) coupled with their outright disinterest in, and obstruction of having their outcomes measured, started giving me doubts. How could they possible show an efficiency advantage (doing more with less) if they managed to game their budget allocations to their advantage and then wouldn’t provide evidence of results?

Unfortunately, I wrongly assumed things would get better as the industry evolved. Further, over time, as I completed graduate work studying education finance and policy and became reasonably well versed in school law and education governance (teaching it at the graduate level for over a decade & writing/publishing numerous co-authored articles in law review journals) I became more acutely aware of the potential pitfalls of taking an uniformed leap into sector agnosticism.

Defining superbitude?

First, let’s take Smarick’s sound bite notion that it should matter as long as the school is “superb.” Even with a narrow, test score or graduation & post-secondary matriculation-based measure of “superbitude,” neither charter nor private schools are revealing any decisive edge, holding student characteristics or access to resources constant. Rather, as one might logically expect, these less regulated sectors merely produce greater variation around largely the same mean (if comparing similar students). Across sectors, the drivers of outcome variation continue to be the substantive differences in student populations served and oft correlated variations in access to schooling and non-schooling resources (in public schools, charter schools or private schools).[1]

Why do those KIPP charter middle schools appear to perform so well? What about New York City or Newark charter schools more broadly? And what about years of findings on private schools, or students participating in the New York City private school voucher experiment? It’s not about sectors, but rather about strategies and resources. And if it’s about strategies and resources, then if we can identify what works and the resources needed to legitimately serve all children, we can provide those opportunities within a publicly governed, publicly accountable system of common schools. Indeed, if these measured outcomes were in fact the only issue of concern, we might leverage an appropriately mixed set of schooling providers to get the job done. In fact, the lack of decisive advantage by sector alone is equal justification for agnosticism as it is against it.

But, that’s only if we ignore entirely that there might actually be other tradeoffs involved, beyond whatever test score, graduation, matriculation or employment outcome might be achieved.

Trading Off Legal Rights for Test Scores?

It’s not just about figuring out how to achieve crudely measured “superberific” schooling.  Our children’s schooling exists in a broader social, political and legal context. Kids have legal rights, and under most state constitutions kids a right to access/participate in/gain the benefits of a system of schooling (sometimes, quite explicitly, a system of public schooling). In many states, they not only have a right to access schooling (at times, of some measured degree of quality), but a legal obligation to attend up to a specified age (compulsory schooling laws).

As I’ve discussed on a few previous blog posts, privately governed and/or managed charter schools, more like traditional private than like public schools, may not be (are likely not) subject to the full protection of students’ constitutional or statutory rights (summary table from previous post included below). When attending a private school, it’s clear that kids have no right to continued attendance. They can be expelled, excluded outright for any number of reasons (including admissions testing). They may be compelled to recite school oaths and may be obligated to participate in religious activities and may be restricted in their ability to freely express themselves and subject to disciplinary action including expulsion for failure to comply. Parents may also be obligated to participate in certain activities as a condition of continued enrollment.

While charter advocates love to declare their schools as necessarily “public,” with regard to at least some of these same issues/questions, Charter school legal defense attorneys are quick to argue that they are in fact, private. That, for example, children’s rights under disciplinary codes should be treated as private contracts entered into by parents, just as in private schools – and substantively different from “public” schools – or those formally governed and operated by agents of the state (local elected school boards and public district administrators).

Further, the public-private delineation and murky middle ground of charter schooling raises numerous additional substantive legal questions regarding public employment law and employee rights, taxpayer and citizen rights to open public meetings and public records, and rights, responsibilities, liabilities and protections of “public officials” such as school board members and public employees as opposed to governing boards of private citizens, and employees of private contractors.

Sector agnosticism, as dreadfully simplified by Andrew Smarick requires completely ignoring these substantive tradeoffs.  Trading off constitutional rights to reduce supply of some and increase access to other sectors is not benign, if those sectors could/might possibly yield other advantages.

The Distribution of Lost Rights

Nor do I suspect that the tradeoff of rights will ever be randomly distributed across children by the wealth and income of their families. No-one is asking the superintendent of Scarsdale (great guy, by the way) to Relinquish his schools and adopt a policy of Sector Agnosticism. This is a policy for the children of New Orleans, New York City, Chicago, Philadelphia and Newark.

In the extreme case – a case favored by Smarick and seemingly endorsed (through relinquishment) by Kingsland – a district – or now merely a geographic space – where children have only access to privately governed/managed charter schools may require that any/all that wish to actually exercise their state constitutional right to attend school, have to choose which rights to forgo in the process? Will 100% of parents in that zone be required to enter into contractual agreements (forgoing constitutional & statutory protections) with schools regarding disciplinary policies for their children?

In fact, Kingsland’s logic is that district superintendents should simply succumb or surrender to the forces that wish to forcibly close and takeover their schools, and relinquish those schools or at least the children who would have attended them, to other sectors. Following Smarick’s logic, parents and citizens at large should completely ignore tradeoffs of constitutional protections, or humiliating treatment of children, in lieu of Smarickian measures of “superbification.”

Creating a scenario where only low income minority children in America’s cities must tradeoff their constitutional and statutory protections to gain access to schooling (which they may be compelled to attend) is clearly unacceptable, inequitable treatment.  Before you go there… no… I’m not saying that the responsible policy solution is to make sure that suburban kids and their parents are equally deprived of protections.  What’s not good for some is not good for all.

One logical retort to my arguments here is that if parents want these choices, if they are backed up on waiting lists for existing charters, then we should provide to them. If the demand is there, let the supply meet the demand!? It would be one thing if it was made clear, up front, to potential choosers these hidden tradeoffs, but that’s not the case. If anything, charter advocates are doing their best to conceal that any such tradeoffs exist.

Indeed, appropriate cross-sector regulation might negate some of my concerns raised here, but these issues are too frequently ignored.

Market Manipulation & The Forcible Reduction of the “Public Option”

Worse, in the current policy context, we are not witnessing the emergence of a true, fair and equitable, demand driven and fully open and accessible (driven by open information) system of choice. Policies of relinquishment and sector agnosticism are being pursued in practice as policies of forced relinquishment (read mass closings) of traditional public schooling and sector favoring transfer of assets (public to privately governed charters), coupled with gross misrepresentations of information on sector quality.

In selected cases, we are also witnessing a coordinated effort to provide competitive advantage for non-district alternatives. Where sectors are set up to compete with one another to prove their worth, the likelihood that charter or voucher advocates will lobby for increased resources for district schools is about as likely as the New York Yankee ownership arguing for revenue sharing to help the Kansas City Royals, or Walmart to lobby for tax breaks for Target. Similarly, the likelihood that well endowed charters will share their philanthropy with others less fortunate is slim to none where the emphasis remains on flaunting one’s competitive advantage.

A veneer of demand (as measured by duplicative waiting lists) for private and charter sectors has been induced by forcible reduction of supply of urban schooling, and gross misrepresentations & mismeasures (New Jersey/ New York) of neighborhood schooling quality and manipulation of the playing field.

Closing Thoughts

Before we jump on these reformy bandwagons, and start waiving the white flag of relinquishment and promoting the virtues of sector agnosticism, we need to take a hard look at how this is playing out in our cities. Numerous New Orleans schools were wiped out by a natural disaster, displacing large shares of the lowest income residents to Houston (and elsewhere), many of whom have not been able to return in part because the market based model of New Orleans has chosen not to serve their former, blighted neighborhoods.  This was tragic, and the initial occurrence largely beyond policymakers’ control.  The choice to leave children and their families unserved or underserved was a conscious policy decision (or a least a predictable result of the policy response).

Proposed Chicago (and Philadelphia) school closings would appear comparably poised to induce increased demand for charters, which will likely be used as rationale for expanding charters even further and advancing the cycle toward its ultimate end (as if Katrina by design, and more surgically targeted at schools with low test scores and poor minority children). In most U.S. cities, however charter market shares remain modest, and publicly subsidized private school enrollment even smaller, providing an opportunity to pause and rethink current strategies.

So then, what do we do about all of this? First, reformers and non-reformers alike (and anti-reformers too!) need to step back from these oversimplified talking points and buzz phrases which so illustrate the worst of intellectually lazy, undisciplined, under-informed policy development. I don’t mean to be a hypercritical, ivory tower (actually, public university 1960s era building basement) academic … okay… yeah… that is what I mean to be here. Why? Because it matters! Exploring and understanding these tradeoffs matters. Ignoring them is reckless.

Yesterday, Matt Di          Carlo over at Shankerblog put out his April fools post. The genius of the post is in its subtlety.  Matt put together a few graphs of longitudinal NAEP data showing that Maryland had made greater than average national gains on NAEP and then asserted that these gains must therefore be a function of some policy conditions that exist in Maryland. In the Post-RTTT era, Maryland has been the scorn of “reformers” because it just won’t get on board with large scale vouchers and charter expansion and has resisted follow through on test-score based teacher evaluation. Taking a poke a reformy logic, Matt asserted that perhaps the low charter share and lack of emphasis on test score based teacher evaluation… along with a dose of decent funding might be the cause of Maryland’s miracle!

Of course, these assertions are no more a stretch than commonly touted miracles in Texas in the 1990s, Florida or Washington DC, most of which are derived from making loose connections between NAEP trend data and selective discussion of preferred policies that may have concurrently existed.  The difference is that Matt was poking fun at the idea of making bold, decisive, causal inferences from such data. Such data raise interesting questions.

What I found so fun and at the same time deeply disturbing about Matt’s post is that the assertions he made in satire… were nowhere near as absurd as many of the assertions made in studies/reports, etc. I discussed here on my blog over the years. Here are but a few examples of “stuff” presented as serious/legit policy evidence, that make Matt’s satirical assertions seem completely reasonable.

The Many Variations of Money Doesn’t Matter Graphs:

I start with this one, because there are so many versions of it floating around out there, that come and go over time, and are often used to advance the “money doesn’t matter”… we’ve spent ourselves into bankruptcy and gotten nothing for it… graph. Every good reformer has a laminated copy of one version or another of this graph which they carry in wallet-size.

I blogged about this graph when Bill Gates used it in a HuffPo article.

Gates asserted:

 Over the last four decades, the per-student cost of running our K-12 schools has more than doubled, while our student achievement has remained flat, and other countries have raced ahead. The same pattern holds for higher education. Spending has climbed, but our percentage of college graduates has dropped compared to other countries… For more than 30 years, spending has risen while performance stayed flat. Now we need to raise performance without spending a lot more.

Among other things, the chart includes no international comparison, which becomes the centerpiece of the policy argument. Beyond that, the chart provides no real evidence of a lack of connection between spending and outcomes across districts within U.S. States.  Instead, the chart juxtaposes completely different measures on completely different scales to make it look like one number is rising dramatically while  the others are staying flat. This tells us NOTHING. It’s just embarrassing. Simply from a graphing standpoint, a blogger at Junk Charts noted:

Using double axes earns justified heckles but using two gridlines is a scandal!  A scatter plot is the default for this type of data. (See next section for why this particular set of data is not informative anyway.)

Not much else to say about that one. Again, had I used an example this absurd to represent reformy research and thinking, I’ d have likely faced stern criticism for mis-characterizing the rigor of reformy research!

This alternate version comes to us from none other than Andrew Coulson of Cato Institute. Coulson has a stellar record of this kind of stuff. So, what would you do to the Gates graph above if you really wanted to make your case that spending has risen dramatically and we’ve gotten no outcome improvement? First, use total rather than per pupil spending (and call it “cost”) and then stretch the scale on the vertical axis for the spending data to make it look even steeper. And then express the achievement data in percent change terms because NAEP scale scores are in the 215 to 220 range for 4th grade reading, for example, but are scaled such that even small point gains may be important/relevant but won’t even show as a blip if expressed as a percent over the base year.

Chris Cerf’s Poverty Doesn’t Matter Graph!

Now, it’s one thing when and under-informed tech CEO goes all TED-style on us with big screens, gadgets, bells and whistles and info-graphics that just don’t mean crap anyway. But, it’s yet another when a State Commissioner of Education presents something not only equally ridiculous… but arguably far more ridiculous, disingenuous, unethical and downright WRONG.

This is a graph for the ages, and it comes from a presentation by the New Jersey Commissioner of Education given at the NJASA Commissioner’s Convocation in Jackson, NJ on Feb 29. State of NJ Schools presentation 2-29-2012

The title conveys the intended point of the graph – that if you look hard enough across New Jersey – you can find not only some, but MANY higher poverty schools that perform better than lower poverty schools.

This is a bizarre graph to say the least. It’s set up as a scatter plot of proficiency rates with respect to free/reduced lunch rates, but then it only includes those schools/dots that fall in these otherwise unlikely positions. At least put the others there faintly in the background, so we can see where these fit into the overall pattern. The suggestion here is that there is not pattern.

Note: this graph may not even be the worst one in the presentation. You decide!

The apparent inference here? Either poverty itself really isn’t that important a factor in determining student success rates on state assessments, or, alternatively, free and reduced lunch simply isn’t a very good measure of poverty even if poverty is a good predictor. Either way, something’s clearly amiss if we have so many higher poverty schools outperforming lower poverty ones. In fact, the only dots included in the graph are high poverty districts outperforming lower poverty ones. There can’t be much of a pattern between these two variables at all, can there? If anything, the trendline must be sloped uphill? (that is, higher poverty leads to higher outcomes!)

Note that the graph doesn’t even tell us which or how many dots/schools are in each group and/or what percent of all schools these represent. Are they the norm? or the outliers?

Well, here’s what the pattern really looks like with all schools included:

Hmmm… looks a little different when you put it that way. Yeah, it’s a scatter, not a perfectly straight line of dots. And yes, there are some dots to the right hand side that land above the 65 line and some dots to the left that land below it.

Note: New Jersey’s Chris Cerf is not alone among state commissioners in promoting completely bogus analysis posing as empirical validation. In fact, New York’s John King presented a completely fabricated graph provided to him by a consultant to the state and has used that graph to frame his state’s policy initiatives.

Rishawn Biddle’s Graph of, well, something? What?

Not to be outdone, Rishawn Biddle who on occasion fashions himself a “researcher” on education policy issues, provides a graph that comes close to the degrees of intentional deception presented by Commissioner Cerf above.  I blogged about this graph here!

In response to arguments I had made on my blog regarding the role of substantive and sustained school finance reforms in improving school quality, Biddle argued:

Despite the arguments (and the pretty charts) of such defenders as Rutgers’ Bruce Baker, there is no evidence that spending more on American public education will lead to better results for children.

My claims are substantiated in this peer reviewed article and this separate more comprehensive report:

• Baker, B. D., & Welner, K. G. (2011). School Finance and Courts: Does Reform Matter, and How Can We Tell?. Teachers College Record, 113(11), 2374-2414.
• Baker, B. D. (2012). Revisiting the Age-Old Question: Does Money Matter in Education?. Albert Shanker Institute. http://www.shankerinstitute.org/images/doesmoneymatter_final.pdf

And what does Biddle provide as counter evidence to this – apparent lack of evidence I summarize above (I’ve sent the article link to Biddle on more than one occasion, but he apparently doesn’t read this kind of academic stuff)?

Biddle counters with a link to this graph – a true gem (I’ve added some annotation, not in his original)!

Yes, Biddle’s counter to the body of research he has not and likely will never read, is to use this graph of “promoting power” by student race group for Jersey City, NJ in 2004 and 2009. Note that the infusion of additional funds in NJ occurred mainly from 1998 to 2003, leveling off thereafter. But that’s a tangential point (not really).  So, Biddle’s absolute verification that more money doesn’t matter is to simply assert without verification that Jersey City got a whole lot more money and then to use this graph to argue that nothing improved!

First of all, that analysis wouldn’t pass muster in as a master’s degree level assignment (I teach a class on this stuff at that level), no less major research conclusions. From a graphing standpoint, I often criticize my students’ work for what I refer to as gratuitous use of 3d – especially where the use of 3d bars actually obscures the comparisons by making it hard to see where they align on the axis.

But, the really funny if not warped part of this graph is that there appear to be significant gains for black males between 2004 and 2009, but those gains are obscured by hiding the 2009 black male score behind the 2004 black female score.

Note that the graph also contains no information regarding the actual shares of the student population that fall into each group? Not very useful. Pretty damn amateur. Certainly fails to make any particular point, and certainly doesn’t refute the various citations above – all of which employ more rigorous analytic methods, apply to more than a single district, and most of which appear in rigorous peer reviewed journals.

Reason Foundation’s Today’s Policies Affected Yesterday’s Outcomes Study!

Finally, in my years as a reviewer for the National Education Policy Center’s Think Tank Review Project I’ve reviewed a lot of sketchy stuff. Some of it stands out, and has even won Bunkum awards from NEPC.

For example, a recent report from ConnCAN repeatedly footnoted a claim as being substantiated to earlier reports…only to result in a dead end where the claim was never substantiated… and in fact, when checking the data turned out to be patently false!  So, this one isn’t even a subtle data interpretation issue. It’s just a lie.

Then there was a report by the organization Third Way, which gathered numerous sources of incompatible data, across incompatible time frames (along with many other bizarre claims) in order to make the argument that America’s middle class schools are failing miserably.

Either of these reports make Matt’s assertions in his post on the Maryland Miracle look totally reasonable!

But for me, the winner among all of the think tank reports I’ve read comes from the Reason Foundation in their 2009 Weighted Student Funding Yearbook! Here’s the abstract of my review:

The new Weighted Student Formula Yearbook 2009 from the Reason Foundation provides a simple framework for touting the successes of states and urban school districts that grant greater fiscal autonomy to schools. The report defines the Weighted Student Formula (WSF) reform extremely broadly, presenting a variety of reforms under the WSF umbrella. Accordingly, when the report concludes that WSF is successful and should be widely replicated, it is difficult to sort through the claims and recommendations. Moreover, the approach and recommendations lack critical inquiry, thought, or empirical analysis. Perhaps most disturbing is the fact that in a third of the specific districts presented in the report, the evidence of success provided predates the implementation of the reforms, and the Reason press release makes the outright claim that past improvements are somehow a function of yet-to-be-implemented reforms. While the report does provide some reasonable recommendations, they are overshadowed by others. Overall, the policy guidance provided by the Reason report is reckless and irresponsible.

Yes… you read it correctly…. If you go through the smashing successes claimed by Reason in this report, in 1/3 of the cases, the reforms in question were implemented after the window of test scores discussed! Hence, the Bunkum time machine award!

Matt’s satirical example didn’t go anywhere near this far.

In Closing….

In my view, there are at least two lessons from Matt’s post, for either side of the reformy aisle.

First, as I so often point out in my classes on applied data analysis, we need to always take  time to carefully evaluate what our data – whatever data and whatever measures – can and cannot tell us. The latter is key here. Descriptive data can be very useful… as long as we understand what they can and cannot tell us. For that matter, various types of inferential statistical analyses (regression models) can also be useful (and in policy research are often primarily descriptive), but often don’t tell us what we think or would like them to tell us. I’ll likely write more about this topic in the future.

Second, we all should take time to carefully scrutinize the link between empirical evidence and policy assertions (and many should take time to take some legit graduate level research methods and statistics and measurement courses on these topics if they wish to continue to opine so boldly about policy inferences!). Perhaps most importantly we should actually take more time and put more effort into scrutinizing those reports and claims that appear most agreeable to our own predisposed beliefs/opinions.  Everyone has predisposed beliefs (especially those who pretend not to). I would argue that experienced researchers likely have stronger beliefs and opinions… and we should… precisely as a result of years of experience researching specific topics.

Oh… and a third lesson… Don’t make completely BS, false/fabricated/absurd graphs like those above. That’s just ridiculous. Are you kidding me? Hiding 3d bars? (Rishawn?) Deleting most of the cases that define the trend? (Cerf?) That’s just ridiculous! Infuriating! Sickening!

Okay, it’s not entirely surprising to find mind-boggling ignorance conveyed in the editorial pages of the New York Post. Today’s example comes to us in an Op-Ed written in response to a report released by the Alliance for Quality Education.

Usually, I’d just let it pass. It’s the Post after all. But, for two important reasons I just had to address this one.  First, the editorial was written by a member of the Governor’s Education Reform Commission.  Second, the editorial made use of our School Funding Fairness report to make its most absurd claim. And here is that claim:

Despite all of AQE’s complaints, there is no need to change the way we allocate this money, since the state already directs almost 70 percent of education funding to high-need districts. In fact, School Funding Fairness’s National Report Card gave New York a grade of “A” in its Effort category, putting us among the top five states in that category.

http://www.nypost.com/p/news/opinion/opedcolumnists/ny_schools_money_not_the_problem_qHXdkNNBLssY1Swqj8LfQK

Apparently the authors of this quote have a little difficulty reading and perhaps some problems interpreting relatively simple numbers and letter grades.

Let’s take a look at the graded indicators in our school funding fairness report:

• Effort – This measures differences in state spending for education relative to state fiscal capacity. “Effort” is defined as the ratio of state spending to state per capita gross domestic product (GDP).
• Funding Distribution – This measures the distribution of funding across local districts within a state, relative to student poverty. The measure shows whether a state provides more or less funding to schools based on their poverty concentration, using simulations ranging from 0% to 30% child poverty.

 http://www.schoolfundingfairness.org/National_Report_Card_2012.pdf

The authors use the state’s high grade on “effort” as basis for asserting that New York State has no allocation problem. Hmmm… perhaps the distribution indicator would be a better indicator for whether the state has an allocation problem. After all, even a quick synonym check in MS Word lists distribution as the first synonym for allocation (followed by provision, apportionment, sharing).

New York Received a D on this measure!

And a D is actually rather generous for New York’s distribution/allocation issues. Figure 1 provides the profile for mid-Atlantic states from our funding fairness report.

Figure 1

In other words, in New York State, higher poverty districts have systematically less per pupil state and local revenue than do lower poverty ones. And despite all of the other completely ridiculous a-contextual and otherwise wrong and misleading claims in the Op-ed, high need New York State school districts face significant financial disadvantages relative to their competitive surroundings.

Despite the Op-Ed authors claims, their Governor has done little or nothing to help these districts, and much to harm them (including misguided reforms).

Sean Corcoran of NYU and I dug deeper into New York State’s distribution issues and sources of inequity in a recent report for the Center for American Progress.

First, with updated analysis following the funding fairness methods, we identify New York State as among the least equitable states in the Nation. Here are the numbers:

And here’s a nice colored map (which may resonate with the editorial authors who’s grasp of numbers appears severely limited):

http://www.americanprogress.org/wp-content/uploads/2012/09/StealthInequities.pdf

We explain that a hypothetical, rational, equitable school funding system should look something like this:

But that New York’s system actually looks like this – when not correcting for costs/needs (it’s much, much worse when you do!).

We also then identify and explain the various sources of inequity in New York State’s school finance system, including the allocation of Tax Relief aid disproportionately to wealthier districts, and other adjustments to foundation aid that favor wealthier districts to the detriment of poorer ones. Here are some figures, and descriptions from our report:

Figure 13 puts New York’s School Tax Relief program aid allocations into context. Federal aid to schools is largely designed to improve equity by targeting resources to higher-need, especially higher-poverty, districts. The School Tax Relief pro­gram aid to New York schools tends on average to be slightly less than federal aid. But to the extent that federal aid creates any improvement to the distribu­tion of resources across New York school districts, the state’s School Tax Relief program aid wipes out that improvement entirely. Aid under the program, which is indicated by the darkest blue in Figure 13, is allocated in nearly perfectly inverse proportion to federal aid, such that when the two are stacked on the other, the cumulative effect is that districts receive about the same regardless of their wealth.

 New York’s foundation-aid formula includes a series of “if/then” steps to deter­mine whether a district should receive state aid based on its initial calculation of local fair share or based on an alternative calculation, one of which is the provision of minimum aid of $500 per pupil. Figure 17 shows the pattern of state and local sharing that would occur if foundation aid were based solely on the income-wealth index estimated by the state. Under that index, the lowest-wealth districts would receive about $12,000 to $14,000 in aid per pupil, and districts with an income-wealth index greater than 1 would receive no aid. After including the various alter­native calculations, districts with an income-wealth index above about 2.5 would receive the minimum of $500 per pupil, while districts with index rates from 1 to 2.5 would receive a sliding scale toward the minimum rather than either the mini­mum or $0. The adjusted version is shown in red. Note however, that neither was fully funded in recent years. (Recent reality is achieved by taking the red squares and shifting them downward but preserving the minimum aid.)

If fully funded, the cost of retaining the minimum aid provision tops $1.2 billion, and the cost of preserving the diagonal, sliding-scale adjustment between the income-wealth index of 1 and 2.5 is $2.47 billion (if we exclude the disproportion­ate effects of New York City). That’s real money—money that could be perhaps targeted toward higher-need districts to reduce the overall regressive nature of New York’s finance system.

The cumulative effects of these adjustments and the School Tax Relief program on the distribution of resources across New York school districts is shown above. The left hand portion of Figure 18 shows local revenue with formula aid prior to the adjustments in Figure 17. Even this isn’t a very pretty picture because state aid remains insufficient to provide even nominal funding equity from lower- to higher-poverty districts. But the right hand side of Figure 18 shows the effect of the adjustments in Figure 17, with the icing of school tax relief aid on top.

Prior to foundation-sharing adjustments and the School Tax Relief program aid, the per-pupil difference in state and local revenue per pupil between the lowest- and highest-poverty quintile is about $1,100. After the adjustments, the per-pupil difference is more than $2,300. New York makes adjustments to its aid formula and throws on tax relief funding in a pattern that more than doubles the nominal inequity between the state’s lowest- and highest-poverty districts.

In other words – YES – NEW YORK DOES NEED TO CHANGE THE WAY IT ALLOCATES MONEY. Any suggestion to the contrary displays a mind-boggling degree of ignorance!

Knowing that I’ve been writing a fair amount about various methods for attributing student achievement to their teachers, several colleagues forwarded to me the recently released standards of the Council For the Accreditation of Educator Preparation, or CAEP. Specifically, several colleagues pointed me toward Standard 4.1 Impact on Student Learning:

4.1.The provider documents, using value-added measures where available, other state-supported P-12 impact measures, and any other measures constructed by the provider, that program completers contribute to an expected level of P-12 student growth.

http://caepnet.org/commission/standards/standard4/

Now, it’s one thing when relatively under-informed pundits, think tankers, politicians and their policy advisors pitch a misguided use of statistical information for immediate policy adoption. It’s yet another when professional organizations are complicit in this misguided use. There’s just no excuse for that! (political pressure, public polling data, or otherwise)

The problems associated with attempting to derive any reasonable conclusions about teacher preparation program quality based on value-added or student growth data (of the students they teach in their first assignments) are insurmountable from a research perspective.

Worse, the perverse incentives likely induced by such a policy are far more likely to do real harm than any good, when it comes to the distribution of teacher and teaching quality across school settings within states.

First and foremost, the idea that we can draw this simple line below between preparation and practice contradicts nearly every reality of modern day teacher credentialing and progress into and through the profession:

one teacher prep institution –> one teacher –> one job in one school –> one representative group of students

The modern day teacher collects multiple credentials from multiple institutions, may switch jobs a handful of times early in his/her career and may serve a very specific type of student, unlike those taught by either peers from the same credentialing program or those from other credentialing programs. This model also relies heavily on minimal to no migration of teachers across state borders (well, either little or none, or a ton of it, so that a state would have a large enough share of teachers from specific out of state institutions to compare). I discuss these issues in earlier posts.

Setting aside that none of the oversimplified assumptions of the linear diagram above hold (a lot to ignore!), let’s probe the more geeky technical issues of trying to use VAM to evaluate ed school effectiveness.

There exist a handful of recent studies which attempt to tease out certification program effects on graduate’s student’s outcomes, most of which encounter the same problems. Here’s a look at one of the better studies on this topic.

• Mihaly, K., McCaffrey, D. F., Sass, T. R., & Lockwood, J. R. (2012). Where You Come From or Where You Go?

Specifically, this study tries to tease out the problem that arises when graduates of credentialing programs don’t sort evenly across a state. In other words, a problem that ALWAYS occurs in reality!

Researchy language tends to downplay these problems by phrasing them only in technical terms and always assuming there is some way to overcome them with statistical tweak or two. Sometimes there just isn’t and this is one of those times!

Let’s dig in. Here’s a breakdown of the abstract:

In this paper we consider the challenges and implications of controlling for school contextual bias when modeling teacher preparation program effects. Because teachers from any one preparation program are hired in more than one school and teachers are not randomly distributed across schools, failing to account for contextual factors in achievement models could bias preparation program estimates.

Okay, that’s a significant problem! Teachers from specific prep institutions are certainly not likely to end up randomly distributed across a state, are they? And if they don’t, the estimates of program effectiveness could be “biased.” That is, the estimates are wrong! Too high, or to low, due to where their grads went as opposed to how “good” they were. Okay, so what’s the best way to fix that, assuming you can’t randomly assign all of the teacher grads to similar schools/jobs?

Including school fixed effects controls for school environment by relying on differences among student outcomes within the same schools to identify the program effects.  However, the fixed effect specification may be unidentified, imprecise or biased if certain data requirements are not met.

That means, that the most legit way to compare teachers across programs is if you can compare teachers whose first placements are in the same schools, and ideally where they serve similar groups of kids. And, you’d have to have a large enough sample size at the lowest level of analysis – comparable classrooms within school – to accomplish this goal. So, the best way to compare teachers across prep programs is to have enough of them, from each and every program, in each school, teaching similar kids similar subjects at the same grade level, across grade levels. Hmmmm…. How often are we really likely to meet this data requirement?

Using statewide data from Florida, we examine whether the inclusion of school fixed effects is feasible in this setting, the sensitivity of the estimates to assumptions underlying for fixed effects, and what their inclusion implies about the precision of the preparation program estimates. We also examine whether restricting the estimation sample to inexperienced teachers and whether shortening the data window impacts the magnitude and precision of preparation program effects. Finally, we compare the ranking of preparation programs based on models with no school controls, school covariates and school fixed effects. We find that some preparation program rankings are significantly affected by the model specification. We discuss the implications of these results for policymakers.

With “no school” controls means not accounting at all for differences in the schools where grads teach. With “covariates” means correcting in the model for the measured characteristics of the kids in the schools – so – trying to compare teachers who teach in similar – by measured characteristics – schools. But, measured characteristics often fail to catch all the substantive differences between schools/classrooms.  And where “school fixed” effects means comparing graduates from different institutions who teach in the same school (though not necessarily the same types of kids!).

Okay, so the authors tested their “best” methodological alternative (comparing teachers within schools, by school “fixed” effect) with other approaches, including making no adjustment for where teachers went, or making adjustments based on the characteristics of the schools, even if not matched exactly.

The authors found that the less good alternatives were, to no surprise, less good- potentially biased. The assumption being that the fixed effect models are most correct (which doesn’t, however, guarantee that they are right!).

So, if one can only legitimately (though really not in this case either) compare teacher prep programs in cases where grads across programs are concentrated in the same schools for their first jobs, that’s a pretty severe limitation. How many job openings are there in a specific grade range in a specific school in a given year – or even over a five year period? And how likely is it that those openings can be filled with one teacher each from each teacher prep institution. But wait, really we need more than one from each to do any legit statistical comparison – and ideally we need for this pattern to be replicated over and over across several schools. In other words, the constraint imposed to achieve the “best case” model in this study is a constraint that is unlikely to ever be met for more than a handful of large teacher prep institutions concentrated in a single metropolitan area (or very large state like Florida).

Other recent studies have not found VAM particularly useful in parsing program effects:

We compare teacher preparation programs in Missouri based on the effectiveness of their graduates in the classroom. The differences in effectiveness between teachers from different preparation programs are very small. In fact, virtually all of the variation in teacher effectiveness comes from within-program differences between teachers. Prior research has overstated differences in teacher performance across preparation programs for several reasons, most notably because some sampling variability in the data has been incorrectly attributed to the preparation programs.

Koedel, C., Parsons, E., Podgursky, M., & Ehle, M. (2012). Teacher Preparation Programs and Teacher Quality: Are There Real Differences Across Programs? (No. 1204).

http://econ.missouri.edu/working-papers/2012/WP1204_koedel_et_al.pdf

Example from Kansas

Let’s use the state of Kansas and graduates over a five year period from the state’s major teacher producing institutions to see just how problematic it is to assume that teacher preparation institutions in a given state will produce sufficient numbers of teachers who teach in the same schools as graduates of other programs.

All programs

Specific programs

Indeed, the overlap in more population dense states is somewhat more significant, but still unlikely sufficient to meet the high demands of the fixed effects specification (where you can only essentially compare when you have graduates of different programs working in the same school together, in similar assignments… presumably similar number of years out of their prep programs).

Strategically Gaming Crappy, Biased Measures of “Student Growth”

In practice, I doubt most schools of ed, or state education agencies will actually consider how to best model program effectiveness with these measures. They likely won’t even bother with this technically geeky question of the fixed effects model, and data demands to apply that model. Rather, they’ll be taking existing state provided growth scores or value-added estimates and aggregating them across their graduates.

Given the varied, often poor quality of state adopted metrics, the potential for CAEP Standard 4.1 to decay into absurd gaming is quite high. In fact, I’ve got a gaming recommendation right here for teacher preparation institutions in New York State.

We know from the state’s own consultant analyzing the growth percentile data that:

Despite the model conditioning on prior year test scores, schools and teachers with students who had higher prior year test scores, on average, had higher MGPs. Teachers of classes with higher percentages of economically disadvantaged students had lower MGPs. (p. 1) https://schoolfinance101.com/wp-content/uploads/2012/11/growth-model-11-12-air-technical-report.pdf

We also know from this same technical report that the bias appears to strengthen with aggregation to the school level. It may also strengthen with aggregation across similar schools. And this is after conditioning the model on income status and disability status.

As such, it is in the accreditation interest of any New York State teacher prep institution to place as many grads as possible into lower poverty schools, especially those with fewer children with disabilities. By extension, it is therefore also in the accreditation interest of NY State teacher prep institutions to reduce the numbers of teachers they prepare in the field of special education. As it turns out, the New York State growth percentiles are also highly associated with initial scores – higher initial average scores are positively associated with higher growth. So, getting grads into relatively higher performing schools might be advantageous.

With a little statistical savvy, a few good scatteplots, one can easily mine the biases of any state’s student growth metrics to determine how to best game them in support of CAEP standard 4.1.

Further, because it is nearly if not entirely impossible to use these data to legitimately compare program effects, the best one can do is to find the most advantageous illegitimate approach.

Are these really the incentives we’re looking for?

First, here’s a primer and reading list on the Empire State of School Finance:

1. New York State maintains one of the least equitable state school finance systems in the nation
2. New York State actually allocates a ton of state aid to districts that need it least, exacerbating the disparities
3. Reformy types in New York State thought, under these circumstances, it would be really cool to make any additional state aid a district receives contingent on adopting a teacher evaluation scheme based on their documented deeply flawed metrics!
4. To ice that reformy cake, the legislature saw fit to – after slashing state aid year after year – impose a local property tax limit on districts so that they are unable to even raise the funds they would need to provide a sound basic education, if they could raise those funds locally.

ohhh… but I’m just getting started here.  Then came the lawsuits. That’s what makes this so fun and interesting to watch.

Now, there is already a pending lawsuit challenging the overall adequacy of state funding in New York specifically for high need cities (brought by the state’s small city school districts).

More recently however, we’ve been hearing of two separate cases.

First, we have the state teacher’s union (as reported) suing the state over the imposition of the property tax cap, which, in effect prohibits many districts from making up the difference from the aid they’ve been screwed out of for the past several years – the aid that in theory – by the state’s own definition of its foundation formula – would provide for a sound basic education. That formula was implemented specifically to comply with a previous court order in Campaign for Fiscal Equity. 

Next, we have the lawsuit brought on behalf of children in New York City schools challenging the state’s authority to reduce the city’s funding by $250 million for non-compliance with adopting a teacher evaluation policy.

So far, it would appear that this argument has achieved a positive, immediate response from the judge, who a this stage has blocked the state funding reduction.
As laid out in full here: http://schoolfunding.info/wp-content/uploads/2013/02/Memorandum-of-Law-in-Opp-to-App-for-Prelim-Inj1.pdfAnd as characterized here: http://schoolfunding.info/2013/02/miriam-aristy-v-state-of-new-york/

Assistant Attorney General Steven Schulman described the $250 million that the state will cut from NYC schools as a “drop in the bucket” and argued that it was not great enough to have any effect on schools’ ability to provide a sound basic education.

The state’s defense of its actions is essentially that $250 million really isn’t that much money for New York City and certainly doesn’t deprive NYC schoolchildren of receiving their constitutionally mandated sound, basic education. And that forcing the state to provide the $250 million would undermine their authority. That is, their authority to deprive kids of their constitutionally mandated sound basic education! ? ! ? huh? Now, this is all part of legal maneuvering. Yeah… it would be difficult for NYC to show that holding back this additional 3.3% state funding tips the scales on whether the city can provide a sound basic education. As such, how can the court reason intervening and forcing the state to give this money back?
But that’s only if we set aside that the state of New York is already depriving New York City schoolchildren of 38% of the aid that should be allocated to the city based on the state’s own formula for what the city needs to provide a sound basic education. And that was a bogus, low-balled estimate to begin with. Here’s my quick run down on state aid shortfalls in 2012-13 – with respect to the state’s own estimates – for small city districts and for New York City:

Foundation Aid, Foundation after GEA expressed in Thousands (‘000s)New York City is being shorted about 3.4 billion in aid to achieve what the state has defined as “sound basic” funding. That’s about 38% of their total foundation aid.  That share is even larger for some small city districts. This next table shows that this amounts to thousands per pupil.  Sure, the loss from the teacher evaluation debacle amounts to a few hundred per pupil. But hey, what’s the harm? NYC is already being shorted over $3,000 per pupil.By the state’s logic, we, and the sitting judge are asked to ignore that the bucket into which that drop is to fall (or not) is nearly half empty (no, not half full… well… actually… about 62% full) to begin with.  That the murder who stabs to death a victim one day, and comes back to stab the already dead body one more time the next, is not guilty of any marginal crime for his actions on the second day.Perhaps not in the final legal analysis. I’ll leave that for the judge to figure out. But in my view, it’s still pretty damned offensive.