The following is a hypothetical case I am using as the culminating activity in Public School Law this semester.

The Dismissal of Principal X

Principal X is principal in a local public middle school in a state that has recently adopted through legislation, articulated with greater precision in state department of education regulations, a new teacher evaluation scheme. The teacher evaluation laws and regulations now require that:

• Any teacher who receives two sequential evaluations less than “satisfactory” shall have his/her tenure status revoked;
• Teacher evaluations shall consist of 40 to 50% measures of student growth, where the majority shall be based on state provided metrics.
• By regulatory decree of the State Commissioner of Education, any other measures selected by local district officials for inclusion in evaluations must be proven correlated with state approved and provided measures of student achievement growth.

Further, the state now conditions receipt of “any and all increases to state aid for local public school districts” on full compliance with statutes and regulations pertaining to teacher evaluation.

On September 20^th of 2013, Principal X was provided with growth percentile data on her teachers from the prior year. Of the approximately 40 certified staff in her school, 8 received growth percentile data, two of whom achieved unsatisfactory growth percentile estimates for their students, one of whom received a second unsatisfactory rating in a row ‐Teacher Y.

In keeping with the requirement that any and all other measures used in the state approved teacher evaluations be correlated with the growth percentile measures, the principal was compelled to assign this teacher a second unsatisfactory rating, and thus compelled to revoke the tenure status of Teacher Y. She was a 10 year veteran teacher perceived by the principal and many others in the school to be one of the school’s most valuable human resources. In fact, over the past several years, the principal had relied on this teacher to take the difficult students including playing a more significant role than others in inclusion of children with disabilities in her classroom – and the teacher not only willingly, but eagerly complied.

Frustrated with the outcome of the new state teacher evaluation laws, Principal X took her case to the public and to state officials simultaneously. Without specific reference to the case in question – but via stylized example – the principal used the case of Teacher Y to illustrate how strict requirements of job action based largely on limited and problematic measures could lead to damaging decisions – decisions

1| Page

that she argued were neither in the best interest of the teachers nor the children they served, and decisions likely to negatively affect the quality of education statewide.

The principal made the case for returning discretion on issues of teacher evaluation and human resource management to local officials, including school principals. The principal’s letter led to a sympathetic uprising from community members and parents, who were quick to catch on as to which teacher was actually the basis of the principal’s hypothetical. Parents of that teacher’s students were outraged, and expressed their outrage at local board of education meetings. During this time, the local board of education maintained quiet support of the principal.

The principal had also begun to engage other principals statewide establishing a network of principals publicly proclaiming their opposition to newly adopted state teacher evaluation statutes and regulations. A web site was created, a non‐profit organization (political action organization) was formed, and the original letter of opposition to new state policies posted on the site, along with a petition for other school principals to show support for the group’s cause and/or become an official member.

State officials were less supportive and unamused by this principal’s apparent disrespect for their authority, and her “willful disobedience of existing statutes and regulations” expressed by Principal X’s stalling on submitting relevant evaluation information necessary for revocation of Teacher Y’s tenure status. Further, state officials were less than thrilled with the mounting insurrection initiated by the publicly posted letter to state officials outlining problems with the state teacher evaluation laws.

State officials released a letter to the local board of education indicating that their state aid would be frozen for the coming school year if, in fact, their rogue principal continued to stall and refuse compliance with the teacher evaluation laws. Under pressure from the board, Principal X agreed to initiate procedures that would lead to tenure revocation for Teacher Y. Instead of waiting out this process, Teacher Y chose to resign and pursue employment elsewhere.

But with mounting pressure on the local board of education from state department officials to control the growing movement among principals statewide against the teacher evaluation laws, a movement initiated by one of their most respected principals (who had received only glowing evaluations in prior years), the district board chose to dismiss Principal X, citing that the principal’s activities had distracted her from doing the job required, substantively compromised her effectiveness as a principal and significantly interfered with the ability of district officials to efficiently and effectively carry on district operations (including the uncertainty created over the district’s future state aid receipts).

Principal X is now suing the district for wrongful dismissal, arguing that the district’s dismissal is in violation of her first amendment right to express herself to the public on issues of public interest, for which she, as an informed public school employee has relevant information.

2| Page

Required Reading

Key Cases

Pickering v. Board of Education: http://www.oyez.org/cases/1960‐1969/1967/1967_510

Connick v. Myers: http://www.oyez.org/cases/1980‐1989/1982/1982_81_1251

Garcetti v. Ceballos: http://www.oyez.org/cases/2000‐2009/2005/2005_04_473

Blogs

EdJurist: Garcetti & Schools http://www.edjurist.com/garcetti‐and‐schools

EdJurist: Academic Blogging & Garcetti: http://www.edjurist.com/blog/2008/5/9/academic‐freedom­garcetti‐blogging.html

Law Reviews

Oluwole, J. O. (2007). On the Road to Garcetti: Unpick’erring Pickering and Its Progeny. Cap. UL Rev., 36, 967.

3| Page

Behavioral economics is an interesting and potentially useful field of academic inquiry. At its best, real behavioral economics attempts to address some of the concerns I raise here. But many if not most assumptions about human behavior and response to incentives are not representative of behavioral economics at its best.

Specifically,  I’m increasingly concerned with what I see as the simple-minded projection of economic thinking onto everyone and anyone else, leading to ridiculous policy recommendations – that amazingly – get taken seriously – at least by the media and punditocracy.

See, for example, Roland Fryer’s experiment on loss aversion as a strategy for incenting teachers to make sure that their students gain a few extra test score points in limited content areas. Indeed, if we pay you up front, and threaten to take your salary away if you don’t get those test score points out those kids, the data suggest  a greater likelihood of squeezing the kids for a few more points. Whether that tells us anything about the motivation and morals of teachers, or of the economists framing this argument is an entirely different question. This tells us little or nothing of the appropriate policy response. Thankfully, the policy implications of this paper were sufficiently absurd that they gained little traction.

Let’s assume classic economic assumptions about human behavior really hold steadfast and can be grossly simplified to an anything for an extra buck, or not to lose one, position. I would argue that it is perhaps economists themselves that are most stereotypical in this regard –  at least as represented in the thinking the project onto others.  In fact, I would argue that many, born out of a culture that self-selects into economic professions, are simply going out of their way to project their own thinking on others.

Further, many of these economists operate in a world where they can influence/control public policy and they too have an incentive in how they behave in this system. They are not impartial observers by any stretch of the imagination. Their goal is to use their economic research to shape public policy to their own advantage.

Put simply, just because the average morally bankrupt economist might do pretty much anything for an extra buck (or a billion), doesn’t mean the average teacher, doctor, nurse, fireman or police officer would!

This issue has been on my mind for some time, but recently came to a head when I read this completely ridiculous Washington Post article on health care policy – specifically – how to remove the incentive for hospitals and physicians resulting from surgical complications.

I should note, I come from a medical family, so some of my arguments herein are drawn from dinner table conversations (across generations), coupled with my tendency to read health policy research out of personal interest in exploring connections with education policy.

It was implied in the WaPo article… well… actually it was explicitly stated in the article that hospitals and physicians have a big financial incentive for their patients to have serious complications, leading to extended hospital stays and additional procedures.

Now, the average economist might be so morally bankrupt such that if he/she were in an operating room (OR) considering the implications of complications relative to potential earnings that they might intentionally introduce infection or other complication, but thankfully the average economist is not in the OR. Thankfully, they self-selected into economics and not medicine (likely foreseeing greater opportunity to earn more for much less work and upfront investment).

The WaPo article does make the following statement, to head off this argument:

The study does not imply that hospitals intentionally complicate surgeries to bring in more revenue.

But, I would argue that this is actually a rather half-hearted disclaimer (to a half-assed argument) for an article that very much implies just that.

Certainly the economists’ policy response – how to employ crude economic assumptions of human behavior to fix this dreadful perverse incentive – implies that cutting off this financial benefit for malpractice would improve hospital and physician behavior [meanwhile conflating the hospital and physician incentives & roles in the various related processes]. Here is the policy solution recommended by the economists cited in the WaPo article:

If hospitals receive a set amount for every heart surgery they perform, for example, they suddenly have an incentive to reduce complications — they know the extra medical spending will come out of their own budget.

Lost in the economists’ reasoning here are a) the potential longer term financial and career implications to the physician repeatedly entangled in litigation over post-surgical complications, b) and the stress/mental toll on the physician arising from managing complications in tense moments in the OR.

Indeed this is anecdotal, but I’ve not met a physician – surgeon or anesthesiologist – who prefers a day when things go bad in the OR – or would be likely to see dollar signs in those moments of stress. What kind of sick bastard even thinks that way? Well, perhaps the average economist does.

Economists rarely – uh… NEVER face comparable professional stress to managing a patient’s life on the edge – even when they make a massively stupid spreadsheet error stimulating economic turmoil across the globe. Nor do they pay hefty malpractice premiums to shield themselves from such egregious malpractice (despite measurable financial damages). I would assert that the economist never faces the stress of having to care for a classroom of 20 to 40, 5 to 15 year old kids, whose immediate safety and well-being, as well as their long term futures is on the line.  This is in part, why they get away with such ludicrous thinking.

It’s all freakin’ game (Freakin’ used here in a technical freakonomics sense)… a game of playing with big data – several layers removed from reality – from people – from real human consequences.

Perhaps that’s the central issue… even more so than economists’ financial self-interests?

Taken in perspective, it’s a fun game and a pretty cushy lifestyle to have opportunity to ponder policy implications of big data, as long as we don’t start thinking that what we do is so freakin’ important and indispensable and as long as we understand where we sit in this big messy puzzle of human behavior and incentives.

Of course, the other interesting piece here is the leap we often see these days between what the study behind the headlines actually said, and the resulting spin in the media headlines. We also often see the economists themselves engaging in the spin. This was equally true in the famed Chetty, Rockoff, Friedman Fire Teachers First, Ask Questions Later study.

For example, here’s what the original study – in the Journal of the American Medical Association – on reimbursements associated with complications actually said:

Depending on payer mix, many hospitals have the potential for adverse near-term financial consequences for decreasing postsurgical complications.

It takes one hell of a leap of logic to get from this measured finding to the policy recommendation above.  It takes projecting economists thinking –amoral greed – onto all actors involved. It also takes ignoring entirely a multitude of contextual factors and perverse consequences (economist thinking – first, we assume none of that stuff exists). Indeed, many complications relate to preexisting conditions and/or overall health of the incoming patient. Do we really want to incent risk aversion? (avoiding those far more likely to have complications?). Well, if it leads to lower premiums for and taxes paid by economists, then perhaps?

Tangentially (or not?), there is an equally ill-conceived movement afoot to apply to healthcare management the brilliance of what we have supposedly learned from measuring teacher effectiveness with value-added models, as explained in this policy brief from Mathematica. Notably, I tend to think Mathematica does pretty good work on education policy (better than most. See here, but for more critical perspective, see here) Put in its best light, this policy brief is merely Mathematica researchers engaging in another I’ve got a Hammer… where’s the freakin’ nail exercise.

Put in the light of economic thinking about human behavior – which many economists prefer to project on all others, the incentive here is for Mathematica to broaden its market, gaining contracts to develop value-added metrics for health care systems and for state and Federal government – to ultimately be used in reducing payments for healthcare, and reducing the tax burden and healthcare premiums paid by Mathematica researchers – their funders and their peers. It’s a win/win. More contracts and higher income, and lower taxes and health benefits expenses (not costs, but expenses*).

That is, as long as they are never in need of surgery.

====

*Cost  reduction implies that quality of service remains constant, whereas expenditure reduction may lead to service quality reduction.

Two buzz phrases have been somewhat quietly floating around reformyland of late, for at least a year or so. I suspect that many have not even picked up on these buzz phrases/words.  They are somewhat inner circle concepts in reformyland. The first is the notion of the great relinquisher (a seemingly bizarre contradiction indeed… to be great at surrendering… but I believe that’s the point). The second is the idea that we all must learn to be sector agnostics. That is, we all must stand behind the provision of a system of great schools as logical replacement for existing school systems and that this system of great schools might be provided by any sector – public/government, charter, private non-profit, private for profit. After all, it doesn’t matter how we provide them, as long as they are great schools. Who can argue with that?

Linking these two conceptions, the great relinquishers – primarily public officials perceived as otherwise self-interested bureaucrats – must learn to relinquish their self-interested stronghold on publicly financed schooling to alternative providers.  Among inner circle reformers, these ideas are treated as somehow ground breaking, deep intellectual thoughts about re-envisioning schooling. But in reality, they are anything but.

On Relinquishers & Sector Agnosticism

Some abbreviated backdrop on the relinquisher notion.  I converse (constructively) on occasion via e-mail with Neerav Kingsland who promotes this particular notion. For those who don’t know Neerav, he’s a Yale Law grad who completed a Broad Residency, and is currently CEO for New Schools for New Orleans. Thus, as I interpret it, he derives his core arguments largely on his perception of the (highly debatable) successes of post-Katrina New Orleans.  That in mind, and with all due respect to Neerav, I have grave concerns about what he refers to as the movement toward “relinquishment” or creating a culture of “relinquishers” among current public officials regarding the provision of the public good of schooling (differing substantively from public schooling.)

Neerav introduced the concept of Relinquishers in a letter he wrote to urban (not all, just “urban”) superintendents in Education Week:

Before I begin in full, let me say this: Superintendents, over the years I’ve begun to believe that your identities–how each of you perceives your professional charge–are often misguided. In my experience, most of you view yourselves as system reformers–leaders who can make the current educational system much better. For the sake of the letter, let’s call you, well, Reformers. With great diligence, you fight to make our government-operated system better.

But let me suggest another identity–one whose charge is to return power, in a thoughtful manner, back to parents and educators. Let’s call these types of superintendents Relinquishers. With great diligence, these superintendents attempt to transfer power away from a centralized bureaucracy.

Both Reformers and Relinquishers possess noble aims, but only one group, I think, possesses a sound strategy.

Superintendents, in the rest of this letter I hope to convince you to become Relinquishers. Specifically, I will advocate that you return power to parents and educators through the creation of charter school districts, which are the most politically acceptable mechanisms for empowering educators. (my emphasis)

Let’s start by taking the word “relinquish” literally for a moment. A quick synonym search in Microsoft Word yields: Surrender, Abandon, Renounce, Resign

The implication here is that public officials must “surrender” or “abandon” or “renounce” their schools, handing them over largely to private managers of charter schools (note that Neerav Kingsland has suggested that charter operators are the “politically acceptable” choice, leaving for others including Smarick to consider conventional private schooling and voucher models). Yeah… I get that this is an interesting notion – to suggest that there is some nobility is declaring defeat and handing control over to those who might be able to play a positive role. I get that. But I find this use, and this framing rather disturbing.

This is not to suggest that I don’t believe that many local public school districts, large or small, need work (some, a hell of a lot of work) on how they interact with their local communities and how they balance stakeholder interests (responsiveness to parents/students, etc.). That’s an ongoing concern in any public or private sector business, with differing structural/governance issues involved in public governance. This is also not to suggest that public officials should never look to other sectors for appropriately contracted, sufficiently regulated support. But “relinquishment” is an extreme perversion of this notion, especially when we start considering relinquishment of the system as a whole – Surrendering, abandoning, renouncing any and all role for public governance and centralized public policy.

Now for this notion of “sector agnosticism” – In his book The Urban School System of the Future and in several tweets and blog posts, former deputy commissioner of Education of New Jersey, Andrew Smarick promotes the reformy religion of what he refers to as Sector Agnosticism. A brief explanation is provided in a recent education week post:

Smarick: “Second, we need to have a three-sector accountability system that treats similarly district public schools, charter public schools, and private schools; we must focus on school results, not school operator. I call this “sector agnosticism;” in other words, we shouldn’t care who runs a school as long as it is superb.”

In the 1990s, when this idea arguably first gained some momentum (summarized in Paul Hill’s book Reinventing Public Education), I was actually a pretty big fan of the idea – which consisted primarily of finding ways to employ private contractors through performance contracting to improve urban schools. Heck, my own first conference paper ever was on the issue of private management of public schools, at a time when I thought there might be great hope for such strategies. Unfortunately, the self-interest of the (publicly traded, for profit) private manager (who eventually fell into financial collapse) to extract as much revenue as possible from the urban district (Baltimore) coupled with their outright disinterest in, and obstruction of having their outcomes measured, started giving me doubts. How could they possible show an efficiency advantage (doing more with less) if they managed to game their budget allocations to their advantage and then wouldn’t provide evidence of results?

Unfortunately, I wrongly assumed things would get better as the industry evolved. Further, over time, as I completed graduate work studying education finance and policy and became reasonably well versed in school law and education governance (teaching it at the graduate level for over a decade & writing/publishing numerous co-authored articles in law review journals) I became more acutely aware of the potential pitfalls of taking an uniformed leap into sector agnosticism.

Defining superbitude?

First, let’s take Smarick’s sound bite notion that it should matter as long as the school is “superb.” Even with a narrow, test score or graduation & post-secondary matriculation-based measure of “superbitude,” neither charter nor private schools are revealing any decisive edge, holding student characteristics or access to resources constant. Rather, as one might logically expect, these less regulated sectors merely produce greater variation around largely the same mean (if comparing similar students). Across sectors, the drivers of outcome variation continue to be the substantive differences in student populations served and oft correlated variations in access to schooling and non-schooling resources (in public schools, charter schools or private schools).[1]

Why do those KIPP charter middle schools appear to perform so well? What about New York City or Newark charter schools more broadly? And what about years of findings on private schools, or students participating in the New York City private school voucher experiment? It’s not about sectors, but rather about strategies and resources. And if it’s about strategies and resources, then if we can identify what works and the resources needed to legitimately serve all children, we can provide those opportunities within a publicly governed, publicly accountable system of common schools. Indeed, if these measured outcomes were in fact the only issue of concern, we might leverage an appropriately mixed set of schooling providers to get the job done. In fact, the lack of decisive advantage by sector alone is equal justification for agnosticism as it is against it.

But, that’s only if we ignore entirely that there might actually be other tradeoffs involved, beyond whatever test score, graduation, matriculation or employment outcome might be achieved.

Trading Off Legal Rights for Test Scores?

It’s not just about figuring out how to achieve crudely measured “superberific” schooling.  Our children’s schooling exists in a broader social, political and legal context. Kids have legal rights, and under most state constitutions kids a right to access/participate in/gain the benefits of a system of schooling (sometimes, quite explicitly, a system of public schooling). In many states, they not only have a right to access schooling (at times, of some measured degree of quality), but a legal obligation to attend up to a specified age (compulsory schooling laws).

As I’ve discussed on a few previous blog posts, privately governed and/or managed charter schools, more like traditional private than like public schools, may not be (are likely not) subject to the full protection of students’ constitutional or statutory rights (summary table from previous post included below). When attending a private school, it’s clear that kids have no right to continued attendance. They can be expelled, excluded outright for any number of reasons (including admissions testing). They may be compelled to recite school oaths and may be obligated to participate in religious activities and may be restricted in their ability to freely express themselves and subject to disciplinary action including expulsion for failure to comply. Parents may also be obligated to participate in certain activities as a condition of continued enrollment.

While charter advocates love to declare their schools as necessarily “public,” with regard to at least some of these same issues/questions, Charter school legal defense attorneys are quick to argue that they are in fact, private. That, for example, children’s rights under disciplinary codes should be treated as private contracts entered into by parents, just as in private schools – and substantively different from “public” schools – or those formally governed and operated by agents of the state (local elected school boards and public district administrators).

Further, the public-private delineation and murky middle ground of charter schooling raises numerous additional substantive legal questions regarding public employment law and employee rights, taxpayer and citizen rights to open public meetings and public records, and rights, responsibilities, liabilities and protections of “public officials” such as school board members and public employees as opposed to governing boards of private citizens, and employees of private contractors.

Sector agnosticism, as dreadfully simplified by Andrew Smarick requires completely ignoring these substantive tradeoffs.  Trading off constitutional rights to reduce supply of some and increase access to other sectors is not benign, if those sectors could/might possibly yield other advantages.

The Distribution of Lost Rights

Nor do I suspect that the tradeoff of rights will ever be randomly distributed across children by the wealth and income of their families. No-one is asking the superintendent of Scarsdale (great guy, by the way) to Relinquish his schools and adopt a policy of Sector Agnosticism. This is a policy for the children of New Orleans, New York City, Chicago, Philadelphia and Newark.

In the extreme case – a case favored by Smarick and seemingly endorsed (through relinquishment) by Kingsland – a district – or now merely a geographic space – where children have only access to privately governed/managed charter schools may require that any/all that wish to actually exercise their state constitutional right to attend school, have to choose which rights to forgo in the process? Will 100% of parents in that zone be required to enter into contractual agreements (forgoing constitutional & statutory protections) with schools regarding disciplinary policies for their children?

In fact, Kingsland’s logic is that district superintendents should simply succumb or surrender to the forces that wish to forcibly close and takeover their schools, and relinquish those schools or at least the children who would have attended them, to other sectors. Following Smarick’s logic, parents and citizens at large should completely ignore tradeoffs of constitutional protections, or humiliating treatment of children, in lieu of Smarickian measures of “superbification.”

Creating a scenario where only low income minority children in America’s cities must tradeoff their constitutional and statutory protections to gain access to schooling (which they may be compelled to attend) is clearly unacceptable, inequitable treatment.  Before you go there… no… I’m not saying that the responsible policy solution is to make sure that suburban kids and their parents are equally deprived of protections.  What’s not good for some is not good for all.

One logical retort to my arguments here is that if parents want these choices, if they are backed up on waiting lists for existing charters, then we should provide to them. If the demand is there, let the supply meet the demand!? It would be one thing if it was made clear, up front, to potential choosers these hidden tradeoffs, but that’s not the case. If anything, charter advocates are doing their best to conceal that any such tradeoffs exist.

Indeed, appropriate cross-sector regulation might negate some of my concerns raised here, but these issues are too frequently ignored.

Market Manipulation & The Forcible Reduction of the “Public Option”

Worse, in the current policy context, we are not witnessing the emergence of a true, fair and equitable, demand driven and fully open and accessible (driven by open information) system of choice. Policies of relinquishment and sector agnosticism are being pursued in practice as policies of forced relinquishment (read mass closings) of traditional public schooling and sector favoring transfer of assets (public to privately governed charters), coupled with gross misrepresentations of information on sector quality.

In selected cases, we are also witnessing a coordinated effort to provide competitive advantage for non-district alternatives. Where sectors are set up to compete with one another to prove their worth, the likelihood that charter or voucher advocates will lobby for increased resources for district schools is about as likely as the New York Yankee ownership arguing for revenue sharing to help the Kansas City Royals, or Walmart to lobby for tax breaks for Target. Similarly, the likelihood that well endowed charters will share their philanthropy with others less fortunate is slim to none where the emphasis remains on flaunting one’s competitive advantage.

A veneer of demand (as measured by duplicative waiting lists) for private and charter sectors has been induced by forcible reduction of supply of urban schooling, and gross misrepresentations & mismeasures (New Jersey/ New York) of neighborhood schooling quality and manipulation of the playing field.

Closing Thoughts

Before we jump on these reformy bandwagons, and start waiving the white flag of relinquishment and promoting the virtues of sector agnosticism, we need to take a hard look at how this is playing out in our cities. Numerous New Orleans schools were wiped out by a natural disaster, displacing large shares of the lowest income residents to Houston (and elsewhere), many of whom have not been able to return in part because the market based model of New Orleans has chosen not to serve their former, blighted neighborhoods.  This was tragic, and the initial occurrence largely beyond policymakers’ control.  The choice to leave children and their families unserved or underserved was a conscious policy decision (or a least a predictable result of the policy response).

Proposed Chicago (and Philadelphia) school closings would appear comparably poised to induce increased demand for charters, which will likely be used as rationale for expanding charters even further and advancing the cycle toward its ultimate end (as if Katrina by design, and more surgically targeted at schools with low test scores and poor minority children). In most U.S. cities, however charter market shares remain modest, and publicly subsidized private school enrollment even smaller, providing an opportunity to pause and rethink current strategies.

So then, what do we do about all of this? First, reformers and non-reformers alike (and anti-reformers too!) need to step back from these oversimplified talking points and buzz phrases which so illustrate the worst of intellectually lazy, undisciplined, under-informed policy development. I don’t mean to be a hypercritical, ivory tower (actually, public university 1960s era building basement) academic … okay… yeah… that is what I mean to be here. Why? Because it matters! Exploring and understanding these tradeoffs matters. Ignoring them is reckless.

Yesterday, Matt Di          Carlo over at Shankerblog put out his April fools post. The genius of the post is in its subtlety.  Matt put together a few graphs of longitudinal NAEP data showing that Maryland had made greater than average national gains on NAEP and then asserted that these gains must therefore be a function of some policy conditions that exist in Maryland. In the Post-RTTT era, Maryland has been the scorn of “reformers” because it just won’t get on board with large scale vouchers and charter expansion and has resisted follow through on test-score based teacher evaluation. Taking a poke a reformy logic, Matt asserted that perhaps the low charter share and lack of emphasis on test score based teacher evaluation… along with a dose of decent funding might be the cause of Maryland’s miracle!

Of course, these assertions are no more a stretch than commonly touted miracles in Texas in the 1990s, Florida or Washington DC, most of which are derived from making loose connections between NAEP trend data and selective discussion of preferred policies that may have concurrently existed.  The difference is that Matt was poking fun at the idea of making bold, decisive, causal inferences from such data. Such data raise interesting questions.

What I found so fun and at the same time deeply disturbing about Matt’s post is that the assertions he made in satire… were nowhere near as absurd as many of the assertions made in studies/reports, etc. I discussed here on my blog over the years. Here are but a few examples of “stuff” presented as serious/legit policy evidence, that make Matt’s satirical assertions seem completely reasonable.

The Many Variations of Money Doesn’t Matter Graphs:

I start with this one, because there are so many versions of it floating around out there, that come and go over time, and are often used to advance the “money doesn’t matter”… we’ve spent ourselves into bankruptcy and gotten nothing for it… graph. Every good reformer has a laminated copy of one version or another of this graph which they carry in wallet-size.

I blogged about this graph when Bill Gates used it in a HuffPo article.

Gates asserted:

 Over the last four decades, the per-student cost of running our K-12 schools has more than doubled, while our student achievement has remained flat, and other countries have raced ahead. The same pattern holds for higher education. Spending has climbed, but our percentage of college graduates has dropped compared to other countries… For more than 30 years, spending has risen while performance stayed flat. Now we need to raise performance without spending a lot more.

Among other things, the chart includes no international comparison, which becomes the centerpiece of the policy argument. Beyond that, the chart provides no real evidence of a lack of connection between spending and outcomes across districts within U.S. States.  Instead, the chart juxtaposes completely different measures on completely different scales to make it look like one number is rising dramatically while  the others are staying flat. This tells us NOTHING. It’s just embarrassing. Simply from a graphing standpoint, a blogger at Junk Charts noted:

Using double axes earns justified heckles but using two gridlines is a scandal!  A scatter plot is the default for this type of data. (See next section for why this particular set of data is not informative anyway.)

Not much else to say about that one. Again, had I used an example this absurd to represent reformy research and thinking, I’ d have likely faced stern criticism for mis-characterizing the rigor of reformy research!

This alternate version comes to us from none other than Andrew Coulson of Cato Institute. Coulson has a stellar record of this kind of stuff. So, what would you do to the Gates graph above if you really wanted to make your case that spending has risen dramatically and we’ve gotten no outcome improvement? First, use total rather than per pupil spending (and call it “cost”) and then stretch the scale on the vertical axis for the spending data to make it look even steeper. And then express the achievement data in percent change terms because NAEP scale scores are in the 215 to 220 range for 4th grade reading, for example, but are scaled such that even small point gains may be important/relevant but won’t even show as a blip if expressed as a percent over the base year.

Chris Cerf’s Poverty Doesn’t Matter Graph!

Now, it’s one thing when and under-informed tech CEO goes all TED-style on us with big screens, gadgets, bells and whistles and info-graphics that just don’t mean crap anyway. But, it’s yet another when a State Commissioner of Education presents something not only equally ridiculous… but arguably far more ridiculous, disingenuous, unethical and downright WRONG.

This is a graph for the ages, and it comes from a presentation by the New Jersey Commissioner of Education given at the NJASA Commissioner’s Convocation in Jackson, NJ on Feb 29. State of NJ Schools presentation 2-29-2012

The title conveys the intended point of the graph – that if you look hard enough across New Jersey – you can find not only some, but MANY higher poverty schools that perform better than lower poverty schools.

This is a bizarre graph to say the least. It’s set up as a scatter plot of proficiency rates with respect to free/reduced lunch rates, but then it only includes those schools/dots that fall in these otherwise unlikely positions. At least put the others there faintly in the background, so we can see where these fit into the overall pattern. The suggestion here is that there is not pattern.

Note: this graph may not even be the worst one in the presentation. You decide!

The apparent inference here? Either poverty itself really isn’t that important a factor in determining student success rates on state assessments, or, alternatively, free and reduced lunch simply isn’t a very good measure of poverty even if poverty is a good predictor. Either way, something’s clearly amiss if we have so many higher poverty schools outperforming lower poverty ones. In fact, the only dots included in the graph are high poverty districts outperforming lower poverty ones. There can’t be much of a pattern between these two variables at all, can there? If anything, the trendline must be sloped uphill? (that is, higher poverty leads to higher outcomes!)

Note that the graph doesn’t even tell us which or how many dots/schools are in each group and/or what percent of all schools these represent. Are they the norm? or the outliers?

Well, here’s what the pattern really looks like with all schools included:

Hmmm… looks a little different when you put it that way. Yeah, it’s a scatter, not a perfectly straight line of dots. And yes, there are some dots to the right hand side that land above the 65 line and some dots to the left that land below it.

Note: New Jersey’s Chris Cerf is not alone among state commissioners in promoting completely bogus analysis posing as empirical validation. In fact, New York’s John King presented a completely fabricated graph provided to him by a consultant to the state and has used that graph to frame his state’s policy initiatives.

Rishawn Biddle’s Graph of, well, something? What?

Not to be outdone, Rishawn Biddle who on occasion fashions himself a “researcher” on education policy issues, provides a graph that comes close to the degrees of intentional deception presented by Commissioner Cerf above.  I blogged about this graph here!

In response to arguments I had made on my blog regarding the role of substantive and sustained school finance reforms in improving school quality, Biddle argued:

Despite the arguments (and the pretty charts) of such defenders as Rutgers’ Bruce Baker, there is no evidence that spending more on American public education will lead to better results for children.

My claims are substantiated in this peer reviewed article and this separate more comprehensive report:

• Baker, B. D., & Welner, K. G. (2011). School Finance and Courts: Does Reform Matter, and How Can We Tell?. Teachers College Record, 113(11), 2374-2414.
• Baker, B. D. (2012). Revisiting the Age-Old Question: Does Money Matter in Education?. Albert Shanker Institute. http://www.shankerinstitute.org/images/doesmoneymatter_final.pdf

And what does Biddle provide as counter evidence to this – apparent lack of evidence I summarize above (I’ve sent the article link to Biddle on more than one occasion, but he apparently doesn’t read this kind of academic stuff)?

Biddle counters with a link to this graph – a true gem (I’ve added some annotation, not in his original)!

Yes, Biddle’s counter to the body of research he has not and likely will never read, is to use this graph of “promoting power” by student race group for Jersey City, NJ in 2004 and 2009. Note that the infusion of additional funds in NJ occurred mainly from 1998 to 2003, leveling off thereafter. But that’s a tangential point (not really).  So, Biddle’s absolute verification that more money doesn’t matter is to simply assert without verification that Jersey City got a whole lot more money and then to use this graph to argue that nothing improved!

First of all, that analysis wouldn’t pass muster in as a master’s degree level assignment (I teach a class on this stuff at that level), no less major research conclusions. From a graphing standpoint, I often criticize my students’ work for what I refer to as gratuitous use of 3d – especially where the use of 3d bars actually obscures the comparisons by making it hard to see where they align on the axis.

But, the really funny if not warped part of this graph is that there appear to be significant gains for black males between 2004 and 2009, but those gains are obscured by hiding the 2009 black male score behind the 2004 black female score.

Note that the graph also contains no information regarding the actual shares of the student population that fall into each group? Not very useful. Pretty damn amateur. Certainly fails to make any particular point, and certainly doesn’t refute the various citations above – all of which employ more rigorous analytic methods, apply to more than a single district, and most of which appear in rigorous peer reviewed journals.

Reason Foundation’s Today’s Policies Affected Yesterday’s Outcomes Study!

Finally, in my years as a reviewer for the National Education Policy Center’s Think Tank Review Project I’ve reviewed a lot of sketchy stuff. Some of it stands out, and has even won Bunkum awards from NEPC.

For example, a recent report from ConnCAN repeatedly footnoted a claim as being substantiated to earlier reports…only to result in a dead end where the claim was never substantiated… and in fact, when checking the data turned out to be patently false!  So, this one isn’t even a subtle data interpretation issue. It’s just a lie.

Then there was a report by the organization Third Way, which gathered numerous sources of incompatible data, across incompatible time frames (along with many other bizarre claims) in order to make the argument that America’s middle class schools are failing miserably.

Either of these reports make Matt’s assertions in his post on the Maryland Miracle look totally reasonable!

But for me, the winner among all of the think tank reports I’ve read comes from the Reason Foundation in their 2009 Weighted Student Funding Yearbook! Here’s the abstract of my review:

The new Weighted Student Formula Yearbook 2009 from the Reason Foundation provides a simple framework for touting the successes of states and urban school districts that grant greater fiscal autonomy to schools. The report defines the Weighted Student Formula (WSF) reform extremely broadly, presenting a variety of reforms under the WSF umbrella. Accordingly, when the report concludes that WSF is successful and should be widely replicated, it is difficult to sort through the claims and recommendations. Moreover, the approach and recommendations lack critical inquiry, thought, or empirical analysis. Perhaps most disturbing is the fact that in a third of the specific districts presented in the report, the evidence of success provided predates the implementation of the reforms, and the Reason press release makes the outright claim that past improvements are somehow a function of yet-to-be-implemented reforms. While the report does provide some reasonable recommendations, they are overshadowed by others. Overall, the policy guidance provided by the Reason report is reckless and irresponsible.

Yes… you read it correctly…. If you go through the smashing successes claimed by Reason in this report, in 1/3 of the cases, the reforms in question were implemented after the window of test scores discussed! Hence, the Bunkum time machine award!

Matt’s satirical example didn’t go anywhere near this far.

In Closing….

In my view, there are at least two lessons from Matt’s post, for either side of the reformy aisle.

First, as I so often point out in my classes on applied data analysis, we need to always take  time to carefully evaluate what our data – whatever data and whatever measures – can and cannot tell us. The latter is key here. Descriptive data can be very useful… as long as we understand what they can and cannot tell us. For that matter, various types of inferential statistical analyses (regression models) can also be useful (and in policy research are often primarily descriptive), but often don’t tell us what we think or would like them to tell us. I’ll likely write more about this topic in the future.

Second, we all should take time to carefully scrutinize the link between empirical evidence and policy assertions (and many should take time to take some legit graduate level research methods and statistics and measurement courses on these topics if they wish to continue to opine so boldly about policy inferences!). Perhaps most importantly we should actually take more time and put more effort into scrutinizing those reports and claims that appear most agreeable to our own predisposed beliefs/opinions.  Everyone has predisposed beliefs (especially those who pretend not to). I would argue that experienced researchers likely have stronger beliefs and opinions… and we should… precisely as a result of years of experience researching specific topics.

Oh… and a third lesson… Don’t make completely BS, false/fabricated/absurd graphs like those above. That’s just ridiculous. Are you kidding me? Hiding 3d bars? (Rishawn?) Deleting most of the cases that define the trend? (Cerf?) That’s just ridiculous! Infuriating! Sickening!