The following is a hypothetical case I am using as the culminating activity in Public School Law this semester.

The Dismissal of Principal X

Principal X is principal in a local public middle school in a state that has recently adopted through legislation, articulated with greater precision in state department of education regulations, a new teacher evaluation scheme. The teacher evaluation laws and regulations now require that:

• Any teacher who receives two sequential evaluations less than “satisfactory” shall have his/her tenure status revoked;
• Teacher evaluations shall consist of 40 to 50% measures of student growth, where the majority shall be based on state provided metrics.
• By regulatory decree of the State Commissioner of Education, any other measures selected by local district officials for inclusion in evaluations must be proven correlated with state approved and provided measures of student achievement growth.

Further, the state now conditions receipt of “any and all increases to state aid for local public school districts” on full compliance with statutes and regulations pertaining to teacher evaluation.

On September 20^th of 2013, Principal X was provided with growth percentile data on her teachers from the prior year. Of the approximately 40 certified staff in her school, 8 received growth percentile data, two of whom achieved unsatisfactory growth percentile estimates for their students, one of whom received a second unsatisfactory rating in a row ‐Teacher Y.

In keeping with the requirement that any and all other measures used in the state approved teacher evaluations be correlated with the growth percentile measures, the principal was compelled to assign this teacher a second unsatisfactory rating, and thus compelled to revoke the tenure status of Teacher Y. She was a 10 year veteran teacher perceived by the principal and many others in the school to be one of the school’s most valuable human resources. In fact, over the past several years, the principal had relied on this teacher to take the difficult students including playing a more significant role than others in inclusion of children with disabilities in her classroom – and the teacher not only willingly, but eagerly complied.

Frustrated with the outcome of the new state teacher evaluation laws, Principal X took her case to the public and to state officials simultaneously. Without specific reference to the case in question – but via stylized example – the principal used the case of Teacher Y to illustrate how strict requirements of job action based largely on limited and problematic measures could lead to damaging decisions – decisions

1| Page

that she argued were neither in the best interest of the teachers nor the children they served, and decisions likely to negatively affect the quality of education statewide.

The principal made the case for returning discretion on issues of teacher evaluation and human resource management to local officials, including school principals. The principal’s letter led to a sympathetic uprising from community members and parents, who were quick to catch on as to which teacher was actually the basis of the principal’s hypothetical. Parents of that teacher’s students were outraged, and expressed their outrage at local board of education meetings. During this time, the local board of education maintained quiet support of the principal.

The principal had also begun to engage other principals statewide establishing a network of principals publicly proclaiming their opposition to newly adopted state teacher evaluation statutes and regulations. A web site was created, a non‐profit organization (political action organization) was formed, and the original letter of opposition to new state policies posted on the site, along with a petition for other school principals to show support for the group’s cause and/or become an official member.

State officials were less supportive and unamused by this principal’s apparent disrespect for their authority, and her “willful disobedience of existing statutes and regulations” expressed by Principal X’s stalling on submitting relevant evaluation information necessary for revocation of Teacher Y’s tenure status. Further, state officials were less than thrilled with the mounting insurrection initiated by the publicly posted letter to state officials outlining problems with the state teacher evaluation laws.

State officials released a letter to the local board of education indicating that their state aid would be frozen for the coming school year if, in fact, their rogue principal continued to stall and refuse compliance with the teacher evaluation laws. Under pressure from the board, Principal X agreed to initiate procedures that would lead to tenure revocation for Teacher Y. Instead of waiting out this process, Teacher Y chose to resign and pursue employment elsewhere.

But with mounting pressure on the local board of education from state department officials to control the growing movement among principals statewide against the teacher evaluation laws, a movement initiated by one of their most respected principals (who had received only glowing evaluations in prior years), the district board chose to dismiss Principal X, citing that the principal’s activities had distracted her from doing the job required, substantively compromised her effectiveness as a principal and significantly interfered with the ability of district officials to efficiently and effectively carry on district operations (including the uncertainty created over the district’s future state aid receipts).

Principal X is now suing the district for wrongful dismissal, arguing that the district’s dismissal is in violation of her first amendment right to express herself to the public on issues of public interest, for which she, as an informed public school employee has relevant information.

2| Page

Required Reading

Key Cases

Pickering v. Board of Education: http://www.oyez.org/cases/1960‐1969/1967/1967_510

Connick v. Myers: http://www.oyez.org/cases/1980‐1989/1982/1982_81_1251

Garcetti v. Ceballos: http://www.oyez.org/cases/2000‐2009/2005/2005_04_473

Blogs

EdJurist: Garcetti & Schools http://www.edjurist.com/garcetti‐and‐schools

EdJurist: Academic Blogging & Garcetti: http://www.edjurist.com/blog/2008/5/9/academic‐freedom­garcetti‐blogging.html

Law Reviews

Oluwole, J. O. (2007). On the Road to Garcetti: Unpick’erring Pickering and Its Progeny. Cap. UL Rev., 36, 967.

3| Page

Behavioral economics is an interesting and potentially useful field of academic inquiry. At its best, real behavioral economics attempts to address some of the concerns I raise here. But many if not most assumptions about human behavior and response to incentives are not representative of behavioral economics at its best.

Specifically,  I’m increasingly concerned with what I see as the simple-minded projection of economic thinking onto everyone and anyone else, leading to ridiculous policy recommendations – that amazingly – get taken seriously – at least by the media and punditocracy.

See, for example, Roland Fryer’s experiment on loss aversion as a strategy for incenting teachers to make sure that their students gain a few extra test score points in limited content areas. Indeed, if we pay you up front, and threaten to take your salary away if you don’t get those test score points out those kids, the data suggest  a greater likelihood of squeezing the kids for a few more points. Whether that tells us anything about the motivation and morals of teachers, or of the economists framing this argument is an entirely different question. This tells us little or nothing of the appropriate policy response. Thankfully, the policy implications of this paper were sufficiently absurd that they gained little traction.

Let’s assume classic economic assumptions about human behavior really hold steadfast and can be grossly simplified to an anything for an extra buck, or not to lose one, position. I would argue that it is perhaps economists themselves that are most stereotypical in this regard –  at least as represented in the thinking the project onto others.  In fact, I would argue that many, born out of a culture that self-selects into economic professions, are simply going out of their way to project their own thinking on others.

Further, many of these economists operate in a world where they can influence/control public policy and they too have an incentive in how they behave in this system. They are not impartial observers by any stretch of the imagination. Their goal is to use their economic research to shape public policy to their own advantage.

Put simply, just because the average morally bankrupt economist might do pretty much anything for an extra buck (or a billion), doesn’t mean the average teacher, doctor, nurse, fireman or police officer would!

This issue has been on my mind for some time, but recently came to a head when I read this completely ridiculous Washington Post article on health care policy – specifically – how to remove the incentive for hospitals and physicians resulting from surgical complications.

I should note, I come from a medical family, so some of my arguments herein are drawn from dinner table conversations (across generations), coupled with my tendency to read health policy research out of personal interest in exploring connections with education policy.

It was implied in the WaPo article… well… actually it was explicitly stated in the article that hospitals and physicians have a big financial incentive for their patients to have serious complications, leading to extended hospital stays and additional procedures.

Now, the average economist might be so morally bankrupt such that if he/she were in an operating room (OR) considering the implications of complications relative to potential earnings that they might intentionally introduce infection or other complication, but thankfully the average economist is not in the OR. Thankfully, they self-selected into economics and not medicine (likely foreseeing greater opportunity to earn more for much less work and upfront investment).

The WaPo article does make the following statement, to head off this argument:

The study does not imply that hospitals intentionally complicate surgeries to bring in more revenue.

But, I would argue that this is actually a rather half-hearted disclaimer (to a half-assed argument) for an article that very much implies just that.

Certainly the economists’ policy response – how to employ crude economic assumptions of human behavior to fix this dreadful perverse incentive – implies that cutting off this financial benefit for malpractice would improve hospital and physician behavior [meanwhile conflating the hospital and physician incentives & roles in the various related processes]. Here is the policy solution recommended by the economists cited in the WaPo article:

If hospitals receive a set amount for every heart surgery they perform, for example, they suddenly have an incentive to reduce complications — they know the extra medical spending will come out of their own budget.

Lost in the economists’ reasoning here are a) the potential longer term financial and career implications to the physician repeatedly entangled in litigation over post-surgical complications, b) and the stress/mental toll on the physician arising from managing complications in tense moments in the OR.

Indeed this is anecdotal, but I’ve not met a physician – surgeon or anesthesiologist – who prefers a day when things go bad in the OR – or would be likely to see dollar signs in those moments of stress. What kind of sick bastard even thinks that way? Well, perhaps the average economist does.

Economists rarely – uh… NEVER face comparable professional stress to managing a patient’s life on the edge – even when they make a massively stupid spreadsheet error stimulating economic turmoil across the globe. Nor do they pay hefty malpractice premiums to shield themselves from such egregious malpractice (despite measurable financial damages). I would assert that the economist never faces the stress of having to care for a classroom of 20 to 40, 5 to 15 year old kids, whose immediate safety and well-being, as well as their long term futures is on the line.  This is in part, why they get away with such ludicrous thinking.

It’s all freakin’ game (Freakin’ used here in a technical freakonomics sense)… a game of playing with big data – several layers removed from reality – from people – from real human consequences.

Perhaps that’s the central issue… even more so than economists’ financial self-interests?

Taken in perspective, it’s a fun game and a pretty cushy lifestyle to have opportunity to ponder policy implications of big data, as long as we don’t start thinking that what we do is so freakin’ important and indispensable and as long as we understand where we sit in this big messy puzzle of human behavior and incentives.

Of course, the other interesting piece here is the leap we often see these days between what the study behind the headlines actually said, and the resulting spin in the media headlines. We also often see the economists themselves engaging in the spin. This was equally true in the famed Chetty, Rockoff, Friedman Fire Teachers First, Ask Questions Later study.

For example, here’s what the original study – in the Journal of the American Medical Association – on reimbursements associated with complications actually said:

Depending on payer mix, many hospitals have the potential for adverse near-term financial consequences for decreasing postsurgical complications.

It takes one hell of a leap of logic to get from this measured finding to the policy recommendation above.  It takes projecting economists thinking –amoral greed – onto all actors involved. It also takes ignoring entirely a multitude of contextual factors and perverse consequences (economist thinking – first, we assume none of that stuff exists). Indeed, many complications relate to preexisting conditions and/or overall health of the incoming patient. Do we really want to incent risk aversion? (avoiding those far more likely to have complications?). Well, if it leads to lower premiums for and taxes paid by economists, then perhaps?

Tangentially (or not?), there is an equally ill-conceived movement afoot to apply to healthcare management the brilliance of what we have supposedly learned from measuring teacher effectiveness with value-added models, as explained in this policy brief from Mathematica. Notably, I tend to think Mathematica does pretty good work on education policy (better than most. See here, but for more critical perspective, see here) Put in its best light, this policy brief is merely Mathematica researchers engaging in another I’ve got a Hammer… where’s the freakin’ nail exercise.

Put in the light of economic thinking about human behavior – which many economists prefer to project on all others, the incentive here is for Mathematica to broaden its market, gaining contracts to develop value-added metrics for health care systems and for state and Federal government – to ultimately be used in reducing payments for healthcare, and reducing the tax burden and healthcare premiums paid by Mathematica researchers – their funders and their peers. It’s a win/win. More contracts and higher income, and lower taxes and health benefits expenses (not costs, but expenses*).

That is, as long as they are never in need of surgery.

====

*Cost  reduction implies that quality of service remains constant, whereas expenditure reduction may lead to service quality reduction.

Two buzz phrases have been somewhat quietly floating around reformyland of late, for at least a year or so. I suspect that many have not even picked up on these buzz phrases/words.  They are somewhat inner circle concepts in reformyland. The first is the notion of the great relinquisher (a seemingly bizarre contradiction indeed… to be great at surrendering… but I believe that’s the point). The second is the idea that we all must learn to be sector agnostics. That is, we all must stand behind the provision of a system of great schools as logical replacement for existing school systems and that this system of great schools might be provided by any sector – public/government, charter, private non-profit, private for profit. After all, it doesn’t matter how we provide them, as long as they are great schools. Who can argue with that?

Linking these two conceptions, the great relinquishers – primarily public officials perceived as otherwise self-interested bureaucrats – must learn to relinquish their self-interested stronghold on publicly financed schooling to alternative providers.  Among inner circle reformers, these ideas are treated as somehow ground breaking, deep intellectual thoughts about re-envisioning schooling. But in reality, they are anything but.

On Relinquishers & Sector Agnosticism

Some abbreviated backdrop on the relinquisher notion.  I converse (constructively) on occasion via e-mail with Neerav Kingsland who promotes this particular notion. For those who don’t know Neerav, he’s a Yale Law grad who completed a Broad Residency, and is currently CEO for New Schools for New Orleans. Thus, as I interpret it, he derives his core arguments largely on his perception of the (highly debatable) successes of post-Katrina New Orleans.  That in mind, and with all due respect to Neerav, I have grave concerns about what he refers to as the movement toward “relinquishment” or creating a culture of “relinquishers” among current public officials regarding the provision of the public good of schooling (differing substantively from public schooling.)

Neerav introduced the concept of Relinquishers in a letter he wrote to urban (not all, just “urban”) superintendents in Education Week:

Before I begin in full, let me say this: Superintendents, over the years I’ve begun to believe that your identities–how each of you perceives your professional charge–are often misguided. In my experience, most of you view yourselves as system reformers–leaders who can make the current educational system much better. For the sake of the letter, let’s call you, well, Reformers. With great diligence, you fight to make our government-operated system better.

But let me suggest another identity–one whose charge is to return power, in a thoughtful manner, back to parents and educators. Let’s call these types of superintendents Relinquishers. With great diligence, these superintendents attempt to transfer power away from a centralized bureaucracy.

Both Reformers and Relinquishers possess noble aims, but only one group, I think, possesses a sound strategy.

Superintendents, in the rest of this letter I hope to convince you to become Relinquishers. Specifically, I will advocate that you return power to parents and educators through the creation of charter school districts, which are the most politically acceptable mechanisms for empowering educators. (my emphasis)

Let’s start by taking the word “relinquish” literally for a moment. A quick synonym search in Microsoft Word yields: Surrender, Abandon, Renounce, Resign

The implication here is that public officials must “surrender” or “abandon” or “renounce” their schools, handing them over largely to private managers of charter schools (note that Neerav Kingsland has suggested that charter operators are the “politically acceptable” choice, leaving for others including Smarick to consider conventional private schooling and voucher models). Yeah… I get that this is an interesting notion – to suggest that there is some nobility is declaring defeat and handing control over to those who might be able to play a positive role. I get that. But I find this use, and this framing rather disturbing.

This is not to suggest that I don’t believe that many local public school districts, large or small, need work (some, a hell of a lot of work) on how they interact with their local communities and how they balance stakeholder interests (responsiveness to parents/students, etc.). That’s an ongoing concern in any public or private sector business, with differing structural/governance issues involved in public governance. This is also not to suggest that public officials should never look to other sectors for appropriately contracted, sufficiently regulated support. But “relinquishment” is an extreme perversion of this notion, especially when we start considering relinquishment of the system as a whole – Surrendering, abandoning, renouncing any and all role for public governance and centralized public policy.

Now for this notion of “sector agnosticism” – In his book The Urban School System of the Future and in several tweets and blog posts, former deputy commissioner of Education of New Jersey, Andrew Smarick promotes the reformy religion of what he refers to as Sector Agnosticism. A brief explanation is provided in a recent education week post:

Smarick: “Second, we need to have a three-sector accountability system that treats similarly district public schools, charter public schools, and private schools; we must focus on school results, not school operator. I call this “sector agnosticism;” in other words, we shouldn’t care who runs a school as long as it is superb.”

In the 1990s, when this idea arguably first gained some momentum (summarized in Paul Hill’s book Reinventing Public Education), I was actually a pretty big fan of the idea – which consisted primarily of finding ways to employ private contractors through performance contracting to improve urban schools. Heck, my own first conference paper ever was on the issue of private management of public schools, at a time when I thought there might be great hope for such strategies. Unfortunately, the self-interest of the (publicly traded, for profit) private manager (who eventually fell into financial collapse) to extract as much revenue as possible from the urban district (Baltimore) coupled with their outright disinterest in, and obstruction of having their outcomes measured, started giving me doubts. How could they possible show an efficiency advantage (doing more with less) if they managed to game their budget allocations to their advantage and then wouldn’t provide evidence of results?

Unfortunately, I wrongly assumed things would get better as the industry evolved. Further, over time, as I completed graduate work studying education finance and policy and became reasonably well versed in school law and education governance (teaching it at the graduate level for over a decade & writing/publishing numerous co-authored articles in law review journals) I became more acutely aware of the potential pitfalls of taking an uniformed leap into sector agnosticism.

Defining superbitude?

First, let’s take Smarick’s sound bite notion that it should matter as long as the school is “superb.” Even with a narrow, test score or graduation & post-secondary matriculation-based measure of “superbitude,” neither charter nor private schools are revealing any decisive edge, holding student characteristics or access to resources constant. Rather, as one might logically expect, these less regulated sectors merely produce greater variation around largely the same mean (if comparing similar students). Across sectors, the drivers of outcome variation continue to be the substantive differences in student populations served and oft correlated variations in access to schooling and non-schooling resources (in public schools, charter schools or private schools).[1]

Why do those KIPP charter middle schools appear to perform so well? What about New York City or Newark charter schools more broadly? And what about years of findings on private schools, or students participating in the New York City private school voucher experiment? It’s not about sectors, but rather about strategies and resources. And if it’s about strategies and resources, then if we can identify what works and the resources needed to legitimately serve all children, we can provide those opportunities within a publicly governed, publicly accountable system of common schools. Indeed, if these measured outcomes were in fact the only issue of concern, we might leverage an appropriately mixed set of schooling providers to get the job done. In fact, the lack of decisive advantage by sector alone is equal justification for agnosticism as it is against it.

But, that’s only if we ignore entirely that there might actually be other tradeoffs involved, beyond whatever test score, graduation, matriculation or employment outcome might be achieved.

Trading Off Legal Rights for Test Scores?

It’s not just about figuring out how to achieve crudely measured “superberific” schooling.  Our children’s schooling exists in a broader social, political and legal context. Kids have legal rights, and under most state constitutions kids a right to access/participate in/gain the benefits of a system of schooling (sometimes, quite explicitly, a system of public schooling). In many states, they not only have a right to access schooling (at times, of some measured degree of quality), but a legal obligation to attend up to a specified age (compulsory schooling laws).

As I’ve discussed on a few previous blog posts, privately governed and/or managed charter schools, more like traditional private than like public schools, may not be (are likely not) subject to the full protection of students’ constitutional or statutory rights (summary table from previous post included below). When attending a private school, it’s clear that kids have no right to continued attendance. They can be expelled, excluded outright for any number of reasons (including admissions testing). They may be compelled to recite school oaths and may be obligated to participate in religious activities and may be restricted in their ability to freely express themselves and subject to disciplinary action including expulsion for failure to comply. Parents may also be obligated to participate in certain activities as a condition of continued enrollment.

While charter advocates love to declare their schools as necessarily “public,” with regard to at least some of these same issues/questions, Charter school legal defense attorneys are quick to argue that they are in fact, private. That, for example, children’s rights under disciplinary codes should be treated as private contracts entered into by parents, just as in private schools – and substantively different from “public” schools – or those formally governed and operated by agents of the state (local elected school boards and public district administrators).

Further, the public-private delineation and murky middle ground of charter schooling raises numerous additional substantive legal questions regarding public employment law and employee rights, taxpayer and citizen rights to open public meetings and public records, and rights, responsibilities, liabilities and protections of “public officials” such as school board members and public employees as opposed to governing boards of private citizens, and employees of private contractors.

Sector agnosticism, as dreadfully simplified by Andrew Smarick requires completely ignoring these substantive tradeoffs.  Trading off constitutional rights to reduce supply of some and increase access to other sectors is not benign, if those sectors could/might possibly yield other advantages.

The Distribution of Lost Rights

Nor do I suspect that the tradeoff of rights will ever be randomly distributed across children by the wealth and income of their families. No-one is asking the superintendent of Scarsdale (great guy, by the way) to Relinquish his schools and adopt a policy of Sector Agnosticism. This is a policy for the children of New Orleans, New York City, Chicago, Philadelphia and Newark.

In the extreme case – a case favored by Smarick and seemingly endorsed (through relinquishment) by Kingsland – a district – or now merely a geographic space – where children have only access to privately governed/managed charter schools may require that any/all that wish to actually exercise their state constitutional right to attend school, have to choose which rights to forgo in the process? Will 100% of parents in that zone be required to enter into contractual agreements (forgoing constitutional & statutory protections) with schools regarding disciplinary policies for their children?

In fact, Kingsland’s logic is that district superintendents should simply succumb or surrender to the forces that wish to forcibly close and takeover their schools, and relinquish those schools or at least the children who would have attended them, to other sectors. Following Smarick’s logic, parents and citizens at large should completely ignore tradeoffs of constitutional protections, or humiliating treatment of children, in lieu of Smarickian measures of “superbification.”

Creating a scenario where only low income minority children in America’s cities must tradeoff their constitutional and statutory protections to gain access to schooling (which they may be compelled to attend) is clearly unacceptable, inequitable treatment.  Before you go there… no… I’m not saying that the responsible policy solution is to make sure that suburban kids and their parents are equally deprived of protections.  What’s not good for some is not good for all.

One logical retort to my arguments here is that if parents want these choices, if they are backed up on waiting lists for existing charters, then we should provide to them. If the demand is there, let the supply meet the demand!? It would be one thing if it was made clear, up front, to potential choosers these hidden tradeoffs, but that’s not the case. If anything, charter advocates are doing their best to conceal that any such tradeoffs exist.

Indeed, appropriate cross-sector regulation might negate some of my concerns raised here, but these issues are too frequently ignored.

Market Manipulation & The Forcible Reduction of the “Public Option”

Worse, in the current policy context, we are not witnessing the emergence of a true, fair and equitable, demand driven and fully open and accessible (driven by open information) system of choice. Policies of relinquishment and sector agnosticism are being pursued in practice as policies of forced relinquishment (read mass closings) of traditional public schooling and sector favoring transfer of assets (public to privately governed charters), coupled with gross misrepresentations of information on sector quality.

In selected cases, we are also witnessing a coordinated effort to provide competitive advantage for non-district alternatives. Where sectors are set up to compete with one another to prove their worth, the likelihood that charter or voucher advocates will lobby for increased resources for district schools is about as likely as the New York Yankee ownership arguing for revenue sharing to help the Kansas City Royals, or Walmart to lobby for tax breaks for Target. Similarly, the likelihood that well endowed charters will share their philanthropy with others less fortunate is slim to none where the emphasis remains on flaunting one’s competitive advantage.

A veneer of demand (as measured by duplicative waiting lists) for private and charter sectors has been induced by forcible reduction of supply of urban schooling, and gross misrepresentations & mismeasures (New Jersey/ New York) of neighborhood schooling quality and manipulation of the playing field.

Closing Thoughts

Before we jump on these reformy bandwagons, and start waiving the white flag of relinquishment and promoting the virtues of sector agnosticism, we need to take a hard look at how this is playing out in our cities. Numerous New Orleans schools were wiped out by a natural disaster, displacing large shares of the lowest income residents to Houston (and elsewhere), many of whom have not been able to return in part because the market based model of New Orleans has chosen not to serve their former, blighted neighborhoods.  This was tragic, and the initial occurrence largely beyond policymakers’ control.  The choice to leave children and their families unserved or underserved was a conscious policy decision (or a least a predictable result of the policy response).

Proposed Chicago (and Philadelphia) school closings would appear comparably poised to induce increased demand for charters, which will likely be used as rationale for expanding charters even further and advancing the cycle toward its ultimate end (as if Katrina by design, and more surgically targeted at schools with low test scores and poor minority children). In most U.S. cities, however charter market shares remain modest, and publicly subsidized private school enrollment even smaller, providing an opportunity to pause and rethink current strategies.

So then, what do we do about all of this? First, reformers and non-reformers alike (and anti-reformers too!) need to step back from these oversimplified talking points and buzz phrases which so illustrate the worst of intellectually lazy, undisciplined, under-informed policy development. I don’t mean to be a hypercritical, ivory tower (actually, public university 1960s era building basement) academic … okay… yeah… that is what I mean to be here. Why? Because it matters! Exploring and understanding these tradeoffs matters. Ignoring them is reckless.

Yesterday, Matt Di          Carlo over at Shankerblog put out his April fools post. The genius of the post is in its subtlety.  Matt put together a few graphs of longitudinal NAEP data showing that Maryland had made greater than average national gains on NAEP and then asserted that these gains must therefore be a function of some policy conditions that exist in Maryland. In the Post-RTTT era, Maryland has been the scorn of “reformers” because it just won’t get on board with large scale vouchers and charter expansion and has resisted follow through on test-score based teacher evaluation. Taking a poke a reformy logic, Matt asserted that perhaps the low charter share and lack of emphasis on test score based teacher evaluation… along with a dose of decent funding might be the cause of Maryland’s miracle!

Of course, these assertions are no more a stretch than commonly touted miracles in Texas in the 1990s, Florida or Washington DC, most of which are derived from making loose connections between NAEP trend data and selective discussion of preferred policies that may have concurrently existed.  The difference is that Matt was poking fun at the idea of making bold, decisive, causal inferences from such data. Such data raise interesting questions.

What I found so fun and at the same time deeply disturbing about Matt’s post is that the assertions he made in satire… were nowhere near as absurd as many of the assertions made in studies/reports, etc. I discussed here on my blog over the years. Here are but a few examples of “stuff” presented as serious/legit policy evidence, that make Matt’s satirical assertions seem completely reasonable.

The Many Variations of Money Doesn’t Matter Graphs:

I start with this one, because there are so many versions of it floating around out there, that come and go over time, and are often used to advance the “money doesn’t matter”… we’ve spent ourselves into bankruptcy and gotten nothing for it… graph. Every good reformer has a laminated copy of one version or another of this graph which they carry in wallet-size.

I blogged about this graph when Bill Gates used it in a HuffPo article.

Gates asserted:

 Over the last four decades, the per-student cost of running our K-12 schools has more than doubled, while our student achievement has remained flat, and other countries have raced ahead. The same pattern holds for higher education. Spending has climbed, but our percentage of college graduates has dropped compared to other countries… For more than 30 years, spending has risen while performance stayed flat. Now we need to raise performance without spending a lot more.

Among other things, the chart includes no international comparison, which becomes the centerpiece of the policy argument. Beyond that, the chart provides no real evidence of a lack of connection between spending and outcomes across districts within U.S. States.  Instead, the chart juxtaposes completely different measures on completely different scales to make it look like one number is rising dramatically while  the others are staying flat. This tells us NOTHING. It’s just embarrassing. Simply from a graphing standpoint, a blogger at Junk Charts noted:

Using double axes earns justified heckles but using two gridlines is a scandal!  A scatter plot is the default for this type of data. (See next section for why this particular set of data is not informative anyway.)

Not much else to say about that one. Again, had I used an example this absurd to represent reformy research and thinking, I’ d have likely faced stern criticism for mis-characterizing the rigor of reformy research!

This alternate version comes to us from none other than Andrew Coulson of Cato Institute. Coulson has a stellar record of this kind of stuff. So, what would you do to the Gates graph above if you really wanted to make your case that spending has risen dramatically and we’ve gotten no outcome improvement? First, use total rather than per pupil spending (and call it “cost”) and then stretch the scale on the vertical axis for the spending data to make it look even steeper. And then express the achievement data in percent change terms because NAEP scale scores are in the 215 to 220 range for 4th grade reading, for example, but are scaled such that even small point gains may be important/relevant but won’t even show as a blip if expressed as a percent over the base year.

Chris Cerf’s Poverty Doesn’t Matter Graph!

Now, it’s one thing when and under-informed tech CEO goes all TED-style on us with big screens, gadgets, bells and whistles and info-graphics that just don’t mean crap anyway. But, it’s yet another when a State Commissioner of Education presents something not only equally ridiculous… but arguably far more ridiculous, disingenuous, unethical and downright WRONG.

This is a graph for the ages, and it comes from a presentation by the New Jersey Commissioner of Education given at the NJASA Commissioner’s Convocation in Jackson, NJ on Feb 29. State of NJ Schools presentation 2-29-2012

The title conveys the intended point of the graph – that if you look hard enough across New Jersey – you can find not only some, but MANY higher poverty schools that perform better than lower poverty schools.

This is a bizarre graph to say the least. It’s set up as a scatter plot of proficiency rates with respect to free/reduced lunch rates, but then it only includes those schools/dots that fall in these otherwise unlikely positions. At least put the others there faintly in the background, so we can see where these fit into the overall pattern. The suggestion here is that there is not pattern.

Note: this graph may not even be the worst one in the presentation. You decide!

The apparent inference here? Either poverty itself really isn’t that important a factor in determining student success rates on state assessments, or, alternatively, free and reduced lunch simply isn’t a very good measure of poverty even if poverty is a good predictor. Either way, something’s clearly amiss if we have so many higher poverty schools outperforming lower poverty ones. In fact, the only dots included in the graph are high poverty districts outperforming lower poverty ones. There can’t be much of a pattern between these two variables at all, can there? If anything, the trendline must be sloped uphill? (that is, higher poverty leads to higher outcomes!)

Note that the graph doesn’t even tell us which or how many dots/schools are in each group and/or what percent of all schools these represent. Are they the norm? or the outliers?

Well, here’s what the pattern really looks like with all schools included:

Hmmm… looks a little different when you put it that way. Yeah, it’s a scatter, not a perfectly straight line of dots. And yes, there are some dots to the right hand side that land above the 65 line and some dots to the left that land below it.

Note: New Jersey’s Chris Cerf is not alone among state commissioners in promoting completely bogus analysis posing as empirical validation. In fact, New York’s John King presented a completely fabricated graph provided to him by a consultant to the state and has used that graph to frame his state’s policy initiatives.

Rishawn Biddle’s Graph of, well, something? What?

Not to be outdone, Rishawn Biddle who on occasion fashions himself a “researcher” on education policy issues, provides a graph that comes close to the degrees of intentional deception presented by Commissioner Cerf above.  I blogged about this graph here!

In response to arguments I had made on my blog regarding the role of substantive and sustained school finance reforms in improving school quality, Biddle argued:

Despite the arguments (and the pretty charts) of such defenders as Rutgers’ Bruce Baker, there is no evidence that spending more on American public education will lead to better results for children.

My claims are substantiated in this peer reviewed article and this separate more comprehensive report:

• Baker, B. D., & Welner, K. G. (2011). School Finance and Courts: Does Reform Matter, and How Can We Tell?. Teachers College Record, 113(11), 2374-2414.
• Baker, B. D. (2012). Revisiting the Age-Old Question: Does Money Matter in Education?. Albert Shanker Institute. http://www.shankerinstitute.org/images/doesmoneymatter_final.pdf

And what does Biddle provide as counter evidence to this – apparent lack of evidence I summarize above (I’ve sent the article link to Biddle on more than one occasion, but he apparently doesn’t read this kind of academic stuff)?

Biddle counters with a link to this graph – a true gem (I’ve added some annotation, not in his original)!

Yes, Biddle’s counter to the body of research he has not and likely will never read, is to use this graph of “promoting power” by student race group for Jersey City, NJ in 2004 and 2009. Note that the infusion of additional funds in NJ occurred mainly from 1998 to 2003, leveling off thereafter. But that’s a tangential point (not really).  So, Biddle’s absolute verification that more money doesn’t matter is to simply assert without verification that Jersey City got a whole lot more money and then to use this graph to argue that nothing improved!

First of all, that analysis wouldn’t pass muster in as a master’s degree level assignment (I teach a class on this stuff at that level), no less major research conclusions. From a graphing standpoint, I often criticize my students’ work for what I refer to as gratuitous use of 3d – especially where the use of 3d bars actually obscures the comparisons by making it hard to see where they align on the axis.

But, the really funny if not warped part of this graph is that there appear to be significant gains for black males between 2004 and 2009, but those gains are obscured by hiding the 2009 black male score behind the 2004 black female score.

Note that the graph also contains no information regarding the actual shares of the student population that fall into each group? Not very useful. Pretty damn amateur. Certainly fails to make any particular point, and certainly doesn’t refute the various citations above – all of which employ more rigorous analytic methods, apply to more than a single district, and most of which appear in rigorous peer reviewed journals.

Reason Foundation’s Today’s Policies Affected Yesterday’s Outcomes Study!

Finally, in my years as a reviewer for the National Education Policy Center’s Think Tank Review Project I’ve reviewed a lot of sketchy stuff. Some of it stands out, and has even won Bunkum awards from NEPC.

For example, a recent report from ConnCAN repeatedly footnoted a claim as being substantiated to earlier reports…only to result in a dead end where the claim was never substantiated… and in fact, when checking the data turned out to be patently false!  So, this one isn’t even a subtle data interpretation issue. It’s just a lie.

Then there was a report by the organization Third Way, which gathered numerous sources of incompatible data, across incompatible time frames (along with many other bizarre claims) in order to make the argument that America’s middle class schools are failing miserably.

Either of these reports make Matt’s assertions in his post on the Maryland Miracle look totally reasonable!

But for me, the winner among all of the think tank reports I’ve read comes from the Reason Foundation in their 2009 Weighted Student Funding Yearbook! Here’s the abstract of my review:

The new Weighted Student Formula Yearbook 2009 from the Reason Foundation provides a simple framework for touting the successes of states and urban school districts that grant greater fiscal autonomy to schools. The report defines the Weighted Student Formula (WSF) reform extremely broadly, presenting a variety of reforms under the WSF umbrella. Accordingly, when the report concludes that WSF is successful and should be widely replicated, it is difficult to sort through the claims and recommendations. Moreover, the approach and recommendations lack critical inquiry, thought, or empirical analysis. Perhaps most disturbing is the fact that in a third of the specific districts presented in the report, the evidence of success provided predates the implementation of the reforms, and the Reason press release makes the outright claim that past improvements are somehow a function of yet-to-be-implemented reforms. While the report does provide some reasonable recommendations, they are overshadowed by others. Overall, the policy guidance provided by the Reason report is reckless and irresponsible.

Yes… you read it correctly…. If you go through the smashing successes claimed by Reason in this report, in 1/3 of the cases, the reforms in question were implemented after the window of test scores discussed! Hence, the Bunkum time machine award!

Matt’s satirical example didn’t go anywhere near this far.

In Closing….

In my view, there are at least two lessons from Matt’s post, for either side of the reformy aisle.

First, as I so often point out in my classes on applied data analysis, we need to always take  time to carefully evaluate what our data – whatever data and whatever measures – can and cannot tell us. The latter is key here. Descriptive data can be very useful… as long as we understand what they can and cannot tell us. For that matter, various types of inferential statistical analyses (regression models) can also be useful (and in policy research are often primarily descriptive), but often don’t tell us what we think or would like them to tell us. I’ll likely write more about this topic in the future.

Second, we all should take time to carefully scrutinize the link between empirical evidence and policy assertions (and many should take time to take some legit graduate level research methods and statistics and measurement courses on these topics if they wish to continue to opine so boldly about policy inferences!). Perhaps most importantly we should actually take more time and put more effort into scrutinizing those reports and claims that appear most agreeable to our own predisposed beliefs/opinions.  Everyone has predisposed beliefs (especially those who pretend not to). I would argue that experienced researchers likely have stronger beliefs and opinions… and we should… precisely as a result of years of experience researching specific topics.

Oh… and a third lesson… Don’t make completely BS, false/fabricated/absurd graphs like those above. That’s just ridiculous. Are you kidding me? Hiding 3d bars? (Rishawn?) Deleting most of the cases that define the trend? (Cerf?) That’s just ridiculous! Infuriating! Sickening!

Okay, it’s not entirely surprising to find mind-boggling ignorance conveyed in the editorial pages of the New York Post. Today’s example comes to us in an Op-Ed written in response to a report released by the Alliance for Quality Education.

Usually, I’d just let it pass. It’s the Post after all. But, for two important reasons I just had to address this one.  First, the editorial was written by a member of the Governor’s Education Reform Commission.  Second, the editorial made use of our School Funding Fairness report to make its most absurd claim. And here is that claim:

Despite all of AQE’s complaints, there is no need to change the way we allocate this money, since the state already directs almost 70 percent of education funding to high-need districts. In fact, School Funding Fairness’s National Report Card gave New York a grade of “A” in its Effort category, putting us among the top five states in that category.

http://www.nypost.com/p/news/opinion/opedcolumnists/ny_schools_money_not_the_problem_qHXdkNNBLssY1Swqj8LfQK

Apparently the authors of this quote have a little difficulty reading and perhaps some problems interpreting relatively simple numbers and letter grades.

Let’s take a look at the graded indicators in our school funding fairness report:

• Effort – This measures differences in state spending for education relative to state fiscal capacity. “Effort” is defined as the ratio of state spending to state per capita gross domestic product (GDP).
• Funding Distribution – This measures the distribution of funding across local districts within a state, relative to student poverty. The measure shows whether a state provides more or less funding to schools based on their poverty concentration, using simulations ranging from 0% to 30% child poverty.

 http://www.schoolfundingfairness.org/National_Report_Card_2012.pdf

The authors use the state’s high grade on “effort” as basis for asserting that New York State has no allocation problem. Hmmm… perhaps the distribution indicator would be a better indicator for whether the state has an allocation problem. After all, even a quick synonym check in MS Word lists distribution as the first synonym for allocation (followed by provision, apportionment, sharing).

New York Received a D on this measure!

And a D is actually rather generous for New York’s distribution/allocation issues. Figure 1 provides the profile for mid-Atlantic states from our funding fairness report.

Figure 1

In other words, in New York State, higher poverty districts have systematically less per pupil state and local revenue than do lower poverty ones. And despite all of the other completely ridiculous a-contextual and otherwise wrong and misleading claims in the Op-ed, high need New York State school districts face significant financial disadvantages relative to their competitive surroundings.

Despite the Op-Ed authors claims, their Governor has done little or nothing to help these districts, and much to harm them (including misguided reforms).

Sean Corcoran of NYU and I dug deeper into New York State’s distribution issues and sources of inequity in a recent report for the Center for American Progress.

First, with updated analysis following the funding fairness methods, we identify New York State as among the least equitable states in the Nation. Here are the numbers:

And here’s a nice colored map (which may resonate with the editorial authors who’s grasp of numbers appears severely limited):

http://www.americanprogress.org/wp-content/uploads/2012/09/StealthInequities.pdf

We explain that a hypothetical, rational, equitable school funding system should look something like this:

But that New York’s system actually looks like this – when not correcting for costs/needs (it’s much, much worse when you do!).

We also then identify and explain the various sources of inequity in New York State’s school finance system, including the allocation of Tax Relief aid disproportionately to wealthier districts, and other adjustments to foundation aid that favor wealthier districts to the detriment of poorer ones. Here are some figures, and descriptions from our report:

Figure 13 puts New York’s School Tax Relief program aid allocations into context. Federal aid to schools is largely designed to improve equity by targeting resources to higher-need, especially higher-poverty, districts. The School Tax Relief pro­gram aid to New York schools tends on average to be slightly less than federal aid. But to the extent that federal aid creates any improvement to the distribu­tion of resources across New York school districts, the state’s School Tax Relief program aid wipes out that improvement entirely. Aid under the program, which is indicated by the darkest blue in Figure 13, is allocated in nearly perfectly inverse proportion to federal aid, such that when the two are stacked on the other, the cumulative effect is that districts receive about the same regardless of their wealth.

 New York’s foundation-aid formula includes a series of “if/then” steps to deter­mine whether a district should receive state aid based on its initial calculation of local fair share or based on an alternative calculation, one of which is the provision of minimum aid of $500 per pupil. Figure 17 shows the pattern of state and local sharing that would occur if foundation aid were based solely on the income-wealth index estimated by the state. Under that index, the lowest-wealth districts would receive about $12,000 to $14,000 in aid per pupil, and districts with an income-wealth index greater than 1 would receive no aid. After including the various alter­native calculations, districts with an income-wealth index above about 2.5 would receive the minimum of $500 per pupil, while districts with index rates from 1 to 2.5 would receive a sliding scale toward the minimum rather than either the mini­mum or $0. The adjusted version is shown in red. Note however, that neither was fully funded in recent years. (Recent reality is achieved by taking the red squares and shifting them downward but preserving the minimum aid.)

If fully funded, the cost of retaining the minimum aid provision tops $1.2 billion, and the cost of preserving the diagonal, sliding-scale adjustment between the income-wealth index of 1 and 2.5 is $2.47 billion (if we exclude the disproportion­ate effects of New York City). That’s real money—money that could be perhaps targeted toward higher-need districts to reduce the overall regressive nature of New York’s finance system.

The cumulative effects of these adjustments and the School Tax Relief program on the distribution of resources across New York school districts is shown above. The left hand portion of Figure 18 shows local revenue with formula aid prior to the adjustments in Figure 17. Even this isn’t a very pretty picture because state aid remains insufficient to provide even nominal funding equity from lower- to higher-poverty districts. But the right hand side of Figure 18 shows the effect of the adjustments in Figure 17, with the icing of school tax relief aid on top.

Prior to foundation-sharing adjustments and the School Tax Relief program aid, the per-pupil difference in state and local revenue per pupil between the lowest- and highest-poverty quintile is about $1,100. After the adjustments, the per-pupil difference is more than $2,300. New York makes adjustments to its aid formula and throws on tax relief funding in a pattern that more than doubles the nominal inequity between the state’s lowest- and highest-poverty districts.

In other words – YES – NEW YORK DOES NEED TO CHANGE THE WAY IT ALLOCATES MONEY. Any suggestion to the contrary displays a mind-boggling degree of ignorance!

Knowing that I’ve been writing a fair amount about various methods for attributing student achievement to their teachers, several colleagues forwarded to me the recently released standards of the Council For the Accreditation of Educator Preparation, or CAEP. Specifically, several colleagues pointed me toward Standard 4.1 Impact on Student Learning:

4.1.The provider documents, using value-added measures where available, other state-supported P-12 impact measures, and any other measures constructed by the provider, that program completers contribute to an expected level of P-12 student growth.

http://caepnet.org/commission/standards/standard4/

Now, it’s one thing when relatively under-informed pundits, think tankers, politicians and their policy advisors pitch a misguided use of statistical information for immediate policy adoption. It’s yet another when professional organizations are complicit in this misguided use. There’s just no excuse for that! (political pressure, public polling data, or otherwise)

The problems associated with attempting to derive any reasonable conclusions about teacher preparation program quality based on value-added or student growth data (of the students they teach in their first assignments) are insurmountable from a research perspective.

Worse, the perverse incentives likely induced by such a policy are far more likely to do real harm than any good, when it comes to the distribution of teacher and teaching quality across school settings within states.

First and foremost, the idea that we can draw this simple line below between preparation and practice contradicts nearly every reality of modern day teacher credentialing and progress into and through the profession:

one teacher prep institution –> one teacher –> one job in one school –> one representative group of students

The modern day teacher collects multiple credentials from multiple institutions, may switch jobs a handful of times early in his/her career and may serve a very specific type of student, unlike those taught by either peers from the same credentialing program or those from other credentialing programs. This model also relies heavily on minimal to no migration of teachers across state borders (well, either little or none, or a ton of it, so that a state would have a large enough share of teachers from specific out of state institutions to compare). I discuss these issues in earlier posts.

Setting aside that none of the oversimplified assumptions of the linear diagram above hold (a lot to ignore!), let’s probe the more geeky technical issues of trying to use VAM to evaluate ed school effectiveness.

There exist a handful of recent studies which attempt to tease out certification program effects on graduate’s student’s outcomes, most of which encounter the same problems. Here’s a look at one of the better studies on this topic.

• Mihaly, K., McCaffrey, D. F., Sass, T. R., & Lockwood, J. R. (2012). Where You Come From or Where You Go?

Specifically, this study tries to tease out the problem that arises when graduates of credentialing programs don’t sort evenly across a state. In other words, a problem that ALWAYS occurs in reality!

Researchy language tends to downplay these problems by phrasing them only in technical terms and always assuming there is some way to overcome them with statistical tweak or two. Sometimes there just isn’t and this is one of those times!

Let’s dig in. Here’s a breakdown of the abstract:

In this paper we consider the challenges and implications of controlling for school contextual bias when modeling teacher preparation program effects. Because teachers from any one preparation program are hired in more than one school and teachers are not randomly distributed across schools, failing to account for contextual factors in achievement models could bias preparation program estimates.

Okay, that’s a significant problem! Teachers from specific prep institutions are certainly not likely to end up randomly distributed across a state, are they? And if they don’t, the estimates of program effectiveness could be “biased.” That is, the estimates are wrong! Too high, or to low, due to where their grads went as opposed to how “good” they were. Okay, so what’s the best way to fix that, assuming you can’t randomly assign all of the teacher grads to similar schools/jobs?

Including school fixed effects controls for school environment by relying on differences among student outcomes within the same schools to identify the program effects.  However, the fixed effect specification may be unidentified, imprecise or biased if certain data requirements are not met.

That means, that the most legit way to compare teachers across programs is if you can compare teachers whose first placements are in the same schools, and ideally where they serve similar groups of kids. And, you’d have to have a large enough sample size at the lowest level of analysis – comparable classrooms within school – to accomplish this goal. So, the best way to compare teachers across prep programs is to have enough of them, from each and every program, in each school, teaching similar kids similar subjects at the same grade level, across grade levels. Hmmmm…. How often are we really likely to meet this data requirement?

Using statewide data from Florida, we examine whether the inclusion of school fixed effects is feasible in this setting, the sensitivity of the estimates to assumptions underlying for fixed effects, and what their inclusion implies about the precision of the preparation program estimates. We also examine whether restricting the estimation sample to inexperienced teachers and whether shortening the data window impacts the magnitude and precision of preparation program effects. Finally, we compare the ranking of preparation programs based on models with no school controls, school covariates and school fixed effects. We find that some preparation program rankings are significantly affected by the model specification. We discuss the implications of these results for policymakers.

With “no school” controls means not accounting at all for differences in the schools where grads teach. With “covariates” means correcting in the model for the measured characteristics of the kids in the schools – so – trying to compare teachers who teach in similar – by measured characteristics – schools. But, measured characteristics often fail to catch all the substantive differences between schools/classrooms.  And where “school fixed” effects means comparing graduates from different institutions who teach in the same school (though not necessarily the same types of kids!).

Okay, so the authors tested their “best” methodological alternative (comparing teachers within schools, by school “fixed” effect) with other approaches, including making no adjustment for where teachers went, or making adjustments based on the characteristics of the schools, even if not matched exactly.

The authors found that the less good alternatives were, to no surprise, less good- potentially biased. The assumption being that the fixed effect models are most correct (which doesn’t, however, guarantee that they are right!).

So, if one can only legitimately (though really not in this case either) compare teacher prep programs in cases where grads across programs are concentrated in the same schools for their first jobs, that’s a pretty severe limitation. How many job openings are there in a specific grade range in a specific school in a given year – or even over a five year period? And how likely is it that those openings can be filled with one teacher each from each teacher prep institution. But wait, really we need more than one from each to do any legit statistical comparison – and ideally we need for this pattern to be replicated over and over across several schools. In other words, the constraint imposed to achieve the “best case” model in this study is a constraint that is unlikely to ever be met for more than a handful of large teacher prep institutions concentrated in a single metropolitan area (or very large state like Florida).

Other recent studies have not found VAM particularly useful in parsing program effects:

We compare teacher preparation programs in Missouri based on the effectiveness of their graduates in the classroom. The differences in effectiveness between teachers from different preparation programs are very small. In fact, virtually all of the variation in teacher effectiveness comes from within-program differences between teachers. Prior research has overstated differences in teacher performance across preparation programs for several reasons, most notably because some sampling variability in the data has been incorrectly attributed to the preparation programs.

Koedel, C., Parsons, E., Podgursky, M., & Ehle, M. (2012). Teacher Preparation Programs and Teacher Quality: Are There Real Differences Across Programs? (No. 1204).

http://econ.missouri.edu/working-papers/2012/WP1204_koedel_et_al.pdf

Example from Kansas

Let’s use the state of Kansas and graduates over a five year period from the state’s major teacher producing institutions to see just how problematic it is to assume that teacher preparation institutions in a given state will produce sufficient numbers of teachers who teach in the same schools as graduates of other programs.

All programs

Specific programs

Indeed, the overlap in more population dense states is somewhat more significant, but still unlikely sufficient to meet the high demands of the fixed effects specification (where you can only essentially compare when you have graduates of different programs working in the same school together, in similar assignments… presumably similar number of years out of their prep programs).

Strategically Gaming Crappy, Biased Measures of “Student Growth”

In practice, I doubt most schools of ed, or state education agencies will actually consider how to best model program effectiveness with these measures. They likely won’t even bother with this technically geeky question of the fixed effects model, and data demands to apply that model. Rather, they’ll be taking existing state provided growth scores or value-added estimates and aggregating them across their graduates.

Given the varied, often poor quality of state adopted metrics, the potential for CAEP Standard 4.1 to decay into absurd gaming is quite high. In fact, I’ve got a gaming recommendation right here for teacher preparation institutions in New York State.

We know from the state’s own consultant analyzing the growth percentile data that:

Despite the model conditioning on prior year test scores, schools and teachers with students who had higher prior year test scores, on average, had higher MGPs. Teachers of classes with higher percentages of economically disadvantaged students had lower MGPs. (p. 1) https://schoolfinance101.com/wp-content/uploads/2012/11/growth-model-11-12-air-technical-report.pdf

We also know from this same technical report that the bias appears to strengthen with aggregation to the school level. It may also strengthen with aggregation across similar schools. And this is after conditioning the model on income status and disability status.

As such, it is in the accreditation interest of any New York State teacher prep institution to place as many grads as possible into lower poverty schools, especially those with fewer children with disabilities. By extension, it is therefore also in the accreditation interest of NY State teacher prep institutions to reduce the numbers of teachers they prepare in the field of special education. As it turns out, the New York State growth percentiles are also highly associated with initial scores – higher initial average scores are positively associated with higher growth. So, getting grads into relatively higher performing schools might be advantageous.

With a little statistical savvy, a few good scatteplots, one can easily mine the biases of any state’s student growth metrics to determine how to best game them in support of CAEP standard 4.1.

Further, because it is nearly if not entirely impossible to use these data to legitimately compare program effects, the best one can do is to find the most advantageous illegitimate approach.

Are these really the incentives we’re looking for?

First, here’s a primer and reading list on the Empire State of School Finance:

1. New York State maintains one of the least equitable state school finance systems in the nation
2. New York State actually allocates a ton of state aid to districts that need it least, exacerbating the disparities
3. Reformy types in New York State thought, under these circumstances, it would be really cool to make any additional state aid a district receives contingent on adopting a teacher evaluation scheme based on their documented deeply flawed metrics!
4. To ice that reformy cake, the legislature saw fit to – after slashing state aid year after year – impose a local property tax limit on districts so that they are unable to even raise the funds they would need to provide a sound basic education, if they could raise those funds locally.

ohhh… but I’m just getting started here.  Then came the lawsuits. That’s what makes this so fun and interesting to watch.

Now, there is already a pending lawsuit challenging the overall adequacy of state funding in New York specifically for high need cities (brought by the state’s small city school districts).

More recently however, we’ve been hearing of two separate cases.

First, we have the state teacher’s union (as reported) suing the state over the imposition of the property tax cap, which, in effect prohibits many districts from making up the difference from the aid they’ve been screwed out of for the past several years – the aid that in theory – by the state’s own definition of its foundation formula – would provide for a sound basic education. That formula was implemented specifically to comply with a previous court order in Campaign for Fiscal Equity. 

Next, we have the lawsuit brought on behalf of children in New York City schools challenging the state’s authority to reduce the city’s funding by $250 million for non-compliance with adopting a teacher evaluation policy.

So far, it would appear that this argument has achieved a positive, immediate response from the judge, who a this stage has blocked the state funding reduction.
As laid out in full here: http://schoolfunding.info/wp-content/uploads/2013/02/Memorandum-of-Law-in-Opp-to-App-for-Prelim-Inj1.pdfAnd as characterized here: http://schoolfunding.info/2013/02/miriam-aristy-v-state-of-new-york/

Assistant Attorney General Steven Schulman described the $250 million that the state will cut from NYC schools as a “drop in the bucket” and argued that it was not great enough to have any effect on schools’ ability to provide a sound basic education.

The state’s defense of its actions is essentially that $250 million really isn’t that much money for New York City and certainly doesn’t deprive NYC schoolchildren of receiving their constitutionally mandated sound, basic education. And that forcing the state to provide the $250 million would undermine their authority. That is, their authority to deprive kids of their constitutionally mandated sound basic education! ? ! ? huh? Now, this is all part of legal maneuvering. Yeah… it would be difficult for NYC to show that holding back this additional 3.3% state funding tips the scales on whether the city can provide a sound basic education. As such, how can the court reason intervening and forcing the state to give this money back?
But that’s only if we set aside that the state of New York is already depriving New York City schoolchildren of 38% of the aid that should be allocated to the city based on the state’s own formula for what the city needs to provide a sound basic education. And that was a bogus, low-balled estimate to begin with. Here’s my quick run down on state aid shortfalls in 2012-13 – with respect to the state’s own estimates – for small city districts and for New York City:

Foundation Aid, Foundation after GEA expressed in Thousands (‘000s)New York City is being shorted about 3.4 billion in aid to achieve what the state has defined as “sound basic” funding. That’s about 38% of their total foundation aid.  That share is even larger for some small city districts. This next table shows that this amounts to thousands per pupil.  Sure, the loss from the teacher evaluation debacle amounts to a few hundred per pupil. But hey, what’s the harm? NYC is already being shorted over $3,000 per pupil.By the state’s logic, we, and the sitting judge are asked to ignore that the bucket into which that drop is to fall (or not) is nearly half empty (no, not half full… well… actually… about 62% full) to begin with.  That the murder who stabs to death a victim one day, and comes back to stab the already dead body one more time the next, is not guilty of any marginal crime for his actions on the second day.Perhaps not in the final legal analysis. I’ll leave that for the judge to figure out. But in my view, it’s still pretty damned offensive.

This post comes about as a follow up to a previous post where I critiqued the rationale of the Students First policy agenda.  It should be noted that the Students First policy agenda is anything but unique. Like DFER, SFER, ALEC or any policy advocacy organization, the SF policy agenda is little more than an aggregation of largely non-original, template policy prescriptions.

Now, I’m not one who goes all in for the lingo of “corporate reform” or one who perceives “privatization” or “market” mechanisms to be inherently evil and contrary to the public good. However, I am someone who believes we should consider carefully the multitude of tradeoffs involved in shifting between publicness and privateness in the governance and provision of schooling.

What I have found most intriguing over time is that the central messaging of these reformy template policy prescriptions is that they will necessarily improve accountability and transparency of education systems, and that they will do so largely by improving the responsiveness of those intractable systems through altered governance and finance, including but not limited to “market” based choice mechanisms.

The standard list of strategies that are supposedly designed to increase accountability and transparency of our education system include, among other things:

1. Expansion of charter schools, coupled with multiple charter authorizers (including private entities) and minimized charter regulation
2. Adoption of tuition tax credit programs providing individuals and corporations the option to forgo paying a portion of taxes by contributing that amount to a privately governed entity (or entities) that manages tuition scholarships to privately governed/managed schools.
3. Parent trigger policies that permit a simple majority of parents of children currently attending any school within a district to mandate that the local board of education displace the entire staff of the school and potentially turn over governance and management of school’s operations (and physical/capital assets?) to a private management company to be operated as a charter school.

It is argued that current large bureaucratic public education systems are simply intractable, non-responsive and can’t be improved – That they are simply not accountable to anyone because they are run by corrupt self-interested public officials elected by less than 2% of eligible voters (turnout for board elections) and that they have no incentive to be responsive because they are guaranteed a constantly growing pot of revenue regardless of performance/quality/responsiveness.

Whatever problems do exist with the design of our public bureaucracies, I would argue that we should exercise extreme caution in accepting uncritically the belief that we could not possibly do worse, and that large scale privatization and contracting of private entities to provide the public good is necessarily a better and more responsive, more efficient, transparent and accountable option.

Let’s take a walk-through of some of the key aspects of current preferred reforms by comparison to traditional public governance of our education systems.

Privately Governed/Managed Charter Schools vs. Local Education Agencies

Let’s begin with the push for less regulated, expansion of charter schooling with particular emphasis on expansion of privately governed and managed charter schools, and perhaps even charter schools authorized by independent private authorizers (granted authority to operate by a private entity given that authority by the state).  To be absolutely clear, no-matter how many reformy pundits proclaim from their soapbox that Charter Schools are PUBLIC Schools… it just isn’t that simple.  In many critically important ways, under many critically important conditions Charter Schools SIMPLY ARE NOT PUBLIC in every important traditional or legal sense!  See this post for further elaboration!

Note – this varies widely from state to state, depending on whether state charter statutes specifically spell out requirements of privately governed charter schools. 

Let’s explore how/why this might be important when it comes to evaluating whether and how expanded, less regulated chartering either increases or decreases public accountability.

Table 1. Chartering vs. Traditional District Schooling

These differences are not trivial, yet few are discussing them as critical factors for shaping future education policy. Rather, day after day, week after week, we are subjected to more and more vacuous punditry by self-proclaimed “expert” pundits displaying an astounding ignorance of education law and callous disregard for our system of government and the U.S. Constitution.

For example, it would appear that charter schools that are not “state actors” (which may include most that are governed by boards of private citizens and especially those managed by private companies/EMOs or CMOs) may require students to abide by disciplinary/conduct codes which involve compelling those students to recite belief statements about the school (mottos, pledges, loyalty oaths), obligatory participation in indoctrination activities and imposition of financial penalties for disciplinary infractions, none of which would be permissible in traditional public schools. Government entities – state actors – may not compel speech and especially may not compel statements of belief.

So then, what is a family to do when no traditional public schools are available to them (as is practically the case in many areas of New Orleans and increasingly the case in other higher charter market share cities)? Should parents have to choose which rights to forgo? [picking the school with the financial penalties over the one requiring daily recitation of a loyalty oath?]

Can (as some belligerent civic illiterate,  pundits believe) entire urban school systems be replaced with charter schools – or the traditional public schools adopt the lessons of “chartering” which involve infringement of constitutional rights? Is it reasonable to assume that the entire student population of a city would be placed in a position of necessarily forgoing their rights to free expression, free exercise?

I hear those reformy pundits cry… “but who cares about a little constitutional protection here and there if we can squeeze out an extra point or two on state assessments [via selective attrition of low performing peers]? They’ll be better for it in the long run!”

Yeah… sure… that’s all well and good for someone else’s kids. I for one believe the constitution continues to have a purpose and that constitutional rights should be equally available to all people’s children. I believe that constitutional protections are a key element of an accountable education system available to all – not just some.

This is a big freakin’ deal. An important policy trade-off to consider, if you will. This is a critically important tradeoff to consider when adopting policies that expand non-state-actor charter schooling, even if some marginal academic gain can be achieved.

Indeed, under our current public schooling system constitutional battles over free exercise, free speech, discrimination, etc. persist (as any good pro-school-choice libertarian will frequently argue – I, being a former card carrying member in my NH days!). It’s a never ending tension between the preferences of the majority vs. the rights and interests of the minority. Such arguments are often used as the basis for saying that all students/families should simply have the right/option to choose where to attend school – where they can each be their own majority.  The value of our current public (gov’m’nt) system is that the minority does have the right to challenge their mistreatment and that collective participation in the public system forces public debate over these issues (even if/when they end up being handled poorly). It ain’t perfect, but I’m not willing to replace it with a system that requires large numbers of children to forgo these rights in order to participate in schooling.

Poor and minority children should not be disproportionately required to forgo constitutional protections (and a variety of statutory protections) to gain access to those few additional test score points. Further, no-one is telling them that they even have rights to begin with – especially those pitching the charter expansion policies (constantly spewing the rhetoric of the “publicness” of charter schooling).

Charter Schooling & The Market for Lemons

In theory, the accountability and efficiency advantage of charter schooling is driven by the market for choice of one school over another. Increasingly, state education agencies have moved from being impartial technical assistance agencies and accountability reporting agencies to strongly promoting the charter sector. This advocacy behavior corrupts the state agency role and creates what economists refer to as an “asymmetry of information” – in the extreme case a “market for lemons.”

Markets fail when the consumer is misled to believe that the product they are being sold is a miracle product (without counterbalancing information available to the “customer”). Asymmetry of information occurs where the seller has more information on the quality of the product than the buyer and is able to extract from the buyer a higher price than is warranted given the product’s true quality. In this case, we are talking about the parents’ choice to apply their child’s gov’t subsidized education credit, per se, at a charter versus the traditional public school.  They’ve got one credit to spend for each child and the SEA endorsed spin these days is that that credit is nearly always best spent in a charter school (even when it clearly is not).

Taken to the extremes, State Education Agency and public media flaunting of chartery miracles has created a distorted market for those charters that are least proven on the market (perhaps in some cases, lemons), with those charters that are most proven already over-subscribed and not needing to compete openly. So, those most available on the market are those whose actual performance/quality is far lower than that which is capturing the headlines and receiving accolades from state officials. [not quite a true market for lemons since the price – education “credit” is fixed … though perhaps I can expand on this at a later point].

It is the absurd punditry, intentional obfuscation and complete disregard for legitimate data/analysis on charter schooling that have perhaps soured my taste for the movement more than anything else (bearing in mind that I was a founding member of the AERA special interest group on Charter School research and, at the time, was largely an advocate myself).

Tuition Tax Credits & Vouchers vs. Conventional LEA Governance

Next up, let’s talk about tuition tax credits and vouchers. Now, I would argue that in many ways, tuition tax credits and vouchers which provide the option for children to attend schools that are well understood to be private, that not state actors are at least more honest with respect to student and employee rights.

It is understood (or should be more clearly understood) that when choosing a private school or choosing to be employed by a private employer that one’s rights may differ. On very few occasions have I actually heard the rather absurd argument that private schools receiving students on publicly financed scholarships are “public” (yes, they did, without understanding the implications, make this claim in Louisiana when their voucher model was overturned by the state courts).

Now, let’s parse the governance and accountability differences between traditional public LEAs, Vouchers and Tuition Tax Credits.

Table 2. Vouchers & Tuition Tax Credits vs. Traditional District Schooling

The simple part here is that under either the tuition tax credit or voucher program, the schools that children attend are clearly private. It is (or at least should be) understood that students and employees forgo certain rights. As such, it would be plainly illogical to use such a model as the model for an entire city or state, meaning that children would not even have the option of attending a school where they are protected from discrimination and other forms oppression. [notably, while children/families may be oppressed and/or discriminated against by the ruling “majority” in a public school setting, they have a constitutional right to challenge their mistreatment – a right that ceases to exist where only private providers are available].

Other more nuanced delineations here are between the voucher and the tuition tax credit model. The more popular TTC approach is far more convoluted, and in being so, creates additional layers of opaque to non-existent accountability, ultimately negating altogether taxpayer legal rights.

Under a voucher model, like the Cleveland voucher model, taxpayers do have the right to challenge that their tax dollars are being allocated to religious education. Indeed, when such a challenge was brought, the U.S. Supreme Court decided that the voucher mechanism in place was sufficiently neutral (reliant on parental choice) that it did not violate the establishment clause of the U.S. Constitution. But, taxpayers at least had the right to bring this challenge even if they did lose.

What I find most objectionable (in terms of public accountability) about the TTC approach is that when a similar challenge was brought against the Arizona tuition tax credit model, the U.S. Supreme Court determined that the dollars being expended effectively weren’t the taxpayers’ dollars and thus the taxpayer had no right to bring a legal challenge to the policy (no taxpayer “standing”). Quite simply no taxpayer standing means NO taxpayer legal accountability. No taxpayer legal recourse. Arguing that TTC models increase public accountability is absurd.

Further, that these systems rely on creating non-public, non-publicly accountable entities to manage these funds diverted from the public coffers further reduces public accountability.

Parent Trigger vs. Conventional Local Education Agency Governance

Parent trigger is quite possibly the most ludicrous corruption of public governance and accountability on the education reformy education policy table.  Put simply, parent trigger is the most ill-conceived subversion of governance I’ve seen out there in the reformy playbook. Let’s give it a walk-through.

Table 3. Parent Trigger vs. Traditional District Schooling Governance

The most substantive reductions of public accountability, transparency and governance occur when the simple majority of parents of children in one school decide that there school must be converted to a privately managed charter school, which may in turn adopt policies that deprive both children and employees of constitutional and statutory rights. Indeed, the district would likely be required to find a school for the displaced minority of students who don’t wish to forgo these rights. But the simple majority of parents in that school at that point in time should not be granted the authority to displace a minority of students in their school. Further, a simple majority of parents in a school in a district should not be granted the authority to dictate local board funding or contracting policies without input of the broader eligible voter population.

Among other things, Parent Trigger policies assume that the public at large who reside and own property within a school district have no stake in the accountability of that school system. School closures, school quality, school location, etc. affect the value of residential properties by affecting quality of neighborhood life. Quite likely (an open empirical question) conversion to exclusive and/or specifically themed charter schools creates unique effects on property values and neighborhood quality of life, and not necessarily always positive effects.

Finally, schools/school buildings and property are public assets having a lifespan far exceeding that brief moment in time when that trigger pulling simple majority has children attending the school and the public that has invested in those schools over time should thus have some say in their operation, maintenance and management.

The idea that this particular subversion of traditional governance somehow heightens public accountability is simply ridiculous.

Closing Thoughts

Love it or hate it, we’ve got a pretty well defined, reasonably functional system of public governance in this country, with the overarching rule of the land being our U.S. Constitution. I’m not trying to oversell here. I’m not saying it’s perfect, always responsive to all and never intractable, opaque or corrupt. But I am saying that we could certainly do worse and many proposals on the table are likely to do just that.

Importantly, state laws might be written to close many of the gaping holes in student and employee rights identified above, public disclosure requirements and clarify the delineation between publicness and privateness. But the current trend is not necessarily in that direction!

Our current system defines the roles and responsibilities of public officials, holding them to public accountability standards vetted by our federal and state judicial branches for over two centuries. Yeah, I know, many of these reformy pundits would also simply do away with that meddling judicial branch. I for one, think that our courts continue to play a critical role in protecting rights.

Modern education reform efforts, in the name of supposed increased accountability and transparency largely seek to subvert our system of government as we know it and in many cases seek to strip large shares of poor and minority children and the employees in schools of poor and minority children of constitutional protections. And we’re all supposed to be okay with that?

It’s been a fun week for grading the states. First we had the wacky ratings from Students First which graded states largely on the extent to which they had adopted the preferred policies of that organization. Then we had the old-standard Education Week Quality Counts. When it comes to their finance rating system, little has changed in recent years. These two reports, of course, produced substantially conflicting results.

One might argue that both reports and ranking systems, like our School Funding Fairness report, include several indicators intended to identify policy conditions for success. This has been the standard response of Students First when they have been criticized on the basis that the states that they have applauded most tend to have pretty low average outcomes.  But, the Students First report, Quality Counts and our Funding Fairness report differ quite substantially on what we consider to be policy conditions for success. 

Students First has put policy conditions into three categories – 1) elevating the teaching profession, 2) parent empowerment and 3) finance and governance.  Students first gives no consideration across any of these categories to whether teacher wages, for example are sufficient to recruit/retain high quality candidates into teaching or whether wages are specifically competitive in high need schools. Students First gives no consideration to whether funding, overall, is sufficient to provide either/both competitive wages or reasonable class sizes, generally, or specifically in high need schools.  It would appear to be their opinion (as was rather clearly expressed by Eric Lerum in a video conference) that overall level or distribution of funding isn’t the issue – but rather that their preferred policies are what matters, regardless of funding (since the only funding/resource equity considerations in their rankings pertained to whether charter schools received what they consider equal funding – no validation provided!)

Education Week goes old school especially on their school finance rankings. I don’t have time/space to address all of their rankings. As I will show below, some of their old-school measures seem to capture relatively useful information, but others do not. Let’s quickly summarize the measures they use.

• Fiscal Neutrality: Fiscal neutrality measures the relationship between district spending and district wealth. State school finance formulas are partly intended to disrupt this relationship – reduce the likelihood that wealthier districts spend systematically more. This measure is often still useful, but may be complicated by the fact that school finance formulas also try to address differences in student needs and costs. To the extend that higher need kids live in poorer districts (not always the case that taxable property wealth and student need are tightly associated), this indicator may work to partly capture both.
• McLoone Index: Named for school finance legend Gene McLoone! This index tells us how close, on average, the per pupil spending of districts in the lower half (serving the lower half of kids) are to the median. That is, to what extent does the state formula succeed in “leveling up” the bottom half to the middle. A McLoone of 100 would mean that the lower half is equal to the middle. But this index in particular can produce some screwy results. Say for example a state has one or a few very large districts with high need populations and those districts constitute both the lower half and the middle (they have nearly or all of the bottom half of kids). A state with one or a handful of high need large districts with spending lower than everyone else (the upper half) might still get a McLoone of 100. But it would be a really crappy school finance system! (with all due respect to Gene!)
• Coefficient of Variation: The coefficient of variation simply measures the extent of variation in per pupil spending as a percent of the mean per pupil spending. A CV of 10% indicates that 2/3 of children attend districts with per pupil spending within 10% of the mean. The problem with the CV is that, while it measures variation, it doesn’t capture the difference between GOOD variation and BAD variation. Modern state school finance formulas try to create variation in funding to accommodate differences in student needs. Education Week uses nominal weights to “adjust” for differences in student needs, but some state school finance systems actually adjust more aggressively for needs than do their weights. Those states are penalized in the CV.
• Spending Index & Percent at/Above National Mean: A few reports back Education Week wanted to construct a form of “spending adequacy” figure to compare spending levels across states and the shares of kids with access to what they considered more “adequate” spending. So they adopted this measure and index based on the percent of children in each state who attended districts that spent at least the same as the national average district (spending adjusted for regional wage variation). This figure does generally capture spending level differences across states – adjusted for wage variation – but doesn’t, for example capture spending level differences corrected for student population differences, or the shares of students who might be attending very small, remote rural districts.

Ed week includes a few additional indicators like the restricted range – or difference in spending between the 95th and 5th %ile district, but these are largely redundant with the CV & McLoone and suffer the same problems of not accounting for other cost factors – or state aid formulas that aggressively adjust for needs and costs.

We had set out to correct for many of the problems in the Ed Week approach when we started work  on our Funding Fairness report. Specifically, we wanted to make comparisons that better accounted for differences in needs and costs across districts and states and that could be used to characterize state school finance policies consistently, without suffering some of the problems of old-school indicators like the CV or McLoone Index. We also look at spending level – using a statistical model based on 3 years of data to project the per pupil state and local revenue of a district with a) average poverty rate, b) in an average wage labor market and c) with 2,000 or more students and average population density. That is, our projected state and local revenue figures are adjusted for poverty, competitive wages, size and population density. We use the same model to then evaluate whether, on average – and in a predictable pattern – state and local revenues are systematically higher (progressive) or lower (regressive) in higher poverty districts (relative to lower poverty districts). That is, does the system overall target resources to higher poverty districts – controlling for the other factors.

That prerequisite discussion aside, let’s take a look at how all of this stuff lines up – How the Ed Week Indicators line up with the Funding Fairness Indicators and how both line up with the Students First Indicators. Finally, I look at how all line up with various outcome measures.

Again… all of these funding related indicators are about policy conditions for success, rather than success itself.

First up – and here’s a relative no-brainer – both our funding fairness report and Ed Week Quality Counts include an indicator of state funding effort – or share of state capacity allocated to elementary and secondary education.  I can’t speak for Ed Week, but we include ours to acknowledge that some states spend more than others (do better on our spending level measure) because they can and that we should grade them at least partly on their effort.  Figure 1 shows that our effort measure and Ed Week’s effort measure are pretty highly correlated.

Figure 1

Figure 2, by contrast, shows that the Students First funding GPA isn’t related at all with the Ed Weeks effort indicator, and by extension with ours. Ed Week (and we) consider funding effort to be an underlying policy condition for success, apparently, Students First doesn’t .

Figure 2

Figure 3 compares our funding level indicator and Education Week’s spending index – or relative adequacy indicator. Clearly the two are highly related… but the Ed Week indicator caps out at 100% – or where 100% of the children attend districts above the national average spending. Personally, I prefer indicators that capture the full range of variation.  But again, our spending level measure and Ed Week’s spending index are picking up much of the same information – relative spending differences across states.

Figure 3

But, Figure 4 shows that Students First finance rating scheme really doesn’t relate at all to Education Week’s spending index, suggesting that overall availability of resources – like the effort to raise them – is inconsequential in the eyes of Students First.

Figure 4

Figure 5 shows the relationship between our funding progressiveness indicator and Ed Week’s fiscal neutrality indicator. For many states, the two are picking up similar things. In states like New Jersey or Utah, where higher poverty districts have more resources than lower poverty ones, the systems have also achieved fiscal neutrality (disrupted the relationship between wealth and spending). By contrast, in states like Illinois or North Carolina, the wealth-spending relationship remains strong and positive (higher wealth – higher spending) and higher poverty districts receive systematically fewer resources!

Figure 5

Recall that Illinois received one of the best grades on finance from Students First. Apparently, in addition to effort and spending level, fiscal neutrality and need based funding are also inconsequential to Students First when it comes to funding issues. Figure 6 shows the relationship between funding progressiveness and Students Firsts funding related GPA. Note that all of Students First’s funding superstars (Illinois, New York, Rhode Island and Michigan) are less than stellar on funding fairness.

Figure 6

Figure 7 relates the Ed Week CV to our funding fairness measure, showing that states with progressive funding distributions including New Jersey, Ohio and Massachusetts are actually penalized by this measure.  The CV does not differentiate between need based variation as occurs in these states and wealth-drive variation as occurs in New Hampshire.  We all seem to agree – Ed Week, Students First and us… that New Hampshire’s funding is…well… not so good!

Figure 7

Moving on, here’s the relationship between our funding fairness measure and the McLoone Index! Not much going on here… and but for a few specific examples… it’s actually hard to tell what the McLoone really captures these days in complex state school finance systems. At least it captures that New Hampshire school funding… well… sucks! But other than that, the McLoone really doesn’t capture much valuable additional information regarding equity.

Figure 8

So then, how do these various policy conditions for success relate to various outcome measures. In this table and the following graphs I explore that question, using the following outcomes:

1. Reduction in % below proficient (from http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG12-03_CatchingUp.pdf)
2. Annual Standardized Gain (NAEP, from http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG12-03_CatchingUp.pdf)
3. Adjusted (for initial level) Annual Standardized Gain
4. Reading and Math NAEP 8th Grade 2011
5. Reading and Math NAEP 8th Grade for Lowest Income Group (Free Lunch) 2011

Table 1 shows the correlations between each of the indicators addressed above and the outcome measures listed above.  Note that each of these correlations a) is relatively modest to non-existent and b) merely represents a relationship whereby when X is higher, so too is Y. Underlying causal relationships involve a complex web of factors including socio-economic and demographic conditions, etc.

Table 1

Figure 9 ranks the correlations between policy conditions and reduction in % below proficient at the 8th grade level. Interestingly, variation (inequity – bad and good) in spending is most positively associated with reduction in % below proficient. Beyond that, our funding level indicator and the two funding level indicators from Ed Week are next in line.  Students First’s teaching profession indicator is next… but their funding indicator further down. The figure seems to suggest that higher spending states, even where that spending is unequal, are doing okay on reducing % below proficient – but this is a pattern that can clearly be influenced by regional variation.

Figure 9

Figure 10 shows the correlations – ranked high to low – between each policy condition and adjusted standardized gain. In this case, adjusted standardized gains are most highly correlated with our spending level indicator, the Ed Week spending adequacy indicator, and our progressiveness indicator and Ed Week’s neutrality indicator. One might infer from this that more equitable and adequate funding is associated with greater long term average gains on NAEP… but again, regional differences may drive this to an extent. To get an idea of which states have better “adjusted annual gains” see the figure in Appendix A. Higher adjusted gains are states above the trendline and lower adjusted gains are those below the trendline.  Not all states are included (in the graph or correlations) for lack of baseline data year (I may work on updating this with multiple baseline years & tests. This is just a start).

Figure 10

Finally, we have Figure 11, which compares correlations with the NAEP scores of the lowest income children (which across states were not associated with the average income of the families of those children). These are children in families below the 130% income level for poverty.  As in Figure 9, states with the greatest spending variation seemed to have higher low income NAEP scores. Beyond that however, funding level, effort and wage competitiveness (Teaching Penalty data) seem to be positively correlated with low income NAEP scores. That is, states with higher funding levels, that put up more funding effort, and that have more competitive teacher wages (weekly, relative to non-teachers) have higher low income NAEP scores.

In Figure 11, all of the students first indicators (GPAs) are negatively associated with low income student NAEP scores. That is, low income children are doing much worse in states that got good grades from Students First.  That said, many of the conditions Students First included as setting the state for future success are policies only recently implemented in these states.

Figure 11

So that’s it… my run down of the relationship between this week’s state rankings data, how they relate (or not) to our School Funding Fairness Report and how they relate to various outcome measures. I’ll let the rest of you run with it from here! Cheers!

Data Sources:

Students First Report Card

School Funding Fairness

Ed Week Quality Counts (Finance)

Teaching Penalty

Relevant Additional Readings

Appendix A: NAEP Standardized Gains and 1990 Scores

On Monday, the organization Students First came out with their state policy rankings, just in time to promote their policy agenda in state legislatures across the country. Let’s be clear, Students First’s state policy rankings are based on a list of what Students First thinks states should do. It’s entirely about their political preferences – largely reformy policies – template stuff that has been sweeping the reformiest states over the past few years. I’ll have more to say about these preferred policies at the end of this post.

Others have already pointed out that Students First gave good grades to states like Louisiana and Florida, and crummy grades to states like New Jersey or Massachusetts – but that states like Louisiana have notoriously among the worst school systems – lowest test scores – in the nation – whereas states like New Jersey and Massachusetts have pretty darn good test scores and well respected school systems. I’ll go there as well, but not as my primary focus. Clearly there’s  more behind the test score differences than policy context. New Jersey and Massachusetts certainly have more educated, more affluent adult and parent populations than Louisiana, and that makes a difference.

I’ll anxiously await the day good reformers like Mike Petrilli pack their bags and leave their suburban Washington DC districts to move their kids to the amazing future schools of Louisiana!  Heck, given these new Students First ratings, any Louisiana school has to be better- or at least have far more potential – than any school in Montgomery County Maryland, run by that curmudgeonly anti-reformer Suprintendent Joshua Starr!  In fact, in my ideal world, Louisiana would become a reformy wonderland… magnet for all the reformy types… where they could go live in peace – rate their teachers by value-added models, fire 10 to 20% each year – pay them nothing – get rid of any retirement benefits, make every school a charter school (operating primarily with imported Turkish and/or Filipino labor), engage in at least 50% online learning (sitting at a computer doing test prep modules), and provide tuition tax credits to all of the reformies who prefer a purely religious perspective interwoven across subjects. Of course, this must all be done with Lousiana’s current level of financial commitment to public schooling.

But I digress… Now back to the Students First ratings.  Students First created 3 broad categories of preferred policies for their ratings – policies that it believes:

1. Elevate teaching
2. Empower parents
3. Spend wisely and govern well

By elevate teaching, Students First means the usual basket of reformy options including elimination of traditional salary schedules, teacher evaluations based heavily on student test scores, reduction of retirement benefits and reduction or elimination of due process rights, and pay based primarily on test-score driven evaluation systems. They also prefer to expand alternative routes into the teaching profession. Of course, there’s not a whole lot of transparency into how these various elements are factored into the final grades. But there is a rubric!

By empowering parents, Students First essentially means increasing use of school report cards (more school grades & ratings – yes, a report card that endorses use of … report cards!), reporting to parents when their child is assigned to a teacher with a low rating (driven by test scores), and adoption of policies such as Parent Trigger. And of course, charters should be provided everywhere and anywhere… so everyone has the choice to attend one.

Finally, by spend wisely and govern well… I’m quite honestly not even sure what the hell they mean? They include a broad statement about all kids receiving equitable funding, but seem to imply that this means that charter kids get the same funding as district kids – a pretty narrow interpretation of fairness (and an incalculable one in states with no charters). No actual data seem to be used to rate state funding systems. Fairness is also determined by the provision of publicly finance facilities space to charter schools. Somehow, their ratings of funding totally ignore the vast majority of schools and children across which funds are distributed. In their view, Mayors, not local school boards should govern schools, and schools should report their expenditures uniformly – in a way that shows how the spending affects achievement (good luck with that as a reporting requirement/mechanism).

Every item on their list is somehow mysteriously scored on a “0” (you suck) to “4” (wow… you are REFORMERIFIC!) scale without using any actual data (apparently) to inform that ordinal rating. Then in a wonderful leap of number abuse, these ordinal scale data are averaged to create a grade point average for each broad category – on a 0-4 GPA like scale, where most values of course lie in the imaginary spaces between the original ordinal ratings (like kinda-semi-almost-reformerific = 3.49).

That said, let’s dig into those grades, and other stuff that may or may not correlate with them.

RheeFormy Funding Indicators vs. Real Funding Indicators

Let’s start with the funding grades which I will relate to two of our primary indicators in our annual report on school funding fairness. First, let’s look at the relationship between the RheeFormy GPA for funding/governance and our rating on the percent of gross state product allocated to K-12 education. Top scorers on RheeFormy funding are Michigan, Rhode Island, New York, Alaska and Illinois. Right away this is rather absurd since New York and Illinois are quite well known to have among the least equitable school funding systems in the nation…. but that’s the next graph. RheeFormy winners Florida and Louisiana are not particular standouts on their funding effort to education, whereas the states of New Jersey and Massachusetts, despite being much richer to begin with, allocate a much larger share of their economic productivity to elementary and secondary education. But hey, why would anyone want to count how much effort a state actually puts into funding its schools? right? how could that matter?

Figure 1.

Figure 2 compares the RheeFormy finance GPA to our indicator of funding fairness. Our indicator of funding fairness compares the projected state and local revenue of high poverty school districts to that of low poverty school districts. A value of greater than 1.0 indicates a progressive system where more resources are allocated to higher poverty districts and a value of less than 1.0 indicates a regressive system. Not surprisingly, the RheeFormy standouts of Florida and Louisiana and especially New York and Illinois (which are among the top in RheeFormy finance) are all highly regressive states. Uh… that’s bad… not good. High poverty districts get the shaft in these states. But that’s apparently just fine with Students First. In fact, it seems preferable! Way to go!

Meanwhile, New Jersey and Massachusetts, along with Ohio, are relatively progressive on finance. Yes, Utah is too, but that’s because Utah spends next to nothing on most schools and slightly more than next to nothing on lower income schools.

So, apparently, RheeFormy logic dictates that to really make progress on achievement in low income communities, money really doesn’t matter. State school finance systems don’t matter, and in fact, spending less on districts with more poor children is the way to go. Good for New York and Illinois! NOT!

Figure 2.

What we know about school finance reforms, funding level & distribution & student outcomes

…sustained improvements to the level and distribution of funding across local public school districts can lead to improvements in the level and distribution of student outcomes. While money alone may not be the answer, adequate and equitable distributions of financial inputs to schooling provide a necessary underlying condition for improving adequacy and equity of outcomes. That is, if the money isn’t there, schools and districts simply don’t have a “leverage option” that can support strategies that might improve student outcomes. If the money is there, they can use it productively; if it’s not, they can’t. But, even if they have the money, there’s no guarantee that they will. Evidence from Massachusetts, in particular, suggests that appropriate combinations of more funding with more accountability may be most promising.

See: http://www.shankerinstitute.org/images/doesmoneymatter_final.pdf

Put bluntly – equitable and adequate financing for the education of all children is a prerequisite condition for achieving equitable and adequate outcomes. The Students First rating system misses this point entirely – measuring neither the equity nor adequacy – nor effort to raise these prerequisite resources.

No, the Students First ratings don’t pretend to measure these things. But, they seem to argue that their ratings measure the prerequisite conditions for reforming education systems – where I must assume they mean making those systems better. But the reality is that the ratings focus on trivial and tangential reformy preferences, leading them to praise states that are among the worst in the nation on school funding and chastise states that are among the best.

RheeFormy Teacher Quality Indicators vs. Competitive Compensation

Next up are the RheeFormy ratings on elevating the teaching profession, where quite clearly, having competitive wages for teachers (relatively to the workforce of similarly educated workers) is a non-issue. Figure 3 shows the relationship between the relative weekly wage of teachers – compared to same education level non-teachers – and the elevating teaching GPA.  Teachers in Louisiana have among the largest “teaching penalties” earning only about 70% of the weekly wage of non-teachers. That’ll certainly elevate the profession! They are right up there with other stellar reformy states including Colorado and Tennessee.  Teachers in Florida do a bit better. Teachers in Massachusetts don’t do that well either, but Massachusetts is a state where non-teacher wages are quite high. So being relatively low in Massachusetts might not be as bad as being relatively low in Louisiana or Tennessee! (They also have the benefit of being able to send their own kids to pretty good public schools.) Teacher wages in New Jersey are more competitive, even in their higher non-teacher wage competitive context.

Figure 3.

In really simple terms – competitive wages are the first step toward elevating the teaching profession. Given what we actually know from research on “elevating the teaching profession” I don’t expect to see America’s best teachers flocking to Louisiana, Colorado and Tennessee anytime soon. But, I certainly hope to see those reformers in their caravan, moving their families to those reformy promise lands!

What we know about policy conditions for a strong teacher workforce

A substantial body of literature has accumulated to validate the conclusion that both teachers’ overall wages and relative wages affect the quality of those who choose to enter the teaching profession, and whether they stay once they get in. For example, Murnane and Olson (1989) found that salaries affect the decision to enter teaching and the duration of the teaching career,[i] while Figlio (1997, 2002) and Ferguson (1991) concluded that higher salaries are associated with more qualified teachers.[ii] In addition, more recent studies have tackled the specific issues of relative pay noted above. Loeb and Page showed that:

“Once we adjust for labor market factors, we estimate that raising teacher wages by 10 percent reduces high school dropout rates by 3 percent to 4 percent. Our findings suggest that previous studies have failed to produce robust estimates because they lack adequate controls for non-wage aspects of teaching and market differences in alternative occupational opportunities.”[iii]

In short, while salaries are not the only factor involved, they do affect the quality of the teaching workforce, which in turn affects student outcomes.

Research on the flip side of this issue – evaluating spending constraints or reductions – reveals the potential harm to teaching quality that flows from leveling down or reducing spending. For example, David Figlio and Kim Rueben (2001) note that, “Using data from the National Center for Education Statistics we find that tax limits systematically reduce the average quality of education majors, as well as new public school teachers in states that have passed these limits.”[iv]

Salaries also play a potentially important role in improving the equity of student outcomes. While several studies show that higher salaries relative to labor market norms can draw higher quality candidates into teaching, the evidence also indicates that relative teacher salaries across schools and districts may influence the distribution of teaching quality. For example, Ondrich, Pas and Yinger (2008) “find that teachers in districts with higher salaries relative to non-teaching salaries in the same county are less likely to leave teaching and that a teacher is less likely to change districts when he or she teaches in a district near the top of the teacher salary distribution in that county.”[v]

And what do we know about the effectiveness of the preferred Students First policies of providing performance based pay? For recent studies specifically on the topic of “merit pay,” each of which generally finds no positive effects of merit pay on student outcomes, see:

• Glazerman, S., Seifullah, A. (2010) An Evaluation of the Teacher Advancement Program in Chicago: Year Two Impact Report. Mathematica Policy Research Institute. 6319-520
• Springer, M.G., Ballou, D., Hamilton, L., Le, V., Lockwood, J.R., McCaffrey, D., Pepper, M., and Stecher, B. (2010). Teacher Pay for Performance: Experimental Evidence from the Project on Incentives in Teaching. Nashville, TN: National Center on Performance Incentives at Vanderbilt University.
• Marsh, J. A., Springer, M. G., McCaffrey, D. F., Yuan, K., Epstein, S., Koppich, J., Kalra, N., DiMartino, C., & Peng, A. (2011). A Big Apple for Educators: New York City’s Experiment with Schoolwide Performance Bonuses. Final Evaluation Report. RAND Corporation & Vanderbilt University.

A Comment on Accountability, Empowerment, Transparency & Students First Preferred Policies

Finally,  I close with a topic that should be another blog post altogether, and likely will be at some point. I’ve been struck by the logic that the preferred policies in the Students First report are intended – by their framing – to increase accountability, empowerment and transparency. Yet, in all likelihood, most of these proposals accomplish precisely the opposite – substantially eroding public accountability and oversight and compromising statutory and constitutional rights of children, employees and local taxpayers.

Now, we may have our differing perspectives on the structure of our American government and operation of government entities. But, our government has a defined structure with reasonably well conceived overarching laws.

The U.S. Constitution, state constitutions and various federal and state statutes provide important protections to students and employees and the taxpayers that finance public institutions. Importantly, our constitution protects individuals from certain treatments by our government and agents of our government. Public schools – government schools to borrow from libertarian rhetoric – fall under that umbrella, and must, for example, provide children due process before depriving them the right to attend, and must respect – to a limited extent – students rights to free expression, etc. Government schools also cannot promote/endorse a particular religious viewpoint (proselytize).  Other protections, many protections of both children and employees of government institutions are invoked through Section 1983 of the U.S. Code which applies to entities that are ‘state actors’ (uh… government entities). Further, many state laws apply to government entities and to ‘public officials.’

We have a representative system of government with multiple levels, where public officials are elected by (and accountable to) voters (albeit often a small share of those eligible), and where additional layers of public officials may be appointed by elected public officials. And, as noted above, many laws, especially those pertaining to public disclosure, public meetings and public records apply to ‘public officials.’

The Students First state policy rating system – like many other reformy manifestos – implies that the road to ACCOUNTABILITY and TRANSPARENCY is necessarily (perhaps exclusively) paved through shifting larger numbers of students and teachers and larger shares of public funding over to the management of non-government entities and non-public officials, as well as creating entirely new layers of ‘public decision making’ by referendum/petition (Parent Trigger).  Whatever gripes we may have regarding the efficiency or responsiveness of government operated services, we must think this one through carefully.

Unless detailed accountability requirements are explicitly spelled out in a whole new layer of state and federal laws, the preferred policies laid out in the Students First and by other reformy institutions are more likely to lead to less public accountability and transparency rather than more. For example:

1. Shifting substantial numbers of students into private schools or privately managed charter schools means that larger shares of students will have limited constitutional and statutory protections.  When students are educated under privately managed schools – including charters – they do not (unless explicitly laid out in state charter laws) have the same constitutional protections with respect to discipline policies and they lack other important statutory protections that apply only to “state actors” (government institutions). Indeed, parents have a choice of whether to forgo these rights for their children. BUT…. advocates of these policies are deceitfully selling charter schooling as ‘public’ (with all the rights, privileges, etc.) to an unknowing public.
2. Shifting substantial shares of public financing to entities governed by appointed boards of private citizens (not public officials), private management firms and private subcontractors reduces financial transparency because these institutions and individuals may – and most often do when challenged – invoke that they are not subject to open meetings, open records and other disclosure laws that necessarily apply to government entities.

Further, specific policies including parent trigger policies and ‘opportunity scholarship’ tuition tax credits may also substantially erode public accountability.

1. Parent trigger attempts to subvert traditional elected representative local government by granting disproportionate power to a temporary class of citizens – parents of children attending a school at a given moment in time – to make relatively permanent decisions, by simply majority rule, regarding operations of public assets, public programs and services, including the option to make them no-longer public. But these assets, programs and services belong to and serve directly and indirectly the larger community of eligible voters who put in place their elected school board (or city government officials that appointed a portion or all of that board).
2. Establishment of privately governed  entities to manage funds collected through a tax credit program [Opportunity Scholarship Vouchers] is comparable to simply handing over an equivalent sum of funds collected as tax dollars to that entity, except that taxpayers lose any/all rights/accountability over the use of those funds! As with private management firms and schools, private entities of this type are not governed directly by public officials and therefore may not be similarly legally accountable. They may not be subject to the same level of public records or meetings disclosure, etc. This mechanism [Tuition Tax Credit] has been used as a means to get around constitutional concerns over allocation of public tax revenues to religious institutions. That is, this mechanism was created specifically to negate taxpayer standing to mount legal challenges over the use of funds. The U.S. Supreme Court has determined that when tax credit programs are structured in this way, taxpayers have no right [no standing] to bring constitutional (or likely any) challenges over the use of these funds. [http://www.supremecourt.gov/opinions/10pdf/09-987.pdf]

So yes – Students First has their policy preferences – and they’re certainly entitled to that. They’ve built their entire rating system on their idea of what’s good policy. They’ve not tried to justify their policy preferences in any research basis on effectiveness or efficiency of these policy preferences, nor could they.  There simply is no research basis to support the vast majority of their preferences. Even where Charter school policy is concerned, findings of successful charters seem to occur most often where authorizers are few and tightly regulated, and where charter market share is low (as in NYC or Boston).  This is in direct contrast with the SF preference for further deregulating and expanding the sector (as in states with relatively poor charter performance).   So, in short, there’s simply no research based reason to follow the policy agenda of Students First.  But the reasons they provide – accountability, transparency, blah… blah… blah… are also not consistent with their policy agenda.

As a school finance researcher in particular, I’ve been increasingly frustrated by the lack of detailed consistent financial reporting on charter schools, and I’ve written much on this topic. I’ve also written on private school financing, which is even more sparsely reported. The more kids who are shifted into charters, the fewer kids on which we have reliable, comprehensive information on finances, teacher contracts, compensation packages etc. (as charters management companies repeatedly invoke that their employee contracts are private, not public documents).  Similarly, financial arrangements involving land deals and capital financing are more opaque than ever – far more opaque and inaccessible than the public financing world of municipal bond financed infrastructure. And I don’t see any legitimate effort to make these institutions more transparent – NONE!