Knowing that I’ve been writing a fair amount about various methods for attributing student achievement to their teachers, several colleagues forwarded to me the recently released standards of the Council For the Accreditation of Educator Preparation, or CAEP. Specifically, several colleagues pointed me toward Standard 4.1 Impact on Student Learning:

4.1.The provider documents, using value-added measures where available, other state-supported P-12 impact measures, and any other measures constructed by the provider, that program completers contribute to an expected level of P-12 student growth.

http://caepnet.org/commission/standards/standard4/

Now, it’s one thing when relatively under-informed pundits, think tankers, politicians and their policy advisors pitch a misguided use of statistical information for immediate policy adoption. It’s yet another when professional organizations are complicit in this misguided use. There’s just no excuse for that! (political pressure, public polling data, or otherwise)

The problems associated with attempting to derive any reasonable conclusions about teacher preparation program quality based on value-added or student growth data (of the students they teach in their first assignments) are insurmountable from a research perspective.

Worse, the perverse incentives likely induced by such a policy are far more likely to do real harm than any good, when it comes to the distribution of teacher and teaching quality across school settings within states.

First and foremost, the idea that we can draw this simple line below between preparation and practice contradicts nearly every reality of modern day teacher credentialing and progress into and through the profession:

one teacher prep institution –> one teacher –> one job in one school –> one representative group of students

The modern day teacher collects multiple credentials from multiple institutions, may switch jobs a handful of times early in his/her career and may serve a very specific type of student, unlike those taught by either peers from the same credentialing program or those from other credentialing programs. This model also relies heavily on minimal to no migration of teachers across state borders (well, either little or none, or a ton of it, so that a state would have a large enough share of teachers from specific out of state institutions to compare). I discuss these issues in earlier posts.

Setting aside that none of the oversimplified assumptions of the linear diagram above hold (a lot to ignore!), let’s probe the more geeky technical issues of trying to use VAM to evaluate ed school effectiveness.

There exist a handful of recent studies which attempt to tease out certification program effects on graduate’s student’s outcomes, most of which encounter the same problems. Here’s a look at one of the better studies on this topic.

• Mihaly, K., McCaffrey, D. F., Sass, T. R., & Lockwood, J. R. (2012). Where You Come From or Where You Go?

Specifically, this study tries to tease out the problem that arises when graduates of credentialing programs don’t sort evenly across a state. In other words, a problem that ALWAYS occurs in reality!

Researchy language tends to downplay these problems by phrasing them only in technical terms and always assuming there is some way to overcome them with statistical tweak or two. Sometimes there just isn’t and this is one of those times!

Let’s dig in. Here’s a breakdown of the abstract:

In this paper we consider the challenges and implications of controlling for school contextual bias when modeling teacher preparation program effects. Because teachers from any one preparation program are hired in more than one school and teachers are not randomly distributed across schools, failing to account for contextual factors in achievement models could bias preparation program estimates.

Okay, that’s a significant problem! Teachers from specific prep institutions are certainly not likely to end up randomly distributed across a state, are they? And if they don’t, the estimates of program effectiveness could be “biased.” That is, the estimates are wrong! Too high, or to low, due to where their grads went as opposed to how “good” they were. Okay, so what’s the best way to fix that, assuming you can’t randomly assign all of the teacher grads to similar schools/jobs?

Including school fixed effects controls for school environment by relying on differences among student outcomes within the same schools to identify the program effects.  However, the fixed effect specification may be unidentified, imprecise or biased if certain data requirements are not met.

That means, that the most legit way to compare teachers across programs is if you can compare teachers whose first placements are in the same schools, and ideally where they serve similar groups of kids. And, you’d have to have a large enough sample size at the lowest level of analysis – comparable classrooms within school – to accomplish this goal. So, the best way to compare teachers across prep programs is to have enough of them, from each and every program, in each school, teaching similar kids similar subjects at the same grade level, across grade levels. Hmmmm…. How often are we really likely to meet this data requirement?

Using statewide data from Florida, we examine whether the inclusion of school fixed effects is feasible in this setting, the sensitivity of the estimates to assumptions underlying for fixed effects, and what their inclusion implies about the precision of the preparation program estimates. We also examine whether restricting the estimation sample to inexperienced teachers and whether shortening the data window impacts the magnitude and precision of preparation program effects. Finally, we compare the ranking of preparation programs based on models with no school controls, school covariates and school fixed effects. We find that some preparation program rankings are significantly affected by the model specification. We discuss the implications of these results for policymakers.

With “no school” controls means not accounting at all for differences in the schools where grads teach. With “covariates” means correcting in the model for the measured characteristics of the kids in the schools – so – trying to compare teachers who teach in similar – by measured characteristics – schools. But, measured characteristics often fail to catch all the substantive differences between schools/classrooms.  And where “school fixed” effects means comparing graduates from different institutions who teach in the same school (though not necessarily the same types of kids!).

Okay, so the authors tested their “best” methodological alternative (comparing teachers within schools, by school “fixed” effect) with other approaches, including making no adjustment for where teachers went, or making adjustments based on the characteristics of the schools, even if not matched exactly.

The authors found that the less good alternatives were, to no surprise, less good- potentially biased. The assumption being that the fixed effect models are most correct (which doesn’t, however, guarantee that they are right!).

So, if one can only legitimately (though really not in this case either) compare teacher prep programs in cases where grads across programs are concentrated in the same schools for their first jobs, that’s a pretty severe limitation. How many job openings are there in a specific grade range in a specific school in a given year – or even over a five year period? And how likely is it that those openings can be filled with one teacher each from each teacher prep institution. But wait, really we need more than one from each to do any legit statistical comparison – and ideally we need for this pattern to be replicated over and over across several schools. In other words, the constraint imposed to achieve the “best case” model in this study is a constraint that is unlikely to ever be met for more than a handful of large teacher prep institutions concentrated in a single metropolitan area (or very large state like Florida).

Other recent studies have not found VAM particularly useful in parsing program effects:

We compare teacher preparation programs in Missouri based on the effectiveness of their graduates in the classroom. The differences in effectiveness between teachers from different preparation programs are very small. In fact, virtually all of the variation in teacher effectiveness comes from within-program differences between teachers. Prior research has overstated differences in teacher performance across preparation programs for several reasons, most notably because some sampling variability in the data has been incorrectly attributed to the preparation programs.

Koedel, C., Parsons, E., Podgursky, M., & Ehle, M. (2012). Teacher Preparation Programs and Teacher Quality: Are There Real Differences Across Programs? (No. 1204).

http://econ.missouri.edu/working-papers/2012/WP1204_koedel_et_al.pdf

Example from Kansas

Let’s use the state of Kansas and graduates over a five year period from the state’s major teacher producing institutions to see just how problematic it is to assume that teacher preparation institutions in a given state will produce sufficient numbers of teachers who teach in the same schools as graduates of other programs.

All programs

Specific programs

Indeed, the overlap in more population dense states is somewhat more significant, but still unlikely sufficient to meet the high demands of the fixed effects specification (where you can only essentially compare when you have graduates of different programs working in the same school together, in similar assignments… presumably similar number of years out of their prep programs).

Strategically Gaming Crappy, Biased Measures of “Student Growth”

In practice, I doubt most schools of ed, or state education agencies will actually consider how to best model program effectiveness with these measures. They likely won’t even bother with this technically geeky question of the fixed effects model, and data demands to apply that model. Rather, they’ll be taking existing state provided growth scores or value-added estimates and aggregating them across their graduates.

Given the varied, often poor quality of state adopted metrics, the potential for CAEP Standard 4.1 to decay into absurd gaming is quite high. In fact, I’ve got a gaming recommendation right here for teacher preparation institutions in New York State.

We know from the state’s own consultant analyzing the growth percentile data that:

Despite the model conditioning on prior year test scores, schools and teachers with students who had higher prior year test scores, on average, had higher MGPs. Teachers of classes with higher percentages of economically disadvantaged students had lower MGPs. (p. 1) https://schoolfinance101.com/wp-content/uploads/2012/11/growth-model-11-12-air-technical-report.pdf

We also know from this same technical report that the bias appears to strengthen with aggregation to the school level. It may also strengthen with aggregation across similar schools. And this is after conditioning the model on income status and disability status.

As such, it is in the accreditation interest of any New York State teacher prep institution to place as many grads as possible into lower poverty schools, especially those with fewer children with disabilities. By extension, it is therefore also in the accreditation interest of NY State teacher prep institutions to reduce the numbers of teachers they prepare in the field of special education. As it turns out, the New York State growth percentiles are also highly associated with initial scores – higher initial average scores are positively associated with higher growth. So, getting grads into relatively higher performing schools might be advantageous.

With a little statistical savvy, a few good scatteplots, one can easily mine the biases of any state’s student growth metrics to determine how to best game them in support of CAEP standard 4.1.

Further, because it is nearly if not entirely impossible to use these data to legitimately compare program effects, the best one can do is to find the most advantageous illegitimate approach.

Are these really the incentives we’re looking for?

First, here’s a primer and reading list on the Empire State of School Finance:

1. New York State maintains one of the least equitable state school finance systems in the nation
2. New York State actually allocates a ton of state aid to districts that need it least, exacerbating the disparities
3. Reformy types in New York State thought, under these circumstances, it would be really cool to make any additional state aid a district receives contingent on adopting a teacher evaluation scheme based on their documented deeply flawed metrics!
4. To ice that reformy cake, the legislature saw fit to – after slashing state aid year after year – impose a local property tax limit on districts so that they are unable to even raise the funds they would need to provide a sound basic education, if they could raise those funds locally.

ohhh… but I’m just getting started here.  Then came the lawsuits. That’s what makes this so fun and interesting to watch.

Now, there is already a pending lawsuit challenging the overall adequacy of state funding in New York specifically for high need cities (brought by the state’s small city school districts).

More recently however, we’ve been hearing of two separate cases.

First, we have the state teacher’s union (as reported) suing the state over the imposition of the property tax cap, which, in effect prohibits many districts from making up the difference from the aid they’ve been screwed out of for the past several years – the aid that in theory – by the state’s own definition of its foundation formula – would provide for a sound basic education. That formula was implemented specifically to comply with a previous court order in Campaign for Fiscal Equity. 

Next, we have the lawsuit brought on behalf of children in New York City schools challenging the state’s authority to reduce the city’s funding by $250 million for non-compliance with adopting a teacher evaluation policy.

So far, it would appear that this argument has achieved a positive, immediate response from the judge, who a this stage has blocked the state funding reduction.
As laid out in full here: http://schoolfunding.info/wp-content/uploads/2013/02/Memorandum-of-Law-in-Opp-to-App-for-Prelim-Inj1.pdfAnd as characterized here: http://schoolfunding.info/2013/02/miriam-aristy-v-state-of-new-york/

Assistant Attorney General Steven Schulman described the $250 million that the state will cut from NYC schools as a “drop in the bucket” and argued that it was not great enough to have any effect on schools’ ability to provide a sound basic education.

The state’s defense of its actions is essentially that $250 million really isn’t that much money for New York City and certainly doesn’t deprive NYC schoolchildren of receiving their constitutionally mandated sound, basic education. And that forcing the state to provide the $250 million would undermine their authority. That is, their authority to deprive kids of their constitutionally mandated sound basic education! ? ! ? huh? Now, this is all part of legal maneuvering. Yeah… it would be difficult for NYC to show that holding back this additional 3.3% state funding tips the scales on whether the city can provide a sound basic education. As such, how can the court reason intervening and forcing the state to give this money back?
But that’s only if we set aside that the state of New York is already depriving New York City schoolchildren of 38% of the aid that should be allocated to the city based on the state’s own formula for what the city needs to provide a sound basic education. And that was a bogus, low-balled estimate to begin with. Here’s my quick run down on state aid shortfalls in 2012-13 – with respect to the state’s own estimates – for small city districts and for New York City:

Foundation Aid, Foundation after GEA expressed in Thousands (‘000s)New York City is being shorted about 3.4 billion in aid to achieve what the state has defined as “sound basic” funding. That’s about 38% of their total foundation aid.  That share is even larger for some small city districts. This next table shows that this amounts to thousands per pupil.  Sure, the loss from the teacher evaluation debacle amounts to a few hundred per pupil. But hey, what’s the harm? NYC is already being shorted over $3,000 per pupil.By the state’s logic, we, and the sitting judge are asked to ignore that the bucket into which that drop is to fall (or not) is nearly half empty (no, not half full… well… actually… about 62% full) to begin with.  That the murder who stabs to death a victim one day, and comes back to stab the already dead body one more time the next, is not guilty of any marginal crime for his actions on the second day.Perhaps not in the final legal analysis. I’ll leave that for the judge to figure out. But in my view, it’s still pretty damned offensive.

This post comes about as a follow up to a previous post where I critiqued the rationale of the Students First policy agenda.  It should be noted that the Students First policy agenda is anything but unique. Like DFER, SFER, ALEC or any policy advocacy organization, the SF policy agenda is little more than an aggregation of largely non-original, template policy prescriptions.

Now, I’m not one who goes all in for the lingo of “corporate reform” or one who perceives “privatization” or “market” mechanisms to be inherently evil and contrary to the public good. However, I am someone who believes we should consider carefully the multitude of tradeoffs involved in shifting between publicness and privateness in the governance and provision of schooling.

What I have found most intriguing over time is that the central messaging of these reformy template policy prescriptions is that they will necessarily improve accountability and transparency of education systems, and that they will do so largely by improving the responsiveness of those intractable systems through altered governance and finance, including but not limited to “market” based choice mechanisms.

The standard list of strategies that are supposedly designed to increase accountability and transparency of our education system include, among other things:

1. Expansion of charter schools, coupled with multiple charter authorizers (including private entities) and minimized charter regulation
2. Adoption of tuition tax credit programs providing individuals and corporations the option to forgo paying a portion of taxes by contributing that amount to a privately governed entity (or entities) that manages tuition scholarships to privately governed/managed schools.
3. Parent trigger policies that permit a simple majority of parents of children currently attending any school within a district to mandate that the local board of education displace the entire staff of the school and potentially turn over governance and management of school’s operations (and physical/capital assets?) to a private management company to be operated as a charter school.

It is argued that current large bureaucratic public education systems are simply intractable, non-responsive and can’t be improved – That they are simply not accountable to anyone because they are run by corrupt self-interested public officials elected by less than 2% of eligible voters (turnout for board elections) and that they have no incentive to be responsive because they are guaranteed a constantly growing pot of revenue regardless of performance/quality/responsiveness.

Whatever problems do exist with the design of our public bureaucracies, I would argue that we should exercise extreme caution in accepting uncritically the belief that we could not possibly do worse, and that large scale privatization and contracting of private entities to provide the public good is necessarily a better and more responsive, more efficient, transparent and accountable option.

Let’s take a walk-through of some of the key aspects of current preferred reforms by comparison to traditional public governance of our education systems.

Privately Governed/Managed Charter Schools vs. Local Education Agencies

Let’s begin with the push for less regulated, expansion of charter schooling with particular emphasis on expansion of privately governed and managed charter schools, and perhaps even charter schools authorized by independent private authorizers (granted authority to operate by a private entity given that authority by the state).  To be absolutely clear, no-matter how many reformy pundits proclaim from their soapbox that Charter Schools are PUBLIC Schools… it just isn’t that simple.  In many critically important ways, under many critically important conditions Charter Schools SIMPLY ARE NOT PUBLIC in every important traditional or legal sense!  See this post for further elaboration!

Note – this varies widely from state to state, depending on whether state charter statutes specifically spell out requirements of privately governed charter schools. 

Let’s explore how/why this might be important when it comes to evaluating whether and how expanded, less regulated chartering either increases or decreases public accountability.

Table 1. Chartering vs. Traditional District Schooling

These differences are not trivial, yet few are discussing them as critical factors for shaping future education policy. Rather, day after day, week after week, we are subjected to more and more vacuous punditry by self-proclaimed “expert” pundits displaying an astounding ignorance of education law and callous disregard for our system of government and the U.S. Constitution.

For example, it would appear that charter schools that are not “state actors” (which may include most that are governed by boards of private citizens and especially those managed by private companies/EMOs or CMOs) may require students to abide by disciplinary/conduct codes which involve compelling those students to recite belief statements about the school (mottos, pledges, loyalty oaths), obligatory participation in indoctrination activities and imposition of financial penalties for disciplinary infractions, none of which would be permissible in traditional public schools. Government entities – state actors – may not compel speech and especially may not compel statements of belief.

So then, what is a family to do when no traditional public schools are available to them (as is practically the case in many areas of New Orleans and increasingly the case in other higher charter market share cities)? Should parents have to choose which rights to forgo? [picking the school with the financial penalties over the one requiring daily recitation of a loyalty oath?]

Can (as some belligerent civic illiterate,  pundits believe) entire urban school systems be replaced with charter schools – or the traditional public schools adopt the lessons of “chartering” which involve infringement of constitutional rights? Is it reasonable to assume that the entire student population of a city would be placed in a position of necessarily forgoing their rights to free expression, free exercise?

I hear those reformy pundits cry… “but who cares about a little constitutional protection here and there if we can squeeze out an extra point or two on state assessments [via selective attrition of low performing peers]? They’ll be better for it in the long run!”

Yeah… sure… that’s all well and good for someone else’s kids. I for one believe the constitution continues to have a purpose and that constitutional rights should be equally available to all people’s children. I believe that constitutional protections are a key element of an accountable education system available to all – not just some.

This is a big freakin’ deal. An important policy trade-off to consider, if you will. This is a critically important tradeoff to consider when adopting policies that expand non-state-actor charter schooling, even if some marginal academic gain can be achieved.

Indeed, under our current public schooling system constitutional battles over free exercise, free speech, discrimination, etc. persist (as any good pro-school-choice libertarian will frequently argue – I, being a former card carrying member in my NH days!). It’s a never ending tension between the preferences of the majority vs. the rights and interests of the minority. Such arguments are often used as the basis for saying that all students/families should simply have the right/option to choose where to attend school – where they can each be their own majority.  The value of our current public (gov’m’nt) system is that the minority does have the right to challenge their mistreatment and that collective participation in the public system forces public debate over these issues (even if/when they end up being handled poorly). It ain’t perfect, but I’m not willing to replace it with a system that requires large numbers of children to forgo these rights in order to participate in schooling.

Poor and minority children should not be disproportionately required to forgo constitutional protections (and a variety of statutory protections) to gain access to those few additional test score points. Further, no-one is telling them that they even have rights to begin with – especially those pitching the charter expansion policies (constantly spewing the rhetoric of the “publicness” of charter schooling).

Charter Schooling & The Market for Lemons

In theory, the accountability and efficiency advantage of charter schooling is driven by the market for choice of one school over another. Increasingly, state education agencies have moved from being impartial technical assistance agencies and accountability reporting agencies to strongly promoting the charter sector. This advocacy behavior corrupts the state agency role and creates what economists refer to as an “asymmetry of information” – in the extreme case a “market for lemons.”

Markets fail when the consumer is misled to believe that the product they are being sold is a miracle product (without counterbalancing information available to the “customer”). Asymmetry of information occurs where the seller has more information on the quality of the product than the buyer and is able to extract from the buyer a higher price than is warranted given the product’s true quality. In this case, we are talking about the parents’ choice to apply their child’s gov’t subsidized education credit, per se, at a charter versus the traditional public school.  They’ve got one credit to spend for each child and the SEA endorsed spin these days is that that credit is nearly always best spent in a charter school (even when it clearly is not).

Taken to the extremes, State Education Agency and public media flaunting of chartery miracles has created a distorted market for those charters that are least proven on the market (perhaps in some cases, lemons), with those charters that are most proven already over-subscribed and not needing to compete openly. So, those most available on the market are those whose actual performance/quality is far lower than that which is capturing the headlines and receiving accolades from state officials. [not quite a true market for lemons since the price – education “credit” is fixed … though perhaps I can expand on this at a later point].

It is the absurd punditry, intentional obfuscation and complete disregard for legitimate data/analysis on charter schooling that have perhaps soured my taste for the movement more than anything else (bearing in mind that I was a founding member of the AERA special interest group on Charter School research and, at the time, was largely an advocate myself).

Tuition Tax Credits & Vouchers vs. Conventional LEA Governance

Next up, let’s talk about tuition tax credits and vouchers. Now, I would argue that in many ways, tuition tax credits and vouchers which provide the option for children to attend schools that are well understood to be private, that not state actors are at least more honest with respect to student and employee rights.

It is understood (or should be more clearly understood) that when choosing a private school or choosing to be employed by a private employer that one’s rights may differ. On very few occasions have I actually heard the rather absurd argument that private schools receiving students on publicly financed scholarships are “public” (yes, they did, without understanding the implications, make this claim in Louisiana when their voucher model was overturned by the state courts).

Now, let’s parse the governance and accountability differences between traditional public LEAs, Vouchers and Tuition Tax Credits.

Table 2. Vouchers & Tuition Tax Credits vs. Traditional District Schooling

The simple part here is that under either the tuition tax credit or voucher program, the schools that children attend are clearly private. It is (or at least should be) understood that students and employees forgo certain rights. As such, it would be plainly illogical to use such a model as the model for an entire city or state, meaning that children would not even have the option of attending a school where they are protected from discrimination and other forms oppression. [notably, while children/families may be oppressed and/or discriminated against by the ruling “majority” in a public school setting, they have a constitutional right to challenge their mistreatment – a right that ceases to exist where only private providers are available].

Other more nuanced delineations here are between the voucher and the tuition tax credit model. The more popular TTC approach is far more convoluted, and in being so, creates additional layers of opaque to non-existent accountability, ultimately negating altogether taxpayer legal rights.

Under a voucher model, like the Cleveland voucher model, taxpayers do have the right to challenge that their tax dollars are being allocated to religious education. Indeed, when such a challenge was brought, the U.S. Supreme Court decided that the voucher mechanism in place was sufficiently neutral (reliant on parental choice) that it did not violate the establishment clause of the U.S. Constitution. But, taxpayers at least had the right to bring this challenge even if they did lose.

What I find most objectionable (in terms of public accountability) about the TTC approach is that when a similar challenge was brought against the Arizona tuition tax credit model, the U.S. Supreme Court determined that the dollars being expended effectively weren’t the taxpayers’ dollars and thus the taxpayer had no right to bring a legal challenge to the policy (no taxpayer “standing”). Quite simply no taxpayer standing means NO taxpayer legal accountability. No taxpayer legal recourse. Arguing that TTC models increase public accountability is absurd.

Further, that these systems rely on creating non-public, non-publicly accountable entities to manage these funds diverted from the public coffers further reduces public accountability.

Parent Trigger vs. Conventional Local Education Agency Governance

Parent trigger is quite possibly the most ludicrous corruption of public governance and accountability on the education reformy education policy table.  Put simply, parent trigger is the most ill-conceived subversion of governance I’ve seen out there in the reformy playbook. Let’s give it a walk-through.

Table 3. Parent Trigger vs. Traditional District Schooling Governance

The most substantive reductions of public accountability, transparency and governance occur when the simple majority of parents of children in one school decide that there school must be converted to a privately managed charter school, which may in turn adopt policies that deprive both children and employees of constitutional and statutory rights. Indeed, the district would likely be required to find a school for the displaced minority of students who don’t wish to forgo these rights. But the simple majority of parents in that school at that point in time should not be granted the authority to displace a minority of students in their school. Further, a simple majority of parents in a school in a district should not be granted the authority to dictate local board funding or contracting policies without input of the broader eligible voter population.

Among other things, Parent Trigger policies assume that the public at large who reside and own property within a school district have no stake in the accountability of that school system. School closures, school quality, school location, etc. affect the value of residential properties by affecting quality of neighborhood life. Quite likely (an open empirical question) conversion to exclusive and/or specifically themed charter schools creates unique effects on property values and neighborhood quality of life, and not necessarily always positive effects.

Finally, schools/school buildings and property are public assets having a lifespan far exceeding that brief moment in time when that trigger pulling simple majority has children attending the school and the public that has invested in those schools over time should thus have some say in their operation, maintenance and management.

The idea that this particular subversion of traditional governance somehow heightens public accountability is simply ridiculous.

Closing Thoughts

Love it or hate it, we’ve got a pretty well defined, reasonably functional system of public governance in this country, with the overarching rule of the land being our U.S. Constitution. I’m not trying to oversell here. I’m not saying it’s perfect, always responsive to all and never intractable, opaque or corrupt. But I am saying that we could certainly do worse and many proposals on the table are likely to do just that.

Importantly, state laws might be written to close many of the gaping holes in student and employee rights identified above, public disclosure requirements and clarify the delineation between publicness and privateness. But the current trend is not necessarily in that direction!

Our current system defines the roles and responsibilities of public officials, holding them to public accountability standards vetted by our federal and state judicial branches for over two centuries. Yeah, I know, many of these reformy pundits would also simply do away with that meddling judicial branch. I for one, think that our courts continue to play a critical role in protecting rights.

Modern education reform efforts, in the name of supposed increased accountability and transparency largely seek to subvert our system of government as we know it and in many cases seek to strip large shares of poor and minority children and the employees in schools of poor and minority children of constitutional protections. And we’re all supposed to be okay with that?

It’s been a fun week for grading the states. First we had the wacky ratings from Students First which graded states largely on the extent to which they had adopted the preferred policies of that organization. Then we had the old-standard Education Week Quality Counts. When it comes to their finance rating system, little has changed in recent years. These two reports, of course, produced substantially conflicting results.

One might argue that both reports and ranking systems, like our School Funding Fairness report, include several indicators intended to identify policy conditions for success. This has been the standard response of Students First when they have been criticized on the basis that the states that they have applauded most tend to have pretty low average outcomes.  But, the Students First report, Quality Counts and our Funding Fairness report differ quite substantially on what we consider to be policy conditions for success. 

Students First has put policy conditions into three categories – 1) elevating the teaching profession, 2) parent empowerment and 3) finance and governance.  Students first gives no consideration across any of these categories to whether teacher wages, for example are sufficient to recruit/retain high quality candidates into teaching or whether wages are specifically competitive in high need schools. Students First gives no consideration to whether funding, overall, is sufficient to provide either/both competitive wages or reasonable class sizes, generally, or specifically in high need schools.  It would appear to be their opinion (as was rather clearly expressed by Eric Lerum in a video conference) that overall level or distribution of funding isn’t the issue – but rather that their preferred policies are what matters, regardless of funding (since the only funding/resource equity considerations in their rankings pertained to whether charter schools received what they consider equal funding – no validation provided!)

Education Week goes old school especially on their school finance rankings. I don’t have time/space to address all of their rankings. As I will show below, some of their old-school measures seem to capture relatively useful information, but others do not. Let’s quickly summarize the measures they use.

• Fiscal Neutrality: Fiscal neutrality measures the relationship between district spending and district wealth. State school finance formulas are partly intended to disrupt this relationship – reduce the likelihood that wealthier districts spend systematically more. This measure is often still useful, but may be complicated by the fact that school finance formulas also try to address differences in student needs and costs. To the extend that higher need kids live in poorer districts (not always the case that taxable property wealth and student need are tightly associated), this indicator may work to partly capture both.
• McLoone Index: Named for school finance legend Gene McLoone! This index tells us how close, on average, the per pupil spending of districts in the lower half (serving the lower half of kids) are to the median. That is, to what extent does the state formula succeed in “leveling up” the bottom half to the middle. A McLoone of 100 would mean that the lower half is equal to the middle. But this index in particular can produce some screwy results. Say for example a state has one or a few very large districts with high need populations and those districts constitute both the lower half and the middle (they have nearly or all of the bottom half of kids). A state with one or a handful of high need large districts with spending lower than everyone else (the upper half) might still get a McLoone of 100. But it would be a really crappy school finance system! (with all due respect to Gene!)
• Coefficient of Variation: The coefficient of variation simply measures the extent of variation in per pupil spending as a percent of the mean per pupil spending. A CV of 10% indicates that 2/3 of children attend districts with per pupil spending within 10% of the mean. The problem with the CV is that, while it measures variation, it doesn’t capture the difference between GOOD variation and BAD variation. Modern state school finance formulas try to create variation in funding to accommodate differences in student needs. Education Week uses nominal weights to “adjust” for differences in student needs, but some state school finance systems actually adjust more aggressively for needs than do their weights. Those states are penalized in the CV.
• Spending Index & Percent at/Above National Mean: A few reports back Education Week wanted to construct a form of “spending adequacy” figure to compare spending levels across states and the shares of kids with access to what they considered more “adequate” spending. So they adopted this measure and index based on the percent of children in each state who attended districts that spent at least the same as the national average district (spending adjusted for regional wage variation). This figure does generally capture spending level differences across states – adjusted for wage variation – but doesn’t, for example capture spending level differences corrected for student population differences, or the shares of students who might be attending very small, remote rural districts.

Ed week includes a few additional indicators like the restricted range – or difference in spending between the 95th and 5th %ile district, but these are largely redundant with the CV & McLoone and suffer the same problems of not accounting for other cost factors – or state aid formulas that aggressively adjust for needs and costs.

We had set out to correct for many of the problems in the Ed Week approach when we started work  on our Funding Fairness report. Specifically, we wanted to make comparisons that better accounted for differences in needs and costs across districts and states and that could be used to characterize state school finance policies consistently, without suffering some of the problems of old-school indicators like the CV or McLoone Index. We also look at spending level – using a statistical model based on 3 years of data to project the per pupil state and local revenue of a district with a) average poverty rate, b) in an average wage labor market and c) with 2,000 or more students and average population density. That is, our projected state and local revenue figures are adjusted for poverty, competitive wages, size and population density. We use the same model to then evaluate whether, on average – and in a predictable pattern – state and local revenues are systematically higher (progressive) or lower (regressive) in higher poverty districts (relative to lower poverty districts). That is, does the system overall target resources to higher poverty districts – controlling for the other factors.

That prerequisite discussion aside, let’s take a look at how all of this stuff lines up – How the Ed Week Indicators line up with the Funding Fairness Indicators and how both line up with the Students First Indicators. Finally, I look at how all line up with various outcome measures.

Again… all of these funding related indicators are about policy conditions for success, rather than success itself.

First up – and here’s a relative no-brainer – both our funding fairness report and Ed Week Quality Counts include an indicator of state funding effort – or share of state capacity allocated to elementary and secondary education.  I can’t speak for Ed Week, but we include ours to acknowledge that some states spend more than others (do better on our spending level measure) because they can and that we should grade them at least partly on their effort.  Figure 1 shows that our effort measure and Ed Week’s effort measure are pretty highly correlated.

Figure 1

Figure 2, by contrast, shows that the Students First funding GPA isn’t related at all with the Ed Weeks effort indicator, and by extension with ours. Ed Week (and we) consider funding effort to be an underlying policy condition for success, apparently, Students First doesn’t .

Figure 2

Figure 3 compares our funding level indicator and Education Week’s spending index – or relative adequacy indicator. Clearly the two are highly related… but the Ed Week indicator caps out at 100% – or where 100% of the children attend districts above the national average spending. Personally, I prefer indicators that capture the full range of variation.  But again, our spending level measure and Ed Week’s spending index are picking up much of the same information – relative spending differences across states.

Figure 3

But, Figure 4 shows that Students First finance rating scheme really doesn’t relate at all to Education Week’s spending index, suggesting that overall availability of resources – like the effort to raise them – is inconsequential in the eyes of Students First.

Figure 4

Figure 5 shows the relationship between our funding progressiveness indicator and Ed Week’s fiscal neutrality indicator. For many states, the two are picking up similar things. In states like New Jersey or Utah, where higher poverty districts have more resources than lower poverty ones, the systems have also achieved fiscal neutrality (disrupted the relationship between wealth and spending). By contrast, in states like Illinois or North Carolina, the wealth-spending relationship remains strong and positive (higher wealth – higher spending) and higher poverty districts receive systematically fewer resources!

Figure 5

Recall that Illinois received one of the best grades on finance from Students First. Apparently, in addition to effort and spending level, fiscal neutrality and need based funding are also inconsequential to Students First when it comes to funding issues. Figure 6 shows the relationship between funding progressiveness and Students Firsts funding related GPA. Note that all of Students First’s funding superstars (Illinois, New York, Rhode Island and Michigan) are less than stellar on funding fairness.

Figure 6

Figure 7 relates the Ed Week CV to our funding fairness measure, showing that states with progressive funding distributions including New Jersey, Ohio and Massachusetts are actually penalized by this measure.  The CV does not differentiate between need based variation as occurs in these states and wealth-drive variation as occurs in New Hampshire.  We all seem to agree – Ed Week, Students First and us… that New Hampshire’s funding is…well… not so good!

Figure 7

Moving on, here’s the relationship between our funding fairness measure and the McLoone Index! Not much going on here… and but for a few specific examples… it’s actually hard to tell what the McLoone really captures these days in complex state school finance systems. At least it captures that New Hampshire school funding… well… sucks! But other than that, the McLoone really doesn’t capture much valuable additional information regarding equity.

Figure 8

So then, how do these various policy conditions for success relate to various outcome measures. In this table and the following graphs I explore that question, using the following outcomes:

1. Reduction in % below proficient (from http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG12-03_CatchingUp.pdf)
2. Annual Standardized Gain (NAEP, from http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG12-03_CatchingUp.pdf)
3. Adjusted (for initial level) Annual Standardized Gain
4. Reading and Math NAEP 8th Grade 2011
5. Reading and Math NAEP 8th Grade for Lowest Income Group (Free Lunch) 2011

Table 1 shows the correlations between each of the indicators addressed above and the outcome measures listed above.  Note that each of these correlations a) is relatively modest to non-existent and b) merely represents a relationship whereby when X is higher, so too is Y. Underlying causal relationships involve a complex web of factors including socio-economic and demographic conditions, etc.

Table 1

Figure 9 ranks the correlations between policy conditions and reduction in % below proficient at the 8th grade level. Interestingly, variation (inequity – bad and good) in spending is most positively associated with reduction in % below proficient. Beyond that, our funding level indicator and the two funding level indicators from Ed Week are next in line.  Students First’s teaching profession indicator is next… but their funding indicator further down. The figure seems to suggest that higher spending states, even where that spending is unequal, are doing okay on reducing % below proficient – but this is a pattern that can clearly be influenced by regional variation.

Figure 9

Figure 10 shows the correlations – ranked high to low – between each policy condition and adjusted standardized gain. In this case, adjusted standardized gains are most highly correlated with our spending level indicator, the Ed Week spending adequacy indicator, and our progressiveness indicator and Ed Week’s neutrality indicator. One might infer from this that more equitable and adequate funding is associated with greater long term average gains on NAEP… but again, regional differences may drive this to an extent. To get an idea of which states have better “adjusted annual gains” see the figure in Appendix A. Higher adjusted gains are states above the trendline and lower adjusted gains are those below the trendline.  Not all states are included (in the graph or correlations) for lack of baseline data year (I may work on updating this with multiple baseline years & tests. This is just a start).

Figure 10

Finally, we have Figure 11, which compares correlations with the NAEP scores of the lowest income children (which across states were not associated with the average income of the families of those children). These are children in families below the 130% income level for poverty.  As in Figure 9, states with the greatest spending variation seemed to have higher low income NAEP scores. Beyond that however, funding level, effort and wage competitiveness (Teaching Penalty data) seem to be positively correlated with low income NAEP scores. That is, states with higher funding levels, that put up more funding effort, and that have more competitive teacher wages (weekly, relative to non-teachers) have higher low income NAEP scores.

In Figure 11, all of the students first indicators (GPAs) are negatively associated with low income student NAEP scores. That is, low income children are doing much worse in states that got good grades from Students First.  That said, many of the conditions Students First included as setting the state for future success are policies only recently implemented in these states.

Figure 11

So that’s it… my run down of the relationship between this week’s state rankings data, how they relate (or not) to our School Funding Fairness Report and how they relate to various outcome measures. I’ll let the rest of you run with it from here! Cheers!

Data Sources:

Students First Report Card

School Funding Fairness

Ed Week Quality Counts (Finance)

Teaching Penalty

Relevant Additional Readings

Appendix A: NAEP Standardized Gains and 1990 Scores