Spotlight on Ideologies at the Extreme in New Jersey

About a week ago, I characterized “reformy” ideologies in a way that I myself even thought was extreme – so much so, that later on I added caveats to my description to say that this really isn’t what most “reformers” are advocating, but rather “reform” at the extremes.

  • Reformy Ideology #1: Teacher quality is the one single factor that has the greatest effect on a child’s life chances. Get a bad teacher or two in a row, and you’re screwed for life. The “best possible” way to measure teacher quality is by estimating the teacher’s influence on student test scores (value-added). Hiring, retention and dismissal decisions must, that is, MUST be based primarily on this information. This information may be supplemented, but value-added must play the dominant single role.
  • Reformy Ideology #2: Charter schools are the answer to most of the problems of poor urban school districts. Take any poor, failing urban school district, close those dreadfully failing schools and replace them as quickly as possible with charter schools and children in the urban core will have greatly expanded high-quality educational opportunities.

In that same post, I raised some questions about both ideologies, as I’ve done in many previous posts. In short, the “solution” part of reform ideology #1 is deeply problematic for a multitude of reasons and I urge you to read my entire thread on value-added models for assessing teacher effectiveness: https://schoolfinance101.wordpress.com/category/race-to-the-top/value-added-teacher-evaluation/

In addition to discussing the multitude of technical issues with value-added measures specifically, I also discuss extensively the potential labor market consequences for high poverty schools.

As I noted on my post the other day the problem with Reform Ideology #2 is not so much about specific charter schools or specific models and/or whether some work well or don’t, but rather about the idea that massive charter expansion is a panacea for the problems of poor urban districts. I’ve written much about his topic on this blog in the past: https://schoolfinance101.wordpress.com/category/charter-schools/

Who in their right mind would really argue that the solutions to all of our problems – or at least to urban education (code for poor and minority education) problems – is  as simple as charter school expansion, merit pay and tying teacher evaluation to test scores? Really, are there that many, or any, out there who still stick by this two-prong, unfounded, deeply problematic set of reform strategies?

Just when I thought I might have gone overboard and perhaps even been too unfair to the reform crowd, I read this editorial in New Jersey Spotlight today: http://www.njspotlight.com/stories/10/1024/1915/

Here are the conclusions of the editorial:

Here’s one way to get through the rational albeit provincial resistance from leaders of high-performing districts. Let’s just say, we have schools that are among “the very best in the nation.” But we also have schools that are among the nation’s worst. We’ve made this distinction for years, primarily through the State Supreme Court Abbott decisions, which mandate that we fund our poorest districts (recently revised to poorest students) at the same rate as our wealthiest. Why not take this acknowledgement of inequity to its logical conclusion and implement reform efforts — charter school expansion, school choice, higher compensation for great teachers, data-driven instruction — in our chronically failing districts?

Surely school leaders, legislators, New Jersey Education Association (NJEA) executives and the DOE can coalesce around charter school expansion in Pleasantville and Trenton; merit pay in Camden and Plainfield; or tying student growth to teacher evaluations in Newark and Asbury Park. While state-wide school reform will eventually come to New Jersey, our poorest students can’t wait. Targeting progressive educational strategies to failing schools may be politically distasteful, but it’s the only way to get those kids under that big white tent where they belong.

In short, the author is explaining in the first part of the editorial that many wealthy, successful New Jersey school districts haven’t supported aggressive statewide “reformy” strategies because they want no part in those strategies in their own districts. The same districts have been tentative about expanded choice for inter-district transfers. But, as this editorial argues, these districts should band together… should coalesce, to RAM DESTRUCTIVE, ILL-CONCEIVED POLICIES DOWN THE THROATS OF THEIR POOR URBAN NEIGHBORS. That’ll fix ’em! And without comparable adverse effects on their own districts!

I must say that this is about the most offensive call to arms I believe I’ve read in recent months. Yes, I’ve read some absurd arguments, like the argument that the “upper half of charters is better than average” or the argument that if current teacher evaluations are flawed, then the only answer is to replace them with student test scores (and other absurd false dichotomies).

The present NJ Spotlight argument begins with a deeply distorted, selective “factiness” about the failures of New Jersey’s urban districts (some of the nation’s worst! evidence?) and reasons for them (not enough charters, and no merit pay for teachers) and then jumps quickly to the most extreme and dreadfully oversimplified representation of the solutions (solutions, mind you, that may be far worse than the “disease”) to all of our – excuse me – their problems.

All when I thought that I might be getting too tough on the overly simplistic, bombastic and misguided logic of reform.

Money and the Market for High Quality Schools

This post is a revised version of my previous post – If money doesn’t matter…

Here is a draft set of slides to accompany this post: Resource Heterogeneity across Sectors

The theme du jour is that reform (very narrowly defined reform), not money will fix our schools. We’re already spending a lot, the pundits say. Too much in fact, for what we’re getting. We need more charter schools – which obviously do more with less – we need to treat teachers like workers in the private sector (?) by publicly ranking them based on their students’ test scores – and in general, we need to adopt “market” oriented strategies. But…

If money doesn’t matter then why do private independent schools (market driven schools?) spend, on average, so much more per child than nearby public schools?

First off, I am a supporter of private independent schools and former teacher in a private independent school in New York City – An exceptional school where tuition is now about $35,000 per child (where tuition covers only a portion of expense) in a city where the public system is being chastised by politicians and the popular media for spending about $20,000 per child. This despite the fact that the city school system must serve a more diverse and complex student population than the very selective private school where I taught.

About a year ago, I published a study on the private school marketplace in which I compiled the IRS financial filings of about 1,600 private schools around the country. And what did I find in this study? Among many other things, I found that private independent schools, a relatively large diverse sector of schools which includes many elite schools, and also some pretty average ones, spent on average 196% of public school average on same labor market (excluding boarding schools).

I also found that the pupil to teacher ratios in private independent day schools are about 8.8/1 (consistently from 2000 to 2008) compared to those of public schools at about 16.7/1 over the same period.

That is, private independent schools – ON AVERAGE – not just the elite of the elite – spend nearly double what public schools in the same area spend, and private independent schools leverage that money to purchase nearly double the teachers per child, offering much smaller class sizes, deeper and broader elective options, music, arts and other “frills” many public schools have seen evaporate with recent budget cuts.

I am by no means criticizing the choice to provide one’s own child with a more expensive education. That is a rational choice, when more expensive is coupled with substantive, observable differences in what a school offers. I am criticizing the outright hypocritical argument that money wouldn’t/couldn’t possibly help public schools provide opportunities (breadth of high school course offerings, smaller class sizes) more similar to those of elite private independent day schools, when this argument is made by individuals who prefer private schools that spend double what nearby public schools spend.

Private School Spending Study: http://nepc.colorado.edu/publication/private-schooling-US

If money doesn’t matter then why do venture philanthropists continue to throw money at charter schools while throwing stones at traditional public schools?

The standard rhetoric, touted in the media these days is that charter schools are not only doing better than traditional public schools, but that they are doing so with far fewer financial resources. The reality is that charter schools have widely varied resources, from state to state and even from block to block within New York City. Let’s focus on New York City charters for a moment, because New York City charter schools have received so much media attention.

A New York City Independent Budget Office report suggested that charter schools housed in public school facilities have comparable public subsidy to traditional NYC public schools, but charter schools not housed in public school facilities have to make up about $2,500 (per pupil) in difference. In forthcoming report, I explain how the much lower need populations served by NYC charter schools, compared to nearby NYC traditional public schools, more than offsets this difference.  That is, from the start, NYC charter schools are on relatively level financial playing field with the traditional public schools against which they supposedly compete. In fact, charters provided with physical space have a head start, and serve fewer low income children, few or no ELL children and fewer children with disabilities.

And then there’s the philanthropy. Kim Gittleson of Gotham Schools points out that in 2008-09, NYC Charter schools raised an average of $1,654 per pupil through philanthropy. In 2009, Venture Philanthropists granted over $30 million to 77 NYC charter schools, excluding major gifts to management organizations associated with many of the NYC charter schools.

Some NYC charter schools raised more than $8,000 per pupil, and depending on how you calculate it, Harlem Children’s Zone comes in as high as $60,000 per pupil. As a result, some charters – those most favored by venture philanthropists – spend on a per pupil basis much more than traditional NYC public schools.

One might argue that the Venture Philanthropists are trying to spend their way to success – To outspend the public schools in order to beat them!

In fact, a recent study funded by the New Schools Venture Fund indicated “The average CMO relies on philanthropy for approximately 13 percent of its total operating revenues, but many CMO central offices could not exist today without philanthropy.” That is, they need this level of infusion just to stay afloat, running each year in the red, with no sign of break even years in the near future.

But here’s the disconnect – These same Venture Philanthropists – who are committed to spending whatever it takes on charters in order to prove they can succeed, can be frequently heard arguing that public schools a) don’t need and b) could never use effectively any more money. They are trying to argue that charters are doing more with less, when some are doing more with more, others less with less, and some may be doing more with less, and others are actually doing less with more. Shouldn’t traditional public schools be given similar opportunity to do more with more? Blasphemy! Eh? And don’t give me that … “we’ve already tried that and it didn’t work” claim. I’ll gladly provide the evidence to refute that one!

If money doesn’t matter then why do affluent – and/or low poverty – suburban school districts continue in many parts of the country to dramatically outspend their poorer urban neighbors?

Last but not least, why do affluent suburban school districts in many states continue to far outspend poor urban ones? If there is no utility to the additional dollar spent and/or no effect produced by that additional dollar then why spend it?

In a recent article, co-author Kevin Welner and I point out that many pundits have prematurely argued that states have done away with – erased – differences in resources across wealthy and poor districts (article here: http://epaa.asu.edu/ojs/article/view/718) Really, anyone with a grain of information on this topic knows this assumption to be patently false. See also www.schoolfundingfairness.org.

Among other things Kevin Welner and I point out that nationally, there remains a positive relationship between school district spending per pupil and median household income but some progress was made through the early 1990s. It leveled off since. More importantly, that progress varies widely by state, with some states like New Jersey and Massachusetts providing more support in higher poverty settings, but many like Illinois or New York maintaining systems where affluent, predominantly white school districts continue to far outspend poor urban and urban fringe districts.

For example, in the New York Metropolitan area including only New York State districts (2007-08), lower poverty districts (those with fewer than 10% children below the poverty line) had state and local revenues per pupil ON AVERAGE, at about $23,000 to $24,000 per pupil, compared to those with over 20% poverty (census poverty rate) at just over $18,000 per pupil in state and local revenues in that same year. Yet pundits pick the $18,000 per pupil number out of context, call it too high, and argue they should get no more! No more I tell you! The waste is egregious! Kevin Welner and I identify 9 downstate suburban districts that spent more than $10,000 more per pupil than New York City in 2007-08.

If the waste in New York City, or in Newark, New Jersey is so gosh darn egregious – if we’re spending way beyond reasonable levels in poor urban districts, what about those districts spending so much more on kids who would do just fine on so much less? Isn’t that just a massive freakin’ waste? The people in these communities don’t seem to think so.

In conclusion…

Here’s the thing – I don’t believe that private independent schools or affluent local public school districts are just throwing money away. I believe they are trying to provide a high quality product to consumers who demand such a product and who expect such a product, be it through a system of local public financing or through a private market based system.

That’s the interesting twist in all of this. The “reformers” who are choosing expensive private schools for their own children and throwing money at charters are invoking the language of “market based reforms” for traditional public schools – market based reforms as a substitute for more money – because market based reforms will ALWAYS drive down per pupil spending. That’s what competition does, right?

Well, the one set of schools in this mix that are arguably most responsive to “market pressures” are the private independent schools. The schools most responsive to market pressures are the ones that a) spend the most, b) have the smallest class sizes and seem to use small class size in particular as a primary selling point, and c) I would venture to guess are least likely to be moving down the road of evaluating all of their teachers on the basis of test scores alone (most actually have relatively traditional experience driven step-scales). Yeah… yeah… but those are the luxury market products? Are small class sizes and diverse high school curriculum luxuries that should be reserved for only the few? I find this argument most offensive.

Really good education is expensive – and far more expensive than “reformers” are willing to admit or understand. If the “reform movement” is really about mimicking successful business models, these entrepreneurs should be paying close attention to the money being spent producing a high quality product – benchmarking against the “best” public and private schools, and then realizing that achieving comparable outcomes with more needy student populations will cost more – a lot more – not less. For some reason, in this case, they’ve ignored that conversation entirely!


On School Funding Fairness

I’ve been toying around for a while on this blog with different ways to compare state school finance systems. This new website presents a summary of much of that playing:

http://www.schoolfundingfairness.org/index.htm

After much discussion and debate, we landed on the following four indicators.

The Fairness Measures
All 50 states are evaluated on the basis of four separate, but interrelated, fairness measures:

  • Funding Level: Using figures adjusted to account for a variety of interstate differences, this measure allows for a comparison of the average state and local revenue per pupil across states. States are ranked from the highest to lowest per pupil funding.
  • Funding Distribution: This measure shows whether a state provides more or less funding to schools based on their poverty concentration. States are evaluated as “regressive”, “progressive”, or “flat” and are given letter grades that correspond to their relative position compared to other states.
  • Effort: This measures differences in state spending relative to the state’s fiscal capacity. States are ranked according to the ratio of state spending on education to per-capita gross domestic product.
  • Coverage: This measures the proportion of school-age children attending the state’s public schools and also addresses the income disparity between families using private, rather than public, schools. States are ranked according to both the proportion of children in public schools and the income ratio of private and public school families.

It is important to understand that two of these indicators are much more in control of the states than others – Effort and Funding distribution, or Fairness. States control the amount financial effort they put into their schools, as a percent of their capacity. States have less control over their overall funding level produced by that effort. I encourage you to look carefully at differences between states like Louisiana and Tennessee, compared to Mississippi. The first two simply don’t put up the effort. Hence my constant lambasting of Tennessee on this blog, especially in the context of RttT.

Coverage and funding level are not as controlled by states, but that’s not to say they are not significantly controlled by states. Funding levels vary about 50/50 on the basis of state wealth and on the basis of state effort. Effort seems to matter as much as wealth in predicting state spending levels, and that makes sense. Hence we grade on effort.

Coverage is included for a few reasons, and is included along with the ratio of family income of those in and not in the public education system. First, coverage is included because for many states this indicator shows just how many kids we are leaving out of our equity analysis by comparing revenues across only the public system. In a handful of states, the excluded share is around 20% and in some of those states that 20% are from much higher income households – likely increasing the “regressiveness” of the system as a whole (public/private schooling). Second, we include coverage because long term, systemic deprivation of the public system can, in fact, lead to significant flight from the public system. That should not be ignored and should not be treated, as one reader of a previous post argued (comments section), as a smart state policy decision toward further reducing long run costs- by encouraging more affluent families to independently finance their child’s education. Call it a value-laden decision, but we do not accept the argument that depriving the public education system to the point where more kids opt out, so we don’t have to use tax dollars to pay for them, is smart policy.

I also encourage readers not to try to make too much of the between state comparisons of overall spending level. I discuss many angles on these comparisons in a recent post: https://schoolfinance101.wordpress.com/2010/10/04/state-ranking-madness-who-spends-mostleast/ The bottom line is that it’s really hard to make reasonable comparisons of the cost differences of operating schools in Vermont versus Nevada. However, the within region comparisons may be more useful.

I especially encourage comparisons among the “profiles” or those sloped lines among states sharing regions. The New York/New Jersey profile comparisons are particularly interesting. New York affluent suburban districts have far more resources than New Jersey affluent suburban districts, but for poor urban districts, the differences flip.

Be sure also to check out the updated tables with the 2007-08 NCES data.

The research question that wasn’t asked

Recent discussions of the Vanderbilt University study on the effect of merit pay in Nashville raised a common and important issue pertaining to education policy research – or any research for that matter – What about the question that wasn’t asked? Or how important really is the question that was asked?

In the case of the Vanderbilt merit pay study, the researchers essentially asked whether providing sizable financial bonuses to randomly selected teachers could motivate those teachers to try harder and ultimately produce better student outcomes than teachers randomly selected to be in the group that could not get bonuses. That is, does the merit pay serve to make one randomly selected group of teachers produce better student outcomes than a control group?

Pundits quickly leaped on the question NOT ASKED – which was whether or not changing teacher compensation structures more generally – making teaching a profession based on rewards for performance or a profession where one could increase income over time by being a high performer would ultimately change the quality of individuals who would enter the teaching workforce.

That is, the study asked whether financial incentives could change the behavior of those already in the system, but not whether the existence of performance incentives would change those who choose to be in the system.

Now, when “reformy” types pointed to this question NOT ASKED, they also seemed to uniformly imply that we know the answer to the question not asked – and that is – “of course this would encourage better teachers to enter the labor market.” You know what – the question wasn’t asked. It wasn’t tested and we certainly do not know this to be the answer. For now, the answer is “we don’t know,” and it is likely fair to say that the answer is “it depends, on a variety of factors including how compensation is altered, the risk/reward ratios, etc. etc. etc.”

This brings me to a comment made by Andrew Rotherham in his recent Time Magazine post:

For example, it’s clear from abundant research that paying teachers only on the basis of their degrees and years of experience is not in the best interest of students or teachers. As the National Council on Teacher Quality, a research and policy organization whose board of directors I chaired for several years, put it, “the evidence is conclusive that master’s degrees do not make teachers more effective.”

Now, I think even this statement is a bit, well, overstated. The “research” Rotherham seems to draw on here (and NCTQ dreadfully overstates) is research that asked the following questions:

  1. Do teachers who hold general masters degrees, versus those who do not, scattered across a variety of settings, show differences in the average outcome gains of their students?
  2. Do teachers at varied levels of experience, scattered across a variety of settings, show differences in the average outcome gains of their students?

The first of these questions was beaten into the ground over and over in the 1990s, often using data from the National Education Longitudinal Studies (NELS ’88) with many of the studies showing no relationship between holding a masters or not and student outcomes, and at least a few showing positive effects of holding a content area masters in math/science (I’m doing this largely from memory).

The second of these questions has been addressed in a number of recent analyses, as well as some older ones. More recent studies have generally evaluated the average student value added ratings of teachers by their experience levels. Many of these studies find that teachers in their first two to three years tended to show smaller student achievement gains than teachers in their 4th, 5th or 6th years, but after that, things really kind of level off. Here’s an example of such analysis: http://www.urban.org/UploadedPDF/1001455-impact-teacher-experience.pdf

Interestingly, pundits pushing so hard for major changes to the risk/reward structure of teacher compensation who are so quick to point out the question not asked in the Vanderbilt merit pay study fail to recognize that similar labor market questions were never asked in these studies either.  Researchers asked whether teachers with certain attributes had better student outcomes than teachers with different attributes. As far as I recall, no one ever asked whether differential compensation on the basis of these attributes produced any desirable or undesirable labor market effects – changes to the applicant pool, etc.

Studies of the association between different levels of experience and the association between having a masters degree or not and student achievement gains have never attempted to ask about the potential labor market consequences of stopping providing additional compensation for teachers choosing to further their education – even if only for personal interest – or stopping providing any guarantee that a teacher’s compensation will grow at a predictable rate over time throughout the teacher’s career.

Many, like Rotherham but even more so, NCTQ, present this as a “research given.”  That clearly, it’s just dumb to pay teachers more who possess attributes we know are not associated with student achievement differences (across teachers). Is it possible, however, that changing these conditions could have significant labor market consequences? Perhaps good… but equally likely… unintended negative consequences.

Yes, teachers with any old masters degree or teachers with more than 10 years behind them might not, on average, be “measurably more productive.” But does the option to pay and recruit more experienced teachers or teachers with masters’ degrees enhance the likelihood that a district can attract teachers who are actually better teachers? I’m not so sure that the answer to this question unasked is so obvious that we need not ask it. So let’s stop pretending that it is.

Video Blog: School Finance & The Courts

This week, Bruce Baker from Rutgers University discusses his paper co-authored with Kevin Welner regarding research on school finance reform litigation. For more video, visit The Voice.

http://thevoice.pressible.org/edlabteam/school-finance-and-courts-does-reform-matter-and-how-can-we-tell

State Ranking Madness: Who spends most/least?

Ranking the states by different methods

Every year, through many different sources, state politicians and political activists make great waves over which state spends more on public education, and which spends less. Who’s in first place? Who’s in last? Those from differing perspectives have different motives. Politicians and anti-tax, anti-government activists search for their way to find that “our states spends more than everyone else and gets nothing for it,” while others hoping to increase education spending search frantically for low ratings – “We’re in last place and that’s a disgrace!” Of course, not everyone can be in first or last place and it’s pretty damn hard to tweak the numbers to move a state from near the top to near the bottom. Here, I’ll present a few alternative, reasonable rankings – the last two of which, I believe are most reasonable, though for some states still differ significantly.

First, let’s begin with the simplest version of the numbers- the straight up averages of school district state and local revenue per pupil (weighted by the number of pupils in each district). Now, I use the state and local revenue per pupil instead of current expenditures per pupil because state and local revenue gives a complete picture of state and local resources allocated to local public school systems and excludes expenditures of federal funds.

Politicians in New Jersey and New York love to make claims that their state is highest spending in the nation (and we get nothing for it!). Even at this most basic level, these claims are wrong. Close, but still wrong. Hooray for Vermont! But isn’t that really part of Canada anyway?

FIGURE 1

Of course, the cost of running a school varies quite significantly across states with a large share (though not all) of that variation being tied to regional differences in the competitive wage one must pay to teachers. Here, I use a competitive wage index developed by the National Center for Education Statistics (by Lori Taylor and Bill Fowler) which uses variation in non-teacher wages across labor markets to correct for variation in teacher wages. http://nces.ed.gov/edfin/adjustments.asp

FIGURE 2

Some reshuffling occurs. States like California, for example drop quite a bit because California is certainly a more expensive place to live and higher wage state. But, we may be assuming too strong a role for the wage adjustment here – assuming that state and local revenues per pupil should move on a 1 for 1 basis with wage variation. Nonetheless, not a totally unreasonable comparison.

We might also wish to consider the student populations that states must educate with their funding at present levels. That is, how much are these current dollars worth toward achieving common outcomes across students? Many cost factors influence the cost of achieving common outcomes across children, as discussed in this paper –http://surface.syr.edu/cgi/viewcontent.cgi?article=1102&context=cpr – by William Duncombe and John Yinger of the Maxwell School at Syracuse University. But, this particular paper focuses on the additional costs associated with children in poverty. Duncombe and Yinger determine that the additional cost per child falling below the federal poverty line is approximately 150% of the cost of achieving the same outcomes with a non-poor child. They also find that the additional costs associated with counts of children falling below the 185% poverty threshold are approximately 100% above average costs. Now, I go very conservative here, and I apply only a 100% weight of additional cost to children qualifying as being in poverty, using U.S. Census Bureau Small Area Income and Poverty Estimates.

FIGURE 3

But, we’ve been learning more of late about problems with using the same income thresholds for poverty across states with very different costs of living. Recently, the U.S. Census Bureau put out this exceptional paper which includes adjusted poverty measures for each state, based on three different adjustment methods – http://www.census.gov/hhes/www/povmeas/papers/Geo-Adj-Pov-Thld8.pdf. In general, the adjustments lead to much higher actual poverty rates in states with very high costs of living such as New York and California. If we use those poverty rates instead of the previous ones to adjust spending for poverty, we get this:

FIGURE 4

In this case, California moves into 49th place, or third from bottom with Washington DC in the analysis.

We’re still missing a pretty big piece of the puzzle here – additional costs associated with economies of scale and population sparsity (for more information on economies of scale see: http://www-cpr.maxwell.syr.edu/efap/Publications/Revisiting_Economies.pdf). Notice that Wyoming and Alaska are our big spenders here. Now, there’s probably no adjustment we can find to fully account for the many ways in which Alaska is different from the lower 48. Nor do the poverty corrections seem to fully address the difficulties of Washington DC. It’s all pretty imperfect. That said, I take one more stab at things, based on a regression model which attempts to control for a) competitive wage variation, b) economies of scale and population density and c) poverty. The idea here is to account for the fact that some states have a need for very small schools and districts because of their small populations which are spread across vast rural expanses. The model attempts to avoid giving a break to other states like New Jersey or Illinois that have many tiny racially segregated enclaves in densely populated suburbs. And here are the results:

FIGURE 5

An important difference when using a regression model to determine relationships between cost factors and revenue levels – instead of just dividing by the cost factors – is that the model determines the weight of those cost factors. On thing that happens in the figure above is that the influence of wage differentials is “softened.” The model-based projections do not assume a 1 for 1 relationship between competitive wages and revenue. As a result, California does not come out as low as in previous figures. Also, the model is projecting state and local revenues for a district with X% poor children. The projection is for an “equated” condition. But, if the state has far more children in higher poverty settings, and those settings have fewer resources, the model projection does not necessarily reflect the average actual conditions. However, for California in particular, there really isn’t a systematic relationship between poverty and revenues across districts – a finding that is as bad as it might seem good. In fact, it’s just bad!

Aren’t the differences really all about state wealth?

I would be remiss if I didn’t include at least a few ugly scatterplots in this post. So here goes. The first two scatterplots here show that state and local revenues per pupil are somewhat modestly related to state average poverty rates (not adjusted regionally) and to the household income levels of families with children in the public school system.

FIGURE 6

FIGURE 7

However, this final figure shows that state and local revenues per pupil are equally related to the effort a state puts up, where effort is measured in terms of state and local revenue per pupil as a share of gross domestic product by state, or gross state product. That is, some states that don’t raise much revenue per pupil simply don’t try that hard. Very few high spending states have low effort. Tennessee, Louisiana, Oklahoma, Arizona and South Dakota are near the bottom because they don’t put up much effort. Mississippi puts up average effort, but just can’t raise much revenue. I’m far more empathetic to Mississipi’s plight! Well, our highest spender, Vermont in some cases, is off the charts on effort. Despite having less capacity than states like New York or New Jersey, Vermont still manages to outspend them.

FIGURE 8

While it makes great rhetoric to claim “first in the nation” or “last in the nation” or “most expensive,” the best one can really do here is to delineate in terms of relatively high or relatively low. Not great headline stuff, but that’s how it goes. New Jersey – NOT THE HIGHEST IN THE NATION – rather, “relatively high.” California – NOT LAST IN THE NATION – but damn close to it by some measures, and still low by others!

What does the education level of 25 to 34 year olds really mean?

About a week ago, The College Board released their latest status report in their college completion series.

http://completionagenda.collegeboard.org/sites/default/files/reports_pdf/Progress_Executive_Summary.pdf

The parts of the report that seemed to grab the most media attention were those related to a) comparing the US to other countries on the percent of 25 to 34 year olds who hold an associates degree or higher and b) comparing US states to one another on the same measure.

Newspapers across the country ran with this stuff and Twitter was buzzing with punditry on what these indicators meant about the quality of K-12 public schools in each state. Our public schools must be failing us if we’re only 24th on the education level of our younger adults – one Missouri pundit tweeted (related news story here).

The first thing that caught my eye was that Washington, DC was first in the rankings of percent of 25 to 34 year olds with an associates degree or higher.  Of course it is. Washington DC is a magnet for recent college graduates. Clearly, this particular indicator says as much about the employment options for a young, college educated workforce as it does about a state’s own education system. This indicator also tells us something about the education level and expectations of the previous generation – parents of these 25 to 34 year olds, whether in the same state or elsewhere. And, this indicator may also tell us something about the extent to which a state imports or exports college students.

So, I decided to play with some data…’cuz that’s what I like to do… just to see how these rankings might change if I tweaked them a bit.

I decided it might be fun to look at the differences in the rates of college educated adults – % of 25 to 34 year olds with a bachelors degree or higher – across states in three different ways:

  1. percent of 25 to 34 year old current adult residents who hold a BA or higher
  2. percent of 25 to 34 year old adult current residents who were born in the state who hold a BA or higher
  3. percent of 25 to 34 year old adults who were born in the state, whether they continue to reside there or not, who hold a BA or higher

It would seem to me that the second of these measures is most on target – the percent of the native population that holds a certain level of education. Needless to say, when I focus on the second measure, the rankings change somewhat. Here it is:

Table 1

Education Level (% BA or Higher) of the 25 to 34 Year Old Population by State

U.S. Census – American Community Survey 2006 to 2008

Data Source: Steven Ruggles, J. Trent Alexander, Katie Genadek, Ronald Goeken, Matthew B. Schroeder, and Matthew Sobek. Integrated Public Use Microdata Series: Version 5.0 [Machine-readable database]. Minneapolis: University of Minnesota, 2010.

Washington DC which ranks 1st on resident college graduates drops to 24th on native college graduates. MA, NY and NJ which were 2, 5 & 4, are now 1, 2, 3. Virginia goes from 9th to 26th and Maryland goes from 6th to 15th when only natives are considered. This is likely a DC effect as well. NH also drops quite a bit. Wisconsin rises quite a bit. Overall, there are some pretty big changes here.

Here are a few scatterplots – ‘cuz nothin’ is more fun than a good scatterplot.

This one shows on the horizontal axis, the share of 25 to 34 year old residents who are natives (born there).  On the vertical axis is the % BA or higher for all current residents. There’s DC, way above the rest on the vertical axis and pretty far to the left on the horizontal – that is, not too many 25 to 34 year olds who live there, were born there. The native share is only lower in Nevada. But Nevada doesn’t seem to be importing college grads!

This one shows the relationship between the % BA or higher among all current residents (horizontal axis) and % BA or higher among native residents (born and live there). Clearly there’s a pretty strong relationship between the two. But, there is enough variation to really change some rankings. Mass is high either way.  The big movers are those identified above, like Maryland, Virginia and New Hampshire, which have much more educated resident young adult populations than native resident young adult populations.

This one puts the “native share” again on the horizontal axis. On the vertical axis is a measure of the difference in the education level of all current residents (25 to 34) and native current residents. It’s somewhat of a net “import” effect measure. How much more educated is the current resident population than the born and raised population? Now, this is net difference, including the fact that some individuals who were born and raised in a state might have left and become more educated. Big net importers here appear to be Maryland, Virginia and Vermont and New Hampshire (Vermont surprised me a bit here… since there isn’t a whole lot of industry to attract college grads, but Burlington does always make those “great places to live” lists). It might also be a small sample size issue with the Vermont data. At the other end of the picture are Nebraska and Nevada, which don’t appear to importing a more educated adult population. Strangely, all but Nebraska are in the positive zone on this measure (note that this measure does not have to be net-zero across states because between state migration is not the only type of migration occurring. International migration may also affect these differences. This may also reflect the fact that more educated individuals tend to be more mobile. Just pondering).

In this one, we have the “native share” again on the horizontal axis, and the difference between the education level of those born in the state – whether they stayed or not – and those who reside in the state. This is somewhat of a net “export” measure. In this case, it would appear that Wyoming is the big loser. So too are Nebraska and Wisconsin. This is the one interesting piece about Wyoming. In the rankings above, Wyoming doesn’t move much. It’s 47th in % BA for current residents and 48th for native residents. But, Wyoming does much better on the education level of those born in the state, whether they stay or not – which apparently they don’t if they have a BA or higher.

So what does all of this mean? Probably not much. These figures and additional analyses certainly tell a more nuanced story than the media buzz of last week. But, it’s hard to really link much of this back to the quality of states’ underlying elementary and secondary education systems. Far too many factors are in play here, and even tweaking this one factor – whether residents are native residents or not- has significant consequences for state rankings.

So much for attaching any simple, bold statement about [YOUR STATE HERE] to that huge, pull-out multi-color map in the College Board Report!


The Gist Twist(s) & Rhode Island School Finance

So, I’ve tried not to… but I’ve been following the relatively uninformed debate over Rhode Island’s nifty new Foundation Aid formula on the National Journal “Experts” Blog.

http://education.nationaljournal.com/2010/06/a-funding-formula-for-success.php#comments

Yep, Rhode Island has invented the… wheel… or perhaps bread… one or the other. Pretty much a run-of-the-mill foundation aid formula here. And that’s not necessarily a bad thing. But there are a number of “wait and see” issues here… like how well the crafty state-local matching aid formula will work and to what extent the single relatively small and completely arbitrary poverty weight will actually drive additional funding to higher poverty districts.

One thing really caught my eye in Deborah Gist’s response to David Sciarra. Mr. Sciarra criticized the inclusion of New Hampshire in the calculation of the foundation aid level for the 2010-11 incarnation – adoption year incarnation of the nifty new bread/wheel. Here’s how Gist responds:

1. Our core instructional amount was based on national research, using data from the NCES, is sufficient to fund the requirements of the Rhode Island Basic Education Program, and it in no way focused on states with low per-pupil expenditures. In fact, we looked particularly carefully at our neighboring states, which have some of the highest per-pupil expenditures in the nation, and we included only those states that have an organizational structure and staffing patterns similar to ours.

First, I must say that it is a strange use of the term “national research” to refer to simply taking averages of spending data from states collected from a national survey, jointly from the National Center for Education Statistics and Census Bureau. It’s an annual survey. Collection of data. Not national research. It could be used for research. Heck, I love those data and know them oh too well. Which brings me to the Gist Twist here. And, it’s a three part twist.

You see, the goal is to identify an underlying “foundation” level of funding for school districts in Rhode Island.

Twist Part I: The first part of the twist, which I will not dig through here in great detail, is the pruning back of core instructional expenditures, a definition in the NCES data intended to be reported uniformly across states, albeit imperfect. The choice of core versus all current operating expense clearly drops the foundation value, and quite significantly. What remains unknown is the extent to which other aid beyond the foundation formula will actually address those other cost areas. In 2007-08, Rhode Island instructional spending per pupil was about $8,500 and current operating expenditures per pupil over $14,000. That’s a big difference to cover with other aid. Let’s hope they do.

Twist Part II: I was also quite intrigued by Gist’s explanation of how national data were used, and her defense to the accusation that they picked low spending states and took the average of the low spending states. Gist responds by saying they took “neighbors” of Rhode Island, which are, of course high spending states.

Here’s how the actual legislation describes the process:

(1) The core instruction amount shall be an amount equal to a statewide per pupil core instruction amount as established by the department of elementary and secondary education, derived from the average of northeast regional expenditure data for the states of Rhode Island, Massachusetts, Connecticut, and New Hampshire from the National Center for Education Statistics (NCES) that will adequately fund the student instructional needs as described in the basic education program and multiplied by the district average daily membership as defined in section 16-7-22.

http://www.ride.ri.gov/Finance/Funding/FundingFormula/Docs/H8094Aaa_FINAL_6_10_10.pdf

Even though I love maps, I won’t post one here. Maybe it’s because I used to teach in New Hampshire, and once lived in eastern Connecticut that I realize that one of these two is actually a neighbor of Rhode Island and one is not. Okay… for those of you pulling out your maps to figure out how all of those tiny New England states line up… yeah… New Hampshire does not neighbor Rhode Island. So then, why include New Hampshire in the calculation of the average instructional expenditures to set the Rhode Island foundation. Okay… let’s set aside the fact that this whole approach is actually not a reasonable way to identify the costs of meeting Rhode Island’s education standards, in Rhode Island districts and charter schools. But if you’re going to go down this road, the decisions should be somewhat justifiable.

Here’s the average core instructional spending per pupil for the states used:

Hmmmm… which one of these is not like the others? Yeah… New Hampshire’s per pupil spending is somewhat lower. But, it is a smaller state than the other two, and thus has lessened effect on the averages.  Oh… by the way… “similar organizational structure” as noted by Gist above, was her/their way of cutting out Vermont from the averages – because Vermont has too many non-unified districts – or actually – because Vermont is the highest spending of these states.

Here’s the effect on the averages. Including New Hampshire brings the average down by just under $200 per pupil. While this doesn’t seem like a lot, it’s about 1/3 of the difference between Rhode Island’s current spending per pupil and the target spending. That is, including New Hampshire cuts the aggregate increases in funding (difference between RI current and Target) required by about 1/3 … but that’s before we get to Part III of the twist.

Twist Part III: As far as I can tell, the proposed foundation level for fy2010-11 or even fy2011-2012? is to be set at $8,295.  Please correct me if this is not true.  That’s the amount cited here on slide #8:

http://www.ride.ri.gov/Finance/Funding/FundingFormula/Docs/Formula_PPT.pdf

And in any other documentation in which a foundation number is cited. These documents are generally from this past winter/spring leading up to passage of the legislation. So what’s wrong with that?  Well, the average spending of CT, MA and NH which comes out to about $8,295 (actually, mine comes out to $8,259) is from data from fiscal year 2006-07. Are they really basing the 2010-11 or 2011-12 foundation level on 2006-07 data?  Take a look at my second graph above. The 2007-08 data came out the other day. And, as it turns out, the 2007-08 Rhode Island average core instructional spending per pupil was over $8,500. That’s actually more than the new foundation level.

That’s not to say that it can’t be reasonable to have a foundation level that’s less than current average spending. After all, the average spending is the average of all districts, including their varied needs. It is conceivable that the current average is more than sufficient… to achieve current average performance in districts with less than average needs. But that’s not how this is being spun at all. Rather, it’s being spun as a breakthrough based on thorough and thoughtful empirical analysis.  That’s hardly the case.

Quite honestly, Ms. Gist and the RI legislature may have been better off saying that the foundation level will be set at $8,295 because that’s how much we are willing to pay for – not this silly back of the napkin justification of the amount they were willing to pay for. That in mind, this foundation formula and its arbitrary weights – excuse me – weight – actually bring us backwards, not forwards in the school finance debate, making a mockery of “research” and its potential use for informing state school finance policy.

Sorry… got a little edgy at the end there.

And here’s a little extra credit reading which actually covers national research on estimating the cost of achieving state standards. It’s from the National Research Council of all places: http://www7.nationalacademies.org/CFE/Taylor%20Paper.pdf

Follow up note:

As the statute reads, RI itself would also be included in the average calculation, lowering the value further. It makes little sense to include current average (or even 3 year old average) spending of the state you are trying to “fix” in the average spending to inform the foundation level if the assumption is that the state has, for lack of any real formula, fallen behind in regional competitiveness. Of course, it hasn’t fallen behind New Hampshire. So… my above averages do not include Rhode Island itself and are intended only to be illustrative of the arbitrary (well… not really arbitrary… intentional) choice of including New Hampshire in the calculation.

By the way… I wonder if Deborah Gist can see New Hampshire from her window, or does Massachusetts actually get in the way?

An Alternative Look at the Census Financial Data

The spin is on. As soon as the annual school district level U.S. Census fiscal survey data are released, news outlets across the country take their shot a spinning the data to show just where their state stands. New York #1! Utah… dead last! Hawaii “above average.” Spending just really high (totally out of context)! Typically, news outlets point out spending is high when they wish to argue that it’s too high… and we should do something to curb it. No mention is made of outcomes achieved with that spending, or which districts in the state are responsible for the high average. When spending is reported as low, the spin is generally that it is too low, and that state policymakers should do something about it.

Allow me to briefly present a slightly more nuanced picture. For the past few years, and in a number of publications, I have used a statistical model of the national school finance data to correct for such issues as a) economies of scale and population density, b) regional variation in competitive wages, and c) variations in student needs. I use this model to project what a school district, with comparable characteristics, would have in state and local revenue per pupil in each state. The methods of this madness were used in this study: http://epaa.asu.edu/ojs/article/viewFile/718/831

Here are some of the results with the 2007-08 Census Fiscal Survey data (with the model built on data from 2005-06, 2006-07  & 2007-08).

Before getting to the modeled estimates of comparable state and local revenue, lets take a quick look at the relative educational effort of each state, or the combined State and Local Revenues for K-12 education as a share of Gross State Product. Vermont and New Jersey lead the pack on this on, with other states including Maryland and New York in the mix. Note, however, that this effort can be quite unevenly distributed. In fact, it may be the case that a significant amount of effort is going into local property tax revenues being raised by the richest communities in a state. Yeah… it’s still a lot of effort, but selectively distributed among those who can put up that effort and choose to as long as it benefits (or is perceived to benefit) their own children. Total effort provides a limited window, but important one nonetheless.

Fun Fact about this first table – TAKE A LOOK AT OUR RACE TO THE TOP, ROUND 1 WINNERS! (47TH & 50TH ON EFFORT!!!!)

Now to the model based estimates of who’s really in the top and bottom ten on state and local revenue per pupil for elementary and secondary education. Let’s begin by looking at those states where the lowest poverty districts have the highest and lowest resources.

Yep, New York is #1 in per pupil state and local revenues for very low poverty districts! Indeed, very affluent Long Island and Westchester County school districts in New York State spend about as much as any districts in the nation, largely because they have the financial capacity to do so (and partly because the state has enabled them to!)

Next in line in funding for very low poverty districts are Wyoming and Vermont, which really don’t have many children attending incredibly high poverty districts. Notably, New Jersey falls well behind New York state for low poverty districts, and many of New Jersey’s affluent suburbs lie in the same labor market with the higher spending affluent New York suburbs. And then there’s Tennessee – one of our great RttT winners.  Of course, as I have shown on a previous post, this works fine for TN, which as the lowest state assessment cut scores – so most of the kids pass the tests anyway (low standards & low funding – a winning combination indeed)!  

The next table ranks per pupil funding for high poverty districts.  Notably, New York is NOT in first place on this one. New York drops to 6th, but the situation is somewhat more complicated. While this might appear okay, it can be particularly difficult for high poverty New York state school districts to recruit and retain high quality teachers when they are surrounded by so many affluent districts which already hold the recruitment and retention advantage, and have substantially more resources. For high poverty districts, New Jersey and Wyoming come in first. Wyoming is simply high across the board. And yep… there’s Tennessee again – our RttT winner in 47th place!

This next table ranks the within-state FAIRNESS of the state school funding distribution – where fairness is determined by taking the ratio of high poverty funding to low poverty funding – with the implicit assumption that state school finance systems should provide for additional support in districts serving children with greater needs. Now, this table must be taken in the context of the previous two. For example, Utah comes in first on “fairness.” But, in this case, this merely means that low poverty districts in Utah get nothing, and high poverty districts in Utah get next to nothing! In a twisted sense, that’s “fair?????”

Among states not at the bottom in overall resources, New Jersey, Ohio, Minnesota and Massachusetts seem to be driving additional resources into higher need, higher poverty districts.

States  at the other end of the spectrum include New York, Pennsylvania and Illinois. These are among the historically least equitable large, diverse states in the country. Now, to Pennsylvania’s credit, these calculations precede the phase-in of their new funding formula which the governor has continued to support even during the recession. New York and Illinois are another story. Yeah… New York also implemented – okay – kind of planned to implement a new formula. That didn’t get very far, and it is highly unlikely (okay, almost entirely unlikely based on other analysis I’ve conducted on more recent NY data) that NY has actually improved since 2007-08.  Illinois hasn’t even tried – in fact, Illinois just keeps getting worse and worse!

Now for an obligatory point – Many argue that the overall funding level in states is simply a function of their wealth. Wealthier states, like wealthier school districts within states simply have the ability to spend more. That is indeed partly true. But effort also matters – remember that first slide above?  This scatterplot shows the relationship between state effort and funding levels in a hypothetical average poverty school district. There’s actually a reasonably strong relationship here, but for a few quirky outliers. In fact, based on additional analyses, a state’s effort explains about as much of the funding level as does a state’s wealth.

So, Mississippi is a very poor state that puts up relatively average effort, but simply can’t get very far with that effort. By contrast, Tennessee and Louisiana both have much higher fiscal capacity (measured by gross state product per capita) than Mississippi, but they simply don’t use it. Tennessee has little excuse for its spending level! Nor does Louisiana!

Finally, here’s a snapshot of the association between 8th grade reading and math NAEP performance and funding levels across states. As it turns out, funding levels for high poverty settings were most strongly associated with NAEP performance for all students. As one can see, there exists a reasonable correlation between funding levels and NAEP mean scale scores. That said, as I have noted in previous posts regarding such relationships, there’s a lot of circular stuff all tangled up in here. Wealthier states with more educated adult populations supporting higher education spending – and supporting and encouraging their children to do well in school, etc.  But, it is difficult to conceive how a state in the bottom left corner of this picture (very low funding in high poverty districts – and most likely, low funding across the board) can begin to lift itself out of that corner – or Race to the Top. Financial resources are a necessary underlying condition, albeit easier to achieve in some states than in others.

Note: Difficulties arise when trying to make simple comparisons of funding levels and funding gaps with achievement gaps between poor and non-poor children in each state a) because income thresholds used for subsidized lunch status characterize very different populations from one region of the country to another and from rural to urban settings within states, and b) because gaps between non-poor and poor children in states depend significantly on how wealthy are the non-poor and how poor are the poor. Sadly, these complexities make it very difficult if not impossible to use NAEP data to untangle the relationship between funding differences between lower and higher poverty districts, and outcome differences between children attending those districts in different states:

I discuss the poverty measurement problems here:

https://schoolfinance101.wordpress.com/2009/11/27/title-i-does-not-make-rich-states-richer/

Kevin Welner and I discuss evaluating the relationship between state school funding distribution and student outcomes here:

https://schoolfinance101.com/wp-content/uploads/2010/05/doreformsmatter_formatted.pdf

Pondering Legal Implications of Value-Added Teacher Evaluation

I’m going out on a limb here. I’m a finance guy. Not a lawyer. But, I do have a reasonable background on school law thanks to colleagues in the field like Mickey Imber at U. of Kansas and my frequent coauthor Preston Green at Penn State. That said, any screw ups in my legal analysis below are my own and not attributable to either Preston or Mickey. In any case, I’ve been wondering about the validity of the claim that some pundits seem to be making that these new teacher evaluation policies are going to make it easier and less expensive to dismiss teachers.

=====

A handful of states have now adopted legislation which mandates that teacher evaluation be linked to student test data. Specifically, legislation adopted in states like Colorado, Louisiana and Kentucky and legislation vetoed in Florida follow a template of requiring that teacher evaluation for pay increase, for retaining tenure and ultimately for dismissal must be based 50% or 51% on student “value-added” or “growth” test scores alone. That is, student test score data could make or break a salary increase decision, but could also make or break a teacher’s ability to retain tenure. Pundits backing these policies often highlight provisions for multi-year data tracking on teachers so that a teacher would not lose tenure status until he/she shows poor student growth for 2 or 3 years running. These provisions are supposed to eliminate the possibility that random error or a “bad crop of students” alone could determine a teacher’s future.

Pundits are taking the position that these new evaluation criteria will make it easier to dismiss teachers and will reduce the costs of dismissing a teacher that result from litigation. Oh, how foolish!

The way I see it, this new crop of state statutes and regulations which include arbitrary use of questionable data, applied in a questionably appropriate way will most likely lead to a flood of litigation like none that has ever been witnessed.

Why would that be? How can a teacher possibly sue the school district for being fired because he/she was a bad teacher? Simply writing into state statute or department regulations that one’s “property interest” to tenure and continued employment must be primarily tied to student test scores does not by any stretch of the legal imagination guarantee that dismissal based on student test scores will stand up to legal challenges – good and legitimate legal challenges.

There are (at least) two very likely legal challenges that will occur once we start to experience our first rounds of teacher dismissal based on student assessment data.

Due Process Challenges

Removing a teacher’s tenure status is denial of a teacher’s property interest and doing so requires “due process.” That’s not an insurmountable barrier, even under typical teacher contracts that don’t require dismissal based on student test scores. Simply declaring that “a teacher will be fired if he/she shows 2 straight years of bad student test scores (growth or value-added)” and then firing a teacher for as much does not mean that the teacher necessarily was provided due process. Under a policy requiring that 51% of the employment decision be based on student value added test scores, a teacher could be wrongly terminated due to:

a) Temporal instability of the value-added measures

http://www.urban.org/UploadedPDF/1001266_stabilityofvalue.pdf

Ooooh…Temporal instability… what’s that supposed to mean? What it means is that teacher value-added ratings, which are averages of individual student gains, tend not to be that stable over time. The same teacher is highly likely to get a totally different value added rating from one year to the next. The above link points to a policy brief which explains that the year to year correlation for a teacher’s value added rating is only about .2 or .3. Further, most of the change or difference in the teacher’s value added rating from one year to the next is unexplainable – not by differences in observed student characteristics, peer characteristics or school characteristics. 87.5% (elementary math) to 70% (8th grade math) noise! While some statistical corrections and multi-year measures might help, it’s hard to guarantee or even be reasonably sure that a teacher wouldn’t be dismissed simply as a function of unexplainable low performance for 2 or 3 years in a row. That is, simply due to noise, and not the more troublesome issue of how students are clustered across schools, districts and classrooms.

b) Non-random assignment of students

The only fair way to compare teachers’ ability to produce student value-added is to randomly assign all students, statewide to all teachers… and then of course, to have all students live in exactly comparable settings with exactly comparable support structures outside of school, etc., etc. etc. That’s right. We’d have to send all of our teachers and all of our students to a single boarding school location somewhere in the state and make sure, absolutely sure that we randomly assigned students, the same number of students to each and every teacher in the system.

Obviously, that’s not going to happen. Students are not randomly sorted and the fact that they are not has serious consequences for comparing teachers’ ability to produce student value-added. See: http://gsppi.berkeley.edu/faculty/jrothstein/published/rothstein_vam2.pdf

c) Student manipulation of test results

As she travels the nation on her book tour, Diane Ravitch raises another possibility for how a teacher might find him/herself out of a job by no real fault of actual bad teaching. As she puts it, this approach to teacher evaluation puts the teacher’s job directly in the students’ hands. And the students can, if they wish, choose to consciously abuse that responsibility.  That is, the students could actually choose to bomb the state assessments to get a teacher fired, whether it’s a good teacher or a bad one. This would most certainly raise due process concerns.

d) A whole bunch of other uncontrollable stuff

A recent National Academies report noted:

“A student’s scores may be affected by many factors other than a teacher — his or her motivation, for example, or the amount of parental support — and value-added techniques have not yet found a good way to account for these other elements.”

http://www8.nationalacademies.org/onpinews/newsitem.aspx?RecordID=1278

This report generally urged caution regarding overemphasis of student value-added test scores in teacher evaluation – especially in high stakes decisions. Surely, if I was an expert witness testifying on behalf of a teacher who had been wrongly dismissed, I’d be pointing out that the National Academies said that using the student assessment data in this way is not a good idea.

Title VII of the Civil Rights Act Challenges

The non-random assignment of students leads to the second likely legal claim that will flood the courts as student testing based teacher dismissals begin – Claims of racially disparate teacher dismissal under Title VII of the Civil Rights Act of 1964.  Given that students are not randomly assigned and that poor and minority – specifically black – students are densely clustered in certain schools and districts and that black teachers are much more likely to be working in schools with classrooms of low-income black students, it is highly likely that teacher dismissals will occur in a racially disparate pattern. Black teachers of low-income black students will be several times more likely to be dismissed on the basis of poor value-added test scores. This is especially true where a statewide fixed, rigid requirement is adopted and where a teacher must be de-tenured and/or dismissed if he/she shows value-added below some fixed value-added threshold on state assessments.

So, here’s how this one plays out. For every 1 white teacher dismissed on value-added basis, 10 or more black teachers are dismissed –  relative to the overall proportions of black and white teachers. This gives the black teachers the argument that the policy has racially disparate effect. No, it doesn’t end there. A policy doesn’t violate Title VII merely because it has racially disparate effect. That just starts the ball rolling – gets the argument into court.

The state gets to defend itself – by claiming that producing value-added test scores is a legitimate part of a teacher’s job and then explaining how the use of those scores is, in fact neutral with respect to race. It just happens to have the disparate effect. Right? But, as the state would argue, that’s a good thing because it ensures that we can put better teachers in front of these poor minority kids, and get rid of the bad ones.

But, the problem is that the significant body of research on non-random assignment of students and its effect of value added scores indicates that it’s not necessarily differences in the actual effectiveness of black versus white teachers, but that the black teachers are concentrated in the poor black schools and that student clustering and not teacher effectiveness is leading to the disparate rates of teacher dismissal.  So they weren’t fired because they were precisely measurably ineffective, they were fired because they had classrooms of poor minority students year after year? At the very least, it is statistically problematic to distill one effect from the other! As a result, it’s statistically problematic to argue that the teacher should be dismissed! There is at least equal likelihood that the teacher is wrongly dismissed as there is that the teacher is rightly dismissed. I suspect a court might be concerned by this.

Reduction in Force

Note that many of these same concerns apply to all of the recent rhetoric over teacher layoffs and the need to base those layoffs on effectiveness rather than seniority. It all sounds good, until you actually try to go into a school district of any size and identify the 100 “least effective” teachers given the current state of data for teacher evaluation. Simply writing into a reduction in force (RIF) policy a requirement of dismissal based on “effectiveness” does not instantly validate the “effectiveness” measures. And even the best “effectiveness” measures, as discussed above, remain really problematic, providing tenured teachers reduced on grounds of ineffectiveness multiple options for legal action.

Additional Concerns

These two legal arguments ignore the fact that school districts and states will have to establish two separate types of contracts for teachers to begin with, since even in the best of statistical cases, only about 1/5 of teachers (those directly responsible for teaching math or reading in grades three through eight) might possibly be evaluated via student test scores (see: https://schoolfinance101.wordpress.com/2009/12/04/pondering-the-usefulness-of-value-added-assessment-of-teachers/)

I’ve written previously about the technical concerns over value-added assessment of teachers and my concern that pundits are seemingly completely ignorant of the statistical issues. I’m also baffled that few others in the current policy discussion seem even remotely aware of just how few teachers might – in the best possible case – be evaluated via student test scores, and the need for separate contracts. But, I am perhaps most perplexed that no-one seems to be acknowledging the massive legal mess likely to ensue when (or if) these poorly conceived policies are put into action.

I’ll save for another day the discussion of just who will be waiting in line to fill those teaching vacancies created by rigid use of test scores for disproportionately dismissing teachers in poor urban schools. Will they, on average, be better or perhaps worse than those displaced before them? Just who will wait in this line to be unfairly judged?

For a related article on the use of certification exams for credentialing teachers, see:

Green, P.C., Sireci, S.G. (2005) Legal and Psychometric Criteria for Evaluating Teacher Certification Tests.  Educational Measurement: Issues and Practice. Volume 19 Issue 1, Pages 22 – 31