Category: Reformy

The research question that wasn’t asked

Recent discussions of the Vanderbilt University study on the effect of merit pay in Nashville raised a common and important issue pertaining to education policy research – or any research for that matter – What about the question that wasn’t asked? Or how important really is the question that was asked?

In the case of the Vanderbilt merit pay study, the researchers essentially asked whether providing sizable financial bonuses to randomly selected teachers could motivate those teachers to try harder and ultimately produce better student outcomes than teachers randomly selected to be in the group that could not get bonuses. That is, does the merit pay serve to make one randomly selected group of teachers produce better student outcomes than a control group?

Pundits quickly leaped on the question NOT ASKED – which was whether or not changing teacher compensation structures more generally – making teaching a profession based on rewards for performance or a profession where one could increase income over time by being a high performer would ultimately change the quality of individuals who would enter the teaching workforce.

That is, the study asked whether financial incentives could change the behavior of those already in the system, but not whether the existence of performance incentives would change those who choose to be in the system.

Now, when “reformy” types pointed to this question NOT ASKED, they also seemed to uniformly imply that we know the answer to the question not asked – and that is – “of course this would encourage better teachers to enter the labor market.” You know what – the question wasn’t asked. It wasn’t tested and we certainly do not know this to be the answer. For now, the answer is “we don’t know,” and it is likely fair to say that the answer is “it depends, on a variety of factors including how compensation is altered, the risk/reward ratios, etc. etc. etc.”

This brings me to a comment made by Andrew Rotherham in his recent Time Magazine post:

For example, it’s clear from abundant research that paying teachers only on the basis of their degrees and years of experience is not in the best interest of students or teachers. As the National Council on Teacher Quality, a research and policy organization whose board of directors I chaired for several years, put it, “the evidence is conclusive that master’s degrees do not make teachers more effective.”

Now, I think even this statement is a bit, well, overstated. The “research” Rotherham seems to draw on here (and NCTQ dreadfully overstates) is research that asked the following questions:

  1. Do teachers who hold general masters degrees, versus those who do not, scattered across a variety of settings, show differences in the average outcome gains of their students?
  2. Do teachers at varied levels of experience, scattered across a variety of settings, show differences in the average outcome gains of their students?

The first of these questions was beaten into the ground over and over in the 1990s, often using data from the National Education Longitudinal Studies (NELS ’88) with many of the studies showing no relationship between holding a masters or not and student outcomes, and at least a few showing positive effects of holding a content area masters in math/science (I’m doing this largely from memory).

The second of these questions has been addressed in a number of recent analyses, as well as some older ones. More recent studies have generally evaluated the average student value added ratings of teachers by their experience levels. Many of these studies find that teachers in their first two to three years tended to show smaller student achievement gains than teachers in their 4th, 5th or 6th years, but after that, things really kind of level off. Here’s an example of such analysis: http://www.urban.org/UploadedPDF/1001455-impact-teacher-experience.pdf

Interestingly, pundits pushing so hard for major changes to the risk/reward structure of teacher compensation who are so quick to point out the question not asked in the Vanderbilt merit pay study fail to recognize that similar labor market questions were never asked in these studies either.  Researchers asked whether teachers with certain attributes had better student outcomes than teachers with different attributes. As far as I recall, no one ever asked whether differential compensation on the basis of these attributes produced any desirable or undesirable labor market effects – changes to the applicant pool, etc.

Studies of the association between different levels of experience and the association between having a masters degree or not and student achievement gains have never attempted to ask about the potential labor market consequences of stopping providing additional compensation for teachers choosing to further their education – even if only for personal interest – or stopping providing any guarantee that a teacher’s compensation will grow at a predictable rate over time throughout the teacher’s career.

Many, like Rotherham but even more so, NCTQ, present this as a “research given.”  That clearly, it’s just dumb to pay teachers more who possess attributes we know are not associated with student achievement differences (across teachers). Is it possible, however, that changing these conditions could have significant labor market consequences? Perhaps good… but equally likely… unintended negative consequences.

Yes, teachers with any old masters degree or teachers with more than 10 years behind them might not, on average, be “measurably more productive.” But does the option to pay and recruit more experienced teachers or teachers with masters’ degrees enhance the likelihood that a district can attract teachers who are actually better teachers? I’m not so sure that the answer to this question unasked is so obvious that we need not ask it. So let’s stop pretending that it is.

NCTQ: We’re sure it will work! Even if research says it doesn’t!

Last spring, I had the pleasure of presenting on teacher labor market research in the same conference session in which a very interesting paper on mutual consent teacher contract changes was also presented (by Bethany Gross). This paper is a product of an organization I’ve poked fun at in the past (Center for Reinventing Public Education) but this one is good stuff, by credible authors.  The methods are relatively tight, but it is a bit tricky to figure out the implications of the findings – discussed blow.  This study fits into the broader topic and policy concern of “how do we get a better balance of teacher quality across poorer and less poor schools in the same district?”

Now, pundits (not these researchers) like those from Center for American Progress, Education Trust, some from CRPE and those from New Teacher Project and National Council on Teacher Quality (NCTQ) all seem to argue that the biggest teacher quality/sorting problems are those that occur across rich and poor kids within school districts – all because of teacher seniority preferences, tenure and contractual issues which all favor the interests of adults over the interests of children – like letting the senior teacher keep his/her job in the cushy school in the district (or letting the senior teacher bump the junior teacher from such a position).

As I’ve shown in many recent posts, in general, rich kids and poor kids, black kids and white kids don’t often attend the same school districts – but for a few very large urban ones and some other sprawling countywide systems. The BIG disparities in resources and teacher characteristics are across not within districts. They are disparities that result across different bargaining units – not within bargaining units. So it’s pretty hard to argue that most disparities in teacher quality across rich and poor kids from one location to another are caused by seniority focused, adult interest, contractual provisions.

But, setting that broader issue aside, what do we actually know about within district disparities in the distribution of teacher characteristics, and whether changing contracts to remove these “offensive” protections can actually help redistribute teacher quality? Well, here’s what Gross and colleagues found:

http://www.nctq.org/docs/Mutual_Concent_8049.pdf

We conduct an interrupted time-series analysis of data from 1998-2005 and find that the shift from a seniority-based hiring system to a “mutual consent” hiring system leads to an initial increase in both teacher turnover and share of inexperienced teachers, especially in the district’s most disadvantaged schools. For the most part, however, these initial shocks are corrected within four years leaving little change in the distribution of inexperienced teachers or levels of turnover across schools of different advantage.

So, initially the policy change actually made things worse and in the end, the policy change made things no different. There may actually be some reasonable explanations for these findings. Perhaps most problematic, teachers who are really beginning to hit their stride in years 5 to 10 or so, might take advantage of their newly discovered mobility to jump more quickly from positions in higher poverty, higher need schools into more desirable positions once reserved for the most senior teachers. This could create a substantial drain of quality – non-novice but not really old – teachers from high need schools.

Notice the URL for this study. It is posted on the NCTQ website. That doesn’t mean they ever read it though. SOMEHOW, THE NEW NCTQ REPORT WHICH ARGUES THAT THESE CHANGES ARE PART OF THE SOLUTION, DOESN’T PAY ANY ATTENTION TO THIS!

Here’s a link to that report: http://www.nctq.org/tr3/docs/nctq_site_based_hiring.pdf

Here’s the list of the 3 “major barriers” to improving the distribution of teacher quality, as identified by NCTQ:

• Centralized hiring. In most districts, the human resources office controls the hiring process, determining whom to recruit and hire and where to place teachers. Principals, at most, are given the opportunity to voice their preferences.

• Inadequate evaluations. Teachers in most districts are not regularly, or sufficiently, evaluated, meaning that evaluations can only play a minor role in personnel decisions, when they should be paramount. It is seniority, not performance, that decides the movement of teachers within the district.

• Contractual obligations. Most teacher contracts stipulate that, if a teacher loses her current assignment—because of a shift in the student population, for example—the district has to find her a new assignment, regardless of whether another school wants to accept the teacher. Compounding the problem is that most state laws limit the reasons districts can dismiss a teacher, and being without a classroom assignment is not one of them. Districts are left with little choice but to either assign teachers to positions or keep them on the payroll, sometimes for years, even if they aren’t teaching.

See that third one – Yep – it’s those  contractual provisions that keep these disparities in place. Remove then and all will be fixed!

Now briefly on their first point – that centralized hiring is the other really big problem. The answer – let school site principals make decisions and teachers decide which principal they really want to work for in a district. That couldn’t backfire?   Well, I used to believe the same – that this could be a reasonable idea. The problem with this idea is that principal quality is so disparately distributed. I have recently worked on several studies of principal labor markets, the distribution of principals by their academic preparation and other factors across schools within districts and the relationship between principal attributes and the teachers they hire. Given what we are learning from these studies, it is in fact very likely to backfire! The weakest principals tend to be in the highest need schools and weak principals tend to attract and potentially even retain weaker teachers.

This line in the NCTQ press release is particularly fun, because it’s based on nothing but “gut” and “emotional appeal” – which is always the best basis for experimenting with the lives of low income and minority children, right?

“Giving principals the authority to hire who works on their staff is critical,” says Kate Walsh, NCTQ’s president. “It is the only fair way to hold schools accountable for results. But if the principal doesn’t have enough control over the quality of her staff, the school—and, of course, the students—will suffer.”

You know what – not addressing the larger resource disparities across and within districts – which lead to the disparities in leadership quality across schools and districts – and then handing greater control over teacher hiring/firing to the least qualified principals in the highest need schools – yeah… that’s when children will suffer. Even if we start by getting good principals where they are needed most, we must provide them the resources to attract and retain the “better” teachers.

To summarize:

1. Decentralizing control of teacher hiring to principals, where principal quality distribution is disparate, to the disadvantage of high need schools, is likely to lead to worse, not better distribution of teacher quality;

2. Altering contractual provisions, such as moving from seniority based to mutual consent placement, appears to disadvantage higher need schools initially and in the end, leads to little or no substantive change in the distribution of teachers across schools.

Yet, let’s go with it. What the heck. Why not – it chips away at those facially offensive protections of stubborn old selfish, lazy unproductive teachers (yes, I do know a few, but that’s not the point). We know it should work, even if we have no evidence to that effect. When organizations like NCTQ present policy recommendations against their own evidence and built on such flimsy logic, why do we even listen?

Additional resources

The NCTQ report above concludes with a rant about ESEA Comparability Regulations. For my thoughts, see: https://schoolfinance101.wordpress.com/2010/05/31/research-schmresearch-caps-misguided-analysis-again/

Regarding financial resources, “autonomy” and the distribution of teachers, see: Baker, B.D. Re-arranging deck chairs in Dallas: Contextual constraints on within district resource allocation in large urban Texas school districts. DeckChairsinDallas.Baker

Regarding the distribution of school leaders, teacher hiring and all that stuff, see:

Fuller, E., Young, M.D., Baker, B.D. Career Paths and the Influence of School Principals on Teachers. (Available on request)

Baker, B.D., Fuller, E. The Declining Academic Quality of School Principals and Why it May Matter. Baker.Fuller.PrincipalQuality.Mo.Wi_Jan7

Punswick, E., Baker, B.D., Belt, C. Principal backgrounds and school leadership stability: Evidence from Missouri. Educational Administration Quarterly Punswick.Baker.Belt.MoPrins09

Principal moves/exits: http://eaq.sagepub.com/content/46/4/523.abstract

Principals and hiring: http://eaq.sagepub.com/content/41/3/449.abstract

(the references in the above articles may provide some additional useful guidance on the role and the current distribution of principals)