I’ve long been perplexed that the Thomas B. Fordham Institute frames as its top two policy priorities:

1. Implementing the Common Core
2. Advancing Choice

Their new web site layout makes this more obvious.

More recently, a report released by the Council on Foreign Relations (referred to largely as the Rice-Klein report in the media and on twitter) argued that our “failing” education system is  a national security concern, and that the road to addressing that concern involves:

1. expanding the Common Core State Standard initiative to include subjects beyond math and English Language Arts;
2. an expansion of charter schools and vouchers

Now, as I understand it, there’s at least a subtle difference between these two sources on the point regarding vouchers and charter schools in that Fordham does not appear these days to be out front on promoting vouchers and instead seems to be favoring charter expansion (avoiding the word “voucher” but welcoming “other approaches that provide parents and children solid options and the capacity to make maximum use of them”).

Let me be clear that this post isn’t about favoring or slamming either vouchers or the common core, but rather pointing out that favoring both is entirely inconsistent, unless there’s some weird, warped agenda behind it all. This post IS about slamming the two, when used in combination. It just doesn’t make sense.  Let’s throw into this mix other policies promoting standardization of the operations of traditional public schools like forcing those schools to make personnel decisions based largely on student assessment data.

Collectively what we have here is a massive effort on the one hand, to require traditional public school districts to adopt a common curriculum and ultimately to adopt common assessments for evaluating student success on that curriculum and then force those districts to evaluate, retain and/or dismiss their teachers based on student assessment data, while on the other hand, expanding publicly financed subsidies for more children to attend schools that would not be required to do these things (in many cases, for example, relieving charter schools from teacher  evaluation requirements).

For example, if we believe that improving understanding of core scientific concepts is important for our national security or economic competitiveness, why would we be trying to increase the number of students who opt out of those standards, opting instead to attend fundamentalist religious institutions which may be decidedly anti-science? It seems like it would be one or the other? Certainly, TB Fordham Institute appears concerned with the importance of teaching science, and evolution specifically. When they simultaneously promote “other” choice alternatives, are they suggesting the regulation of science curriculum in those alternatives?

Also, if one believes that competitive pressures create improvement across schools (by stimulating innovation), why set up totally different rules – absurd constraints – in fact – for the largest set of schools in the mix. That seems rather counter productive and certainly limits any potential for real innovation. My critique all along about Race to the Top as a stimulus for innovation was that RTTT was anything but a stimulus for innovation and was instead a bribe to get states to fast-track a handful of preferred and completely unfounded reformy template policies – effectively squelching any real innovation that might have otherwise occurred.

One might instead argue for forcing all schools – public, private (if voucher receiving) and charter – to adopt the common core and evaluate teachers with student test data – and to simultaneously promote a broad based choice program. Yeah… let’s try really hard to make all schools the same and then let individuals choose among them? What we would have is a program that allows parents to choose which school adopts the common core better, and uses testing data better when firing teachers. That doesn’t seem to make a whole lot of sense, either.

No matter how you cut it, combining these two broad preferences leads to a ridiculous mix of policies, whichever side you’re coming from (unless, of course, you’re trying to come from both at once).

So, this all has me wondering if the real objective here – among advocates of these seemingly contradictory policies – is actually to make traditional public schooling so utterly unbearable for both teachers and students by expanding the testing and standards driven culture, expanding curricular standards across areas previously untouched, sucking any remaining creativity out of teaching, and mechanizing the teaching workforce in traditional public schools, making even the worst of the less-regulated alternatives seem more desirable for future generations of both teachers and students?

 

Taxpayer outrage arguments are in style these days (as if they ever really go out of style). Two particular taxpayer outrage arguments that have existed for some time seem to be making a bit of resurgence of late. Or, at least I think I’ve been seeing these arguments a bit more lately in the blogosphere and on twitter.  First, since now is the era of crapping on public school teachers and arguing for increased accountability specifically on teachers for improving student outcomes, there’s the “I pay your salary so you should cower to my every demand” argument (I’ve heard only a few warped individuals take this argument this far, but sadly I have!).  Second, there’s the persistent I pay for those schools and don’t even use them argument, or the variant on that argument that I pay twice for schools because I send my kids to private schools.

I (the taxpayer) pay your (the teacher) salary

This is a strange, obnoxious and easily diminished argument. Not that it’s not important to be sensitive to the demands of school constituents, but rather, that it’s more important to be sensitive to the demands of the broader public regarding their preferences for public schools more so than it is to be hypersensitive to any one loud-mouthed individual who would invoke this obnoxious argument. I explain more about that broader public under the next topic below.

For this one, a simple hypothetical is in order. Let’s assume the individual invoking this argument owns a residential property valued at $350,000 in a school district serving 5,000 students, where that district spends about $15,000 per pupil per year and where the effective property tax for schools is about 1.5%.  So, the school property tax bill on this house is 1.5% x 350k = $5,250.  Meanwhile the school total budget is 5k x 15k = 75 million. So, this one household is contributing far less than 1% (about .007%) of the district budget (which is then about $4.20 of a $60,000 teacher salary). And other households and owners of other property types within the district, as well as the broader base of taxpayers contributing to the state aid pot and any federal revenue sources all play a part in paying the salaries of teachers in this school.

This is by no means to suggest that any one person’s “say” in a district should be proportionate to one’s tax bill as a share of the budget. But rather, that one voice is one voice from the broader mix of voices that contribute to the financing of and shaping of public goods and services like schools.

[implications of disproportionate philanthropic giving from within or outside a district raise other serious questions to be addressed another day]

I (the taxpayer) pay for those public schools and don’t even use them!

Thus is the nature of public versus private goods. In the simplest model, taxpayers in a municipality contribute property taxes for a mix of public services, including local parks, fire protection, police and schools. I probably use our public park much less than others, and I rarely get a chance to attend the summer concert series. Should I get a refund for my contributed share, so that I can put that money towards buying Broadway tickets or a family vacation instead? Hey, I’m paying twice for entertainment and don’t even attend those free concerts in the park. And, I never sat on one of those benches. Can I please get a refund for my share of the cost of installing those benches and maintaining them, so I can invest in my own benches in my own back yard? And about those fancy fire trucks. We’ve had a few house fires in my town in my time living there. Why am I paying for the fire trucks to go to someone else’s house? Let’s say I live in a town that has public tennis courts, but I decide I want to join a private tennis club. Should I get a refund for the public courts I don’t use? In the amount of property tax I contributed? How about roads? Should I have to pay for roads I don’t ever intend to drive on?

Of late, I’ve been seeing the private school parent argument that “I pay twice for schools, since I pay my taxes for the public schools and pay private tuition.” This one is frequently invoked in conversations involving voucher and tuition tax credit programs. It should be noted that many residents of any given community pay taxes for schools – and all of the other stuff above – but may or may not use any of all of it. Families without school aged children also pay for schools. Further, since schools are financed by a mix of local, state and federal revenues, lots of different people, within and outside of any given community, are contributing to the financing of that community’s schools (to the extent that the schools receive intergovernmental revenue). Thus is the nature of publicly financed services.

But there’s even more to it than that. The above statements make the uninformed assumption that one receives absolutely no benefit from the presence and quality of these public goods and services simply because one does not make direct use of them. In reality (as well as in economic theory – which doesn’t always match reality), there’s this thing called capitalization! There is value to living in a community with such amenities as nice parks, good schools and police and fire protection. That value exists whether you actually use those things or not. That value is reflected in property values. As the quality and mix of services changes, those changes may be reflected in property values. Communities that have relatively better schools over time (even as reflected in crude grading systems in state accountability systems) see increases to property values. Residential property owners, not just those with kids in the public schools, see this benefit.

In short, the “I pay twice”, or “I pay for a service or amenity I don’t use” argument presents a dreadful oversimplification and misunderstanding of very basic principles of the provision of public goods and  services.

Instead, if taxpayers really want something to fuss about, read my previous post!

 

 

Lately it seems that public policy and the reformy rhetoric that drives it are hardly influenced by the vast body of empirical work and insights from leading academic scholars which suggests that such practices as using value-added metrics to rate teacher quality, or dramatically increasing test-based accountability and pushing for common core standards and tests to go with them are unlikely to lead to substantial improvements in education quality, or equity.

Rather than review relevant empirical evidence or provide new empirical illustrations in this post, I’ll do as I’ve done before on this blog and refer to the wisdom and practices of private independent schools – perhaps the most market driven segment and most elite segment of elementary and secondary schooling in the United States.

Really… if running a school like a ‘business’ (or more precisely running a school as we like to pretend that ‘businesses’ are run… even though ‘most’ businesses aren’t really run the way we pretend they are) was such an awesome idea for elementary and secondary schools, wouldn’t we expect to see that our most elite, market oriented schools would be the ones pushing the envelope on such strategies?

If rating teachers based on standardized test scores was such a brilliant revelation for improving the quality of the teacher workforce, if getting rid of tenure and firing more teachers was clearly the road to excellence, and if standardizing our curriculum and designing tests for each and every component of it were really the way forward, we’d expect to see these strategies all over the home pages of web sites of leading private independent schools, and we’d certainly expect to see these issues addressed throughout the pages of journals geared toward innovative school leaders, like Independent School Magazine.  In fact, they must have been talking about this kind of stuff for at least a decade. You know, how and why merit pay for teachers is the obvious answer for enhancing teacher productivity, and why we need more standardization… more tests… in order to improve curricular rigor? 

So, I went back and did a little browsing through recent, and less recent issues of Independent School Magazine and collected the following few words of wisdom:

From Winter 2003, when the school where I used to teach decided to drop Advanced Placement courses:

A little philosophy, first. Independent schools are privileged. We do not have to respond to the whims of the state, nor to every or any educational trend. We can maximize our time attuned to students and how they learn, and to the development of curriculum that enriches them and encourages the skills and attitudes of independent thinkers. Our founding charters and missions established independence for a range of reasons, but they now give all of us relative curricular autonomy, the ability to bring together a faculty of scholars and thinkers who are equipped to develop rich, developmentally sound programs of study. As Fred Calder, the executive director of New York State Association of Independent Schools, wrote in a letter to member schools a few years ago: “If we cannot design our programs according to our best lights and the needs of our communities, then let the monolith prevail and give up the enterprise. Standardized testing in subject areas essentially smothers original thought, more fatally, because of the irresistible pressure on teachers to teach to the tests.”

http://www.nais.org/publications/ismagazinearticle.cfm?ItemNumber=144300

Blasphemy? Or simply good education!

And from way, way back in 2000, in a particularly thoughtful piece on “business” strategies applied to schools:

Educators do not respond to the same incentives as businesspeople and school heads have much less clout than their corporate counterparts to foster improvement. Most teachers want higher salaries but react badly to offers of money for performance. Merit pay, so routine in the corporate world, has a miserable track record in education. It almost never improves outcomes and almost always damages morale, sowing dissension and distrust, for three excellent reasons, among others: (1) teachers are driven to help their own students, not to outperform other teachers, which violates the ethic of service and the norms of collegiality; (2) as artisans engaged in idiosyncratic work with students whose performance can vary due to factors beyond school control, teachers often feel that there is no rational, fair basis for comparison; and (3) in schools where all faculty feel underpaid, offering a special sum to a few sparks intense resentment. At the same time, school leaders have limited leverage over poor performers. Although few independent schools have unionized staff and formal tenure, all are increasingly vulnerable to legal action for wrongful dismissal; it can take a long time and a large expense to dismiss a teacher. Moreover, the cost of firing is often prohibitive in terms of its damage to morale. Given teachers’ desire for security, the personal nature of their work, and their comparative lack of worldliness, the dismissal of a colleague sends shock waves through a faculty, raising anxiety even among the most talented.

http://nais.org/publications/ismagazinearticle.cfm?ItemNumber=144267

Unheard of! Isn’t firing the bad teacher supposed to make all of those (statistically) great teachers feel better about themselves? Improve the profession? [that said, we have little evidence one way or the other]

How can we allow our leading private, independent, market-based schools to promote such gobbledygook? Why do they do it? Are they a threat to our national security or our global economic competitiveness because they were not then, nor are they now (see recent issues: http://www.nais.org/) fast-tracking the latest reformy fads? Testing out the latest and greatest educational improvement strategies on their own students, before those strategies get tested on low income children in overcrowded urban classrooms? Why aren’t the boards of directors of these schools – many of whom are leaders in “business” – demanding that they change their outmoded ways? Why? Why? Why? Because what they are doing works! At least in terms of their success in continuing to attract students and produce successful graduates.

Now, that’s not to say that these schools are completely stagnant, never adopting new strategies or reforms. They do new stuff all the time (technology integration, etc.) – just not the absurd reformy stuff being dumped upon public schools by policymakers who in many cases choose to send their own children to private independent schools.

In my repeated pleas to private school leaders to provide insights into current movements in teacher evaluation and compensation, I’ve actually found little change from these core principles of nearly a decade ago.  Private independent schools don’t just fire at will and fire often and teacher compensation remains very predictable and traditionally structured. I’d love to know, from my private school readers, how many of their schools have adopted state mandated tests?

Private independent schools pride themselves on offering small class sizes   (see also here) and a diverse array of curricular opportunities, as well as arts, sports and other enrichment – the full package.  And, as I’ve shown in my previous research, private independent schools charge tuition and spend on a per pupil basis at levels much higher than traditional public school districts operating in the same labor market. They also pay their headmasters well! More blasphemy indeed.

In fact, aside from “no excuses” charter schools whose innovative programs consist primarily of rigid discipline coupled with longer hours and small group tutoring (not rocket science), and higher teacher salaries (here, here & here) to compensate the additional work, private independent schools may just be among the least reformy elementary and secondary education options out there.

That’s not to say they are anything like “no excuses” charter schools. They are not in many ways. But they are equally non-reformy.  In fact, the average school year in private independent schools is shorter not longer than in traditional public schools – about 165 days.  And the average student load of teachers working in private independent schools (course sections x class size) is much lower in the typical private independent school than in traditional public schools. But that ain’t reformy stuff at all, any more than trying to improve outcomes of low income kids by adding hours and providing tutoring.

None-the-less, for some reason, well educated people with the available resources, keep choosing these non-reformy and expensive schools. Some of these schools have been around for a while too! Maybe, just maybe, it’s because they are doing the right things – providing good, well rounded educational opportunities as many of them have for centuries, adapting along the way (see: http://www.exeter.edu/admissions/109_1220_11688.aspx) .  Perhaps they’ve not gone down the road of substantially increased testing and curriculum standardization, test-based teacher evaluation – firing their way to Finland – because they understand that these policy initiatives offer little to improve school quality, and much potential damage.

Perhaps there are some lessons to be learned from market based systems. But perhaps we should be looking to those market based systems that have successfully provided high quality schooling for centuries to our nation’s most demanding, affluent and well educated leaders, rather than basing our policy proposals on some make-believe highly productive private sector industry where new technologies reduce production costs to near $0 and where complex statistical models are used to annually deselect non-productive employees.

Just pondering the possibilities, and still waiting for Zuck (an Exeter alum) to invest in Harkness Tables for Newark Public Schools and class sizes of 12 across the board!

Update

Mark Dynarski has added some additional useful recommendations regarding productivity research. Dynarski’s comments come in response to our suggestions for improving the rigor of productivity research, where our suggestions were based on rigorous application of relevant methods that we would expect to see applied in productivity research.

We agree with Mark Dynarski that using relevant methods alone doesn’t guarantee that they are used well.  We were starting from the position that the work of Roza and Hill doesn’t not apply relevant methods at all, no less well.  That in mind, we concur with Dynarski’s argument that it is not only important to use the right methods, but to use them well, and that reasonable standards may be applied. Here are Mark Dynarski’s suggestions:

Here are some examples of what I had in mind for research standards: the analysis has been replicated by another researcher working independently (replication being a lynchpin of scientific method). Predictions from the analysis have explanatory power outside the sample. The modeling framework is mathematically consistent. The research team has no conflicts of interest.

Applying these standards might result in excluding a lot of current research (even peer-reviewed research), but I think that would be the point Welner and Baker are making.

Readers interest in assessing research might take a look at the National Academy of Sciences’ Reference Manual on Scientific Evidence, now in its third edition, especially the chapter by Kaye and Freedman on statistics. It’s highly readable and available for free download from the academy’s website.

Below is my original reply to Mark Dynarski’s comment:

Over at Sara Mead’s Ed Week blog, Mark Dynarski checks in with a few relevant questions and observations. Actually, as it turns out, we agree ALMOST entirely with Dynarski when he says:

And focusing on peer-reviewed research as a form of quality assurance, as Baker and Welner suggest, seems problematic. Peer-reviewed research journals have highly variable degrees of editorial control, and peer review itself can vary from cursory reading to exhaustive and detailed comments. My own observation is that focusing on research with rigorous designs probably is a superior contributor to quality on average. There never seem to be enough of these when difficult debates on education policy issues arise, though.

Our only disagreement here is with his characterization of what we said. We did not uphold peer review as the gold standard. Though we probably used the phrase – peer review – too often in the brief itself. Rather, we believe just as Dynarski stated, that research with rigorous designs is a superior contributor to quality, on average! Hell yes. Absolutely. That’s our point. At the very least, the issues and questions at hand should be framed, or frame-able in relevant terms for rigorous evaluation.

That is precisely our concern with the Roza/Hill and Roza and other colleagues materials we address in our report (see pages 9 to 14). Further, a large section of our report summarizes the relevant methods – those rigorous and appropriate designs that should be applied to the questions at hand, but are noticeably absent even at the most cursory level in Roza and Hill’s materials.

To save you all the trouble of actually reading our entire brief, I’ve copied and pasted below the section of our brief where we address relevant methods:

Summary of Available Methods

Discussions of educational productivity can and should be grounded in the research knowledge base. Therefore, prior to discussing the Department of Education’s improving productivity project website and recommended resources, we think it important to explain the different approaches that researchers use to examine productivity and efficiency questions. Two general bodies of research methods have been widely used for addressing questions of improving educational efficiency. One broad area includes “cost effectiveness analysis” and “cost-benefit analysis.” The other includes two efficiency approaches: “production efficiency” and “cost efficiency.” Each of these is explained below.

 Cost-Effectiveness Analysis and Cost-Benefit Analysis

In the early 1980s Hank Levin produced the seminal resource on applying cost effectiveness analysis in education (with a second edition in 2001, co-written with Patrick McEwan),[i] helpfully titled “Cost-Effectiveness Analysis: Methods and Applications.” The main value of this resource is as a methodological guide for determining which, among a set of options, are more and less cost effective, which produce greater cost-benefit, or which have greater cost-utility.

The two main types of analyses laid out in Levin and McEwan’s book are cost-effectiveness analysis and cost-benefit analysis, the latter of which can focus on either short-term cost savings or longer term economic benefits. All these approaches require an initial determination of the policy alternatives to be compared. Typically, the baseline alternative is the status quo. The status quo is not a necessarily a bad choice. One embarks on cost-effectiveness or cost-benefit analysis to determine whether one might be able to do better than the status quo, but it is not simply a given that anything one might do is better than what is currently being done. It is indeed almost always possible to spend more and get less with new strategies than with maintaining the current course.

Cost-effectiveness analysis compares policy options on the basis of total costs. More specifically, this approach compares the spending required under specific circumstances to fully implement and maintain each option, while also considering the effects of each option on a common set of measures. In short:

Cost of implementation and maintenance of option A

Estimated outcome effect of implementing and maintaining option A

Compared to

Cost of implementation and maintenance of option B

Estimated outcome effect of implementing and maintaining option B

Multiple options may (and arguably should) be compared, but there must be at least two. Ultimately, the goal is to arrive at a cost-effectiveness index or ratio for each alternative in order to determine which provides the greatest effect for a constant level of spending.

The accuracy of cost-effectiveness analyses is contingent, in part, upon carefully considering all direct and indirect expenditures required for the implementation and maintenance of each option. Imagine, for example, program A, where the school incurs the expenses on all materials and supplies. Parents in program B, in contrast, are expected to incur those expenses. It would be inappropriate to compare the two programs without counting those materials and supplies as expenses for Program B. Yes, it is “cheaper” for the district to implement program A, but the effects of program B are contingent upon the parent expenditure.

Similarly, consider an attempt to examine the cost effectiveness of vouchers set at half the amount allotted to public schools per pupil. Assume, as is generally the case, that the measured outcomes are not significantly different for those students who are given the voucher. Finally, assume that the private school expenditures are the same as those for the comparison public schools, with the difference between the voucher amount and those expenditures being picked up through donations and through supplemental tuition charged to the voucher parents. One cannot claim greater “cost effectiveness” for voucher subsidies in this case, since another party is picking up the difference. One can still argue that this voucher policy is wise, but the argument cannot be one of cost effectiveness.

Note also that the expenditure required to implement program alternatives may vary widely depending on setting or location. Labor costs may vary widely, and availability of appropriately trained staff may also vary, as would the cost of building space and materials. If space requirements are much greater for one alternative, while personnel requirements are greater for the second, it is conceivable that the relative cost effectiveness of the two alternatives could flip when evaluated in urban versus rural settings. There are few one-size-fits-all answers.

Cost-effectiveness analysis also requires having common outcome measures across alternative programs. This is relatively straightforward when comparing educational programs geared toward specific reading or math skills. But policy alternatives rarely focus on precisely the same outcomes. As such, cost-effectiveness analysis may require additional consideration of which outcomes have greater value, which are more preferred than others. Levin and McEwan (2001) discuss these issues in terms of “cost-utility” analyses. For example, assume a cost-effectiveness analysis of two math programs, each of which focuses on two goals: conceptual understanding and more basic skills. Assume also that both require comparable levels of expenditure to implement and maintain and that both yield the same average combined scores of conceptual and basic-skills assessments. Program A, however, produces higher conceptual-understanding scores, while program B produces higher basic-skills scores. If school officials or state policy makers believe conceptual understanding to be more important, a weight might be assigned that favors the program that led to greater conceptual understanding.

In contrast to cost-effectiveness analysis, cost-benefit analysis involves dollar-to-dollar comparisons, both short-term and long-term. That is, instead of examining the estimated educational outcome effect of implementing and maintaining a given option, cost-benefit analysis examines the economic effects. But like cost-efficiency analysis, cost-benefit analysis requires comparing alternatives:

Cost of implementation and maintenance of option A

Estimated economic benefit (or dollar savings) of option A

Compared to

Cost of implementation and maintenance of option B

Estimated economic benefit (or dollar savings) of option B

Again, the baseline option is generally the status quo, which is not assumed automatically to be the worst possible alternative. Cost-benefit analysis can be used to search for immediate, or short-term, cost savings. A school in need of computers might, for example, use this approach in deciding whether to buy or lease them or it may use the approach to decide whether to purchase buses or contract out busing services. For a legitimate comparison, one must assume that the quality of service remains constant. Using these examples, the assumption would be that the quality of busing or computers is equal if purchased, leased or contracted, including service, maintenance and all related issues. All else being equal, if the expenses incurred under one option are lower than under another, that option produces cost savings. As we will demonstrate later, this sort of example applies to a handful of recommendations presented on the Department of Education’s website.

Cost-benefit analysis can also be applied to big-picture education policy questions, such as comparing the costs of implementing major reform strategies such as class-size reduction or early childhood programs versus raising existing teachers’ salaries or measuring the long-term economic benefits of those different programmatic options. This is also referred to as return-on-investment analysis.

While cost-effectiveness and cost-benefit analyses are arguably under-used in education policy research, there are a handful of particularly useful examples:

1. Determining whether certain comprehensive school reform models are more cost-effective than others?[ii]
2. Determining whether computer-assisted instruction is more cost-effective than alternatives such as peer tutoring?[iii]
3. Comparing National Board Certification for teachers to alternatives in terms of estimated effects and costs.[iv]
4. Cost-benefit analysis has been used to evaluate the long-term benefits, and associated costs, of participation in certain early-childhood programs.[v]

Another useful example is provided by a recent policy brief prepared by economists Brian Jacob and Jonah Rockoff, which provides insights regarding the potential costs and benefits of seemingly mundane organizational changes to the delivery of public education, including (a) changes to school start times for older students, based on research on learning outcomes by time of day; (b) changes in school-grade configurations, based on an increased body of evidence relating grade configurations, location transitions and student outcomes; and (c) more effective management of teacher assignments.[vi] While the authors do not conduct full-blown cost effectiveness or cost-benefit analyses, they do provide guidance on how pilot studies might be conducted.

Efficiency Framework

As explained above, cost-benefit and cost-effectiveness analyses require analysts to isolate specific reform strategies in order to correspondingly isolate and cost the strategies’ components and estimate their effects. In contrast, relative-efficiency analyses focus on the production efficiency or cost efficiency of organizational units (such as schools or districts) as a whole. In the U.S. public education system, there are approximately 100,000 traditional public schools in roughly 15,000 traditional public school districts, plus 5,000 or so charter schools. Accordingly, there is significant and important variation in the ways these schools get things done. The educational status quo thus entails considerable variation in approaches and in quality, as well as in the level and distribution of funding and the population served.

Each organizational unit, be it a public school district, a neighborhood school, a charter school, a private school, or a virtual school, organizes its human resources, material resources, capital resources, programs, and services at least marginally differently from all others. The basic premise of using relative efficiency analyses to evaluate education reform alternatives is that we can learn from these variations. This premise may seem obvious, but it has been largely ignored in recent policymaking. Too often, it seems that policymakers gravitate toward a policy idea without any empirical basis, assuming that it offers a better approach despite having never been tested. It is far more reasonable, however, to assume that we can learn how to do better by (a) identifying those schools or districts that do excel, and (b) evaluating how they do it. Put another way, not all schools in their current forms are woefully inefficient, and any new reform strategy will not necessarily be more efficient. It is sensible for researchers and policymakers to make use of the variation in those 100,000 schools by studying them to see what works and what does not. These are empirical questions, and they can and should be investigated.

Efficiency analysis can be viewed from either of two perspectives: production efficiency or cost efficiency. Production efficiency (also known as “technical efficiency of production”) measures the outcomes of organizational units such as schools or districts given their inputs and given the circumstances under which production occurs. That is, which schools or districts get the most bang for the buck? Cost efficiency is essentially the flip side of production efficiency. In cost efficiency analyses, the goal is to determine the minimum “cost” at which a given level of outcomes can be produced under given circumstances. That is, what’s the minimum amount of bucks we need to spend to get the bang we desire?

In either case, three moving parts are involved. First, there are measured outcomes, such as student assessment outcomes. Second, there are existing expenditures by those organizational units. Third, there are the conditions, such as the varied student populations,  and the size and location of the school or district, including differences in competitive wages for teachers, health care costs, heating and cooling costs, and transportation costs.

It is important to understand that all efficiency analyses, whether cost efficiency or production efficiency, are relative. Efficiency analysis is about evaluating how some organizational units achieve better or worse outcomes than others (given comparable spending), or how or why the “cost” of achieving specific outcomes using certain approaches and under some circumstances is more or less in some cases than others. Comparisons can be made to the efficiency of average districts or schools, or to those that appear to maximize output at given expense or minimize the cost of a given output. Efficiency analysis in education is useful because there are significant variations in key aspects of schools: what they spend, who they serve and under what conditions, and what they accomplish.

Efficiency analyses involve estimating statistical models to large numbers of schools or districts, typically over multiple years. While debate persists on the best statistical approaches for estimating cost efficiency or technical efficiency of production, the common goal across the available approaches is to determine which organizational units are more and less efficient producers of educational outcomes. Or, more precisely, the goal is to determine which units achieve specific educational outcomes at a lower cost.

Once schools or districts are identified as more (or less) efficient, the next step is to figure out why. Accordingly, researchers explore what variables across these institutions might make some more efficient than others, or what changes have been implemented that might have led to improvements in efficiency. Questions typically take one of two forms:

1. Do districts or schools that do X tend to be more cost efficient than those doing Y?
2. Did the schools or districts that changed their practices from X to Y improve in their relative efficiency compared to districts that did not make similar changes?

That is, the researchers identify and evaluate variations across institutions, looking for insights in those estimated to be more efficient, or alternatively, evaluating changes to efficiency in districts that have altered practices or resource allocation in some way. The latter approach is generally considered more relevant, since it speaks directly to changing practices and resulting changes in efficiency.[vii]

While statistically complex, efficiency analyses have been used to address a variety of practical issues, with implications for state policy, regarding the management and organization of local public school districts:

1. Investigating whether school district consolidation can cut costs and identifying the most cost-efficient school district size.[viii]
2. Investigating whether allocating state aid to subsidize property tax exemptions to affluent suburban school districts compromises relative efficiency.[ix]
3. Investigating whether the allocation of larger shares of school district spending to instructional categories is a more efficient way to produce better educational outcomes.[x]
4. Investigating whether decentralized governance of high schools improves efficiency.[xi]

These analyses have not always produced the results that policymakers would like to hear. Further, like many studies using rigorous scholarly methods, these analyses have limitations. They are necessarily constrained by the availability of data, they are sensitive to the quality of data, and they can produce different results when applied in different settings.[xii] But the results ultimately produced are based on rigorous and relevant analyses, and the U.S. Department of Education should be more concerned with rigor and relevance than convenience or popularity.

 

---

Current reformy rhetoric dictates that we MUST CLOSE FAILING SCHOOLS! That we must close those schools that are dropout factories or have persistently low achievement levels on state assessments. And, that we must, in the process, fire all of the staff in those schools that have caused these dismal conditions year after year, by thinking only of themselves, their tenure, their pensions and their wages – which are clearly too high for workers of their meager cognitive ability.

Take these simple bold steps and things will get better! Surely they will.

But, the bottom line is that you can’t just close down the poorest schools in any city school system and simply replace them with less poor ones – problem solved! That is, unless the larger strategy is actually about closing down entire neighborhoods, allowing them to become blighted, then seeking investors to step in and gentrify the area, replacing the old population with a new, less poor one! Problem solved. Or alternatively, if one relies on the off chance of a large scale natural disaster disproportionately displacing the poorest families to a large urban district in a neighboring state. But I digress.

A major unintended consequence of this ill-conceived reform movement is that it is distracting local school administrators and boards of education from closing and/or reorganizing schools for the right reasons by focusing all of the attention on closing schools for the wrong ones. In fact, even when school officials might wish to consider closing schools for logical reasons, they now seem compelled to say instead that they are proposing specific actions because the schools are “failing!” Not because they are too small to operate at efficient scale, that local demographic shift warrants reconsidering attendance boundaries, or that a facility is simply unsafe, or an unhealthy environment.

In really blunt terms, the current reformy rhetoric is forcing leaders to make stupid arguments for school closures where otherwise legitimate ones might actually exist!

There are legitimate reasons, cost saving reasons and other, to close schools and reorganize the delivery of educational services across organizational units and geographic locations within a district. Often, when I’m pushed to suggest the types of steps districts might take to achieve cost savings, the first issue I turn to is school organization/optimization.  Closing schools is not necessarily a bad thing. Closing schools for the wrong reasons and under the wrong pretexts is a bad thing. Reorganizing schools may lead to staffing reductions. These are cost cutting realities in a labor intensive industry. The fact is that you can’t really cut much from costs without cutting labor costs.  When enrollments decline significantly over time, fewer teachers are needed to get the job done and the staff may need to be reorganized.

But closing schools based on test scores, and pretending that we are somehow appropriately dismissing the staff that “caused” those low test scores is – well – just dumb.

Now let’s talk about some of the more legitimate reasons that a district might choose to close/reorganize schools.

First, let’s define “cost” and “cost cutting.” Cost is the minimum amount that needs to be spent to      achieve any given level of outcomes. It’s certainly possible to spend more than the minimum hypothetical – perfect world – cost of achieving any given level of outcomes. In fact, it’s pretty much a given that spending on outcomes occurs in less than perfect conditions, including unevenly growing and declining enrollments and unevenly distributed facilities capacity, quality and efficiency. Ultimately, the goal is to figure out how to reduce those barriers – less than perfect conditions – in order to get closer to that hypothetical minimum cost of achieving a given level of outcomes. In other words, the goal in times of budget cuts it to figure out how to spend less, but not compromise outcomes.

Here’s a short list of legitimate reasons a district might choose to close schools.

Economies of Scale

Operating unnecessarily small schools within a district creates inappropriate inequities. Providing more resources per pupil in one school necessarily means less in others. If those differences are based on legitimate differences in costs and student needs, that’s fine. It’s a difference that advances rather than erodes equity. But, sustaining inefficiently small schools at the expense of others within a large, population dense school district doesn’t meet those criteria. So, it’s in the best interest of the district as a whole to find ways to optimize the distribution of enrollments across schools within districts. To make sure, for example, that there aren’t elementary schools in one part of town with only 100 or so students, and in another part of town with 1,200 students. That there aren’t high schools with 300 to 400 students drawing  resources from high schools with 1,500 students. This can be really tricky to accomplish. But even moving toward optimal, while not reaching it is better than nothing. The literature on economies of scale suggests that elementary schools of 300 to 500 students and high schools of 600 to 900 students seem to produce optimal outcomes, and these sizes are consistent with literature that suggests that districts with 2000 to 4000 pupils seem to minimize costs of producing outcomes.

Facility efficiency

Some school facilities are simply more efficient to operate than others. They have more efficient mechanical/HVAC systems, are better insulated, have fewer deferred maintenance issues, potentially longer overall projected useful life.   Some facilities simply have more efficient space for accommodating the kinds of programs and services that need to be delivered. Evaluating the costs and benefits of maintaining and upgrading the current stock of facilities and whether children can be more efficiently distributed across “better” spaces with lower operations and maintenance costs is something any/all school districts should be engaged in on an ongoing basis.

Transportation efficiency

As population distribution shifts across spaces within a district, and while considering other reasons for reorganizing and redistributing students across schools – usually via changes to school attendance zones, but potentially with choice programs as well – evaluation of transportation efficiency should also be on the table.  In a district with dramatically declining enrollment or geographically shifting enrollment, school closings may be inevitable. In fact, a district may find itself closing some schools and selling off land, while opening others in different locations (less likely in more densely populated urban centers, but common in sprawling exurbia).

Health & Safety Concerns

This one is (or at least should be) a no brainer. Kids shouldn’t be housed in unsafe or unhealthy facilities.  That in mind, districts should engage in cost-benefit analyses to evaluate/compare the costs of improving the problem facilities/spaces versus other reorganization options.  Closing unsafe, unhealthy schools and appropriately distributing students among “better” spaces is obviously a legitimate reason for school closing.

Socioeconomic integration/balancing

A final reason why a district might close and/or reorganize schools to improve performance while maintaining (or cutting) spending, is to achieve better peer group balance across schools. Of course, this only works when the district is a) heterogeneous enough to be able to create better balanced peer groups and b) geographically small enough to not incur substantial transportation costs when implementing such a policy. A substantial body of research indicates that concentrated poverty and for that matter racial composition (racial isolation) in schools can affect the costs of achieving a given outcome target. Optimizing peer group composition across schools while considering interaction with other cost drivers (transportation) makes sense.  Of course, the U.S. Supreme Court has placed some constraints on the role of race in re-assignment policies, http://www.oyez.org/cases/2000-2009/2006/2006_05_908. But options remain available.

Improving peer group balance, optimizing school sizes, optimizing bus routes, making best use of most operationally efficient and educationally efficient learning spaces all can help districts both reduce costs and improve outcomes.

AND ABSOLUTELY NONE OF THIS HAS ANYTHING TO DO WITH CLOSING FAILING SCHOOLS.  Why, because there’s little or no evidence that closing “failing” schools improves either productivity or efficiency.

It’s not that sexy. It’s not reformy. It’s just good management decision making to get the most bang-for-the-buck. And it’s all stuff that districts can and should be working on constantly.

Closing schools is never easy. Someone will always be irked, no matter what the reason for the closure. A neighborhood will feel that it has lost its identity. Alums will feel that a piece of their childhood has been taken away.  So if we’re going to go down this road, and fight the difficult political fights that school closing plans create, then we ought to be closing the schools for the right reasons, and not the wrong ones!

 

Pau Hill and Marguerite Roza’s response to my recent report – with Kevin Welner – and series of blog posts seems to offer as its central argument that we’re simply a curmudgeons, offering lots of complaints about the rigor of their arguments and their suggestions for improving schooling productivity and efficiency, but providing no creative or immediately useful ideas or solutions for school districts or states in these tough economic times.

My first response would be that bad ideas are bad ideas, even in the absence of alternatives. The fact that budgets are tight and many schools are underperforming is not an argument for implementing unproven, ill-considered policy solutions.

That said, my second response is that Kevin Welner and I did in fact offer our own solutions, both in our policy brief and elsewhere.

On the first response, allow me to point out that many of Hill and Roza’s own suggestions for how states or school districts should deal with these tough economic times are not cost-cutting measures at all. They provide little or no potential short-run cost savings – where cost savings means spending reduction while not harming (or perhaps even improving) student outcomes. For example, among the “cost-cutting” solutions offered in Petrilli and Roza’s Stretching the School Dollar[i], brief, one of the resources recommended by the U.S. Department of Education, is designing and implementing new teacher evaluations. I may disagree with the shape that most of these new systems are taking, but I certainly agree that improving evaluations is a good idea.

It is not, however, a cost-cutting measure. In fact, it would require significant up-front investments, and whether or not these new systems will improve student outcomes (to say nothing of doing so cost-effectively) is a completely open question.

In addition, Kevin Welner and I address numerous concerns regarding other proposals in Roza’s work on pages 9 to 15 of our policy brief, including proposals (in) for states to simply cut off aid for limited English proficient children after two years, or to cap the distribution of special education aid. These are simply spending cuts, not cost savings and there is little or no evidence that such cuts would cause no harm, much less generate improvements.

Hill and Roza also suggest that their brief on curing “Baumol’s disease” provides useful insights into how private sector industries have found cures that might be translated to public education. Kevin Welner and I thoroughly rebut the basic premise of the Baumols’ disease claim and Hill and Roza’s proposed solutions on Page 10-11 of our report. In short, Kevin Welner and I explain about this Baumol’s hypothesis (& solutions):

In sum, the report begins with two highly contestable claims. It then draws an unsupported causal connection between the two claims. Further, it assumes that the problem is universal—that the system as a whole is diseased. In making this assumption, the authors ignore any possibility that lessons may be derived from within the public education system.

My second general response is that, in contrast with Roza and Hill’s characterization of my work, Kevin Welner and I offer up lists of issues that have been studied regarding productivity and efficiency in public schooling, as well as frameworks for studying those issues and for guiding decision making.  That is, our policy brief on this topic is not merely a list of curmudgeonly complaints and critiques of the work of Roza and others, though we do have serious concerns (elaborated through curmudgeonly complaints) about the quality of much of that work and claims made. In fact, I personally have even more serious concerns about related work from Roza and colleagues that has come to light since the writing of that brief. Yes, Kevin and I can really crank out those curmudgeonly complaints. But we don’t stop there… this time.

Among other things, Kevin and I point to research that offers potentially mundane solutions – the non-reformy stuff – like organizing schools within districts to optimize distributions of enrollments (school size) to achieve economies of scale – and we cite a handful of examples provided in a paper we cite by Brian Jacob & Jonah Rockoff[ii], including (a) changes to school start times for older students, based on research on learning outcomes by time of day; (b) changes in school-grade configurations, based on an increased body of evidence relating grade configurations, location transitions and student outcomes; and (c) more effective management of teacher assignments. Perhaps more importantly, we address methods and resources for guiding decision making on the basis of cost-effectiveness, and point out how cost effectiveness of particular options may, in fact, vary across settings.

It also bears mentioning that we don’t necessarily reject all of the ideas that Hill and Roza present. Rather, we explain that some might be explored, and might even be evaluated in pilot settings before we starting pitching them as large scale reforms. For instance, regarding large scale changes to teacher compensation systems, we explain:

That is not to say that such ideas cannot or should not be piloted and tested. They are, to some extent, researchable questions, most appropriately studied through relative efficiency analyses across districts and schools that are applying varied approaches, including the proposed new approaches.

In a forthcoming piece we go further to explain that some of our concerns regarding Roza and Hill’s current proposals are that they encourage large scale policy experimentation on the most vulnerable children, and we find that unacceptable given the extent of unknowns involved.  We explain (forthcoming):

Further, the political rhetoric around the immediacy of reform focuses on so-called failing schools, and failure is identified through performance metrics heavily influenced by student demographics. Simply put, we have ethical concerns with imposing unproven and sometimes unstudied policies on schools in low-income communities of color. And this is what we see happening.

There is time to figure some of this stuff out, and there are things we can look to do more immediately to achieve short run cost savings where necessary. As I often point out, when advocates use language like “And they need these ideas NOW,” it is most often a ploy to compel people into expediting ill-conceived policies with as much potential to do harm as to do good.

Many of the things that can and should be done in the short run, to reorganize local school district budgets, and to cut spending with minimal negative impact on outcomes, are really mundane things. They’re just not sexy enough to make for a good political platform or to generate public outcry. They don’t even lend themselves to good acronyms or catch phrases, like LIFO (last in, first out). They don’t have really cool, catchy names like Parent Trigger (note – I’m not blaming or even trying to connect Roza and Hill to Parent Trigger). And they can’t be likened to a disease to be cured.

They are instead ideas with a strong, empirically-based track record.

In the longer term, yes – there are policies that should be considered and tested, including those pertaining to teacher quality. But they are substantive education policies, not cost-cutting measures, and we should not be using a budget crisis to justify unwarranted haste and recklessness. Good policy making must rely on good policy analysis, and this relationship should not be severed simply because money is tight. If anything, it should be strengthened.

===============

Supplemental Note:

Hill and Roza offer as evidence that their proposed strategies have been evaluated in terms of productive efficiency and cost effectiveness, three links to related “studies”. One of those links points to a paper by Dan Goldhaber and Roddy Theobald, regarding potential costs/benefits of Seniority based versus “Quality-Based” layoffs. On the one hand, the paper does not yield any decisive guidance for short term budget planning and on the other hand, suffers the circular logic I’ve discussed on numerous occasions on my blog regarding measuring the effectiveness of the policy by the same measure used to implement the policy (e.g. did firing teachers with low value added scores leave us with more teachers with high value added scores).  The central conclusion of the paper(s) is that “Finally, simulations suggest that a very different group of teachers would be targeted for layoffs under an effectiveness-based layoff scenario than under the seniority-driven system that exists today.” This is hardly surprising, and of limited usefulness for informing state or local leaders on how to handle personnel decisions in tough budgetary times, or the expected benefits or downsides of such policies, and fails to address such basic issues as the costs of putting into place a system that might be used for making such decisions. I provide a hypothetical discussion of this topic in an earlier blog post. The other study cited is Eric Hanushek’s paper suggesting that teachers whose effectiveness ratings are one standard deviation above the mean can yield $400,000 benefit in present value of student future earnings with a class size of 20. This study also provides no guidance for how district administrators might cut costs, or even hold the line, while attracting or retaining teachers whose value added scores are a standard deviation higher than their current average. Rather, this study speaks to the kind of large scale deselection which I’ve discussed numerous times in this blog. The third “study” is not a study at all, but rather an opinion brief by Roza with relatively meaningless national ball park estimates of job loss under alternative dismissal scenarios.

Interview for In These Times:

[I]t’s much easier to point blame at those working within the system–like teachers–than to actually raise the revenues to provide the resources necessary to really improve the system–to pay sufficient wages to attract and retain top college graduates and to provide the working conditions that would make teaching more appealing–including smaller total student loads… and higher quality infrastructure, materials, supplies, equipment and other supports.

http://www.inthesetimes.com/working/entry/12618/teachers_and_communities_overshadowed_by_corporate_fixes_for_schools/

In my interview with Geoff Mulvihill of AP:

In response to what reforms are needed most in New Jersey?

From a research angle, if you looked at the high-performing and the low-performing schools and you asked yourself what’s different about them, well, our highest-performing schools also have step-structured pay scales, collective bargained agreements, tenure, union contracts as do our low-performing schools. That’s not a differentiating factor.

…

These things that we’re talking about like merit pay, disrupting union contracts and collective bargaining don’t tend to be the things that the high-performing schools are doing.

http://www.courierpostonline.com/article/20120103/NEWS02/301030016/Educating-New-Jersey-s-urban-kids-costs-more-scholar-says?odyssey=nav|head

Follow up in a similar question

If you look at the biggest differences between the schools that are doing well and the schools that are doing poorly, there may be differences in teaching quality. There may be differences in skill-set of the teachers who are sorting themselves among the more and less desirable schools.
…
It may be that we’ve got some inequities in teaching quality. But to suggest that those inequities are a function of not having merit pay or they’re a function of having collective bargaining and a union presence doesn’t seem to fit when those structures also exist in the highly successful and affluent districts.

http://www.courierpostonline.com/article/20120103/NEWS02/301030016/Educating-New-Jersey-s-urban-kids-costs-more-scholar-says?odyssey=nav|head

On where to go from here:

I think we’ve got to keep up the effort of targeting resources toward the high-need districts, and the key is that equitable and adequate funding — and this is my big punchline — is the necessary condition for everything. If you want to run a good charter school, if you want to run a good public school, you’ve got to have enough money to do a good job.