A few weeks back, I noticed several tweets about this recent article in Harvard Education Review which takes a look at racial politics and the rebuilding of New Orleans in the Post-Katrina era.

Here’s the dropbox link tweeted by Diane Ravitch:

http://dl.dropbox.com/u/11116752/Buras_2011-Race_Charter_Schools_Conscious_Capitalism.pdf.pdf

The article is by Kristen Buras of Georgia State University. Buras, like at least a few others, points out that Hurricane Katrina forced the greatest housing displacement in poor black neighborhoods of New Orleans. But, perhaps more disturbing was that in the post Katrina period, redevelopment… and especially redevelopment of the new, mixed delivery schooling system largely ignored those same areas, leading to a system where access to schooling is very disparately distributed geographically.

In her article Buras went to the painstaking steps of hand plotting the locations of post-Katrina schools (See her Figure 3, page 321) to make her point about school locations, and that map certainly does so, though a good before-after might be even clearer.

I’ve been meaning to do some pre-post Katrina school mapping for some time now, but wasn’t quite sure what I wanted to look at, or how I might organize the information. Well, here’s what a little Friday afternoon play has yielded.

First, I used US Census 2000 and American Community Survey 2005 data to set up my background. The background carves New Orleans into Public Use Micro Data Areas (PUMAS, from http://www.ipums.org, boundary files from http://www.census.gov). For the background shading, I used IPUMS data to estimate the percent of resident 5 to 17 year olds in each PUMA that were Black in 2000 and 2005 – pre-Katrina conditions. Those red areas to the right hand side, over toward the lower 9th ward and to the Northeast are almost entirely black, for school aged population. While the entire city has relatively high shares of black population, as Buras notes, uptown and the Garden District are certainly somewhat less black than other parts of the city.

In the first map here, I show the locations and total enrollments of schools (indication of available slots) for the year 2000. I use yellow triangles to indicate if a school is a charter school. There were a few, even in 2000. School locations are based on latitude and longitude data from the National Center for Education Statistics Common Core of Data (www.nces.ed.gov/ccd).

Map 1. Year 2000 distribution of traditional public and charter schools in New Orleans

In the first figure, there are a significant number of decent size schools in the deeper red (higher % black) areas of the city. Citywide, there are a handful of charters scattered around.

Now, here’s the distribution of charters and traditional public schools in 2010. Yes, the city as a whole lost a lot of population (but did rebound somewhat between 2006 and 2010, hence the interest in 2010). Quite strikingly, there are simply very few schools of any size now available in those deep red zones (shading still based on pre-Katrina population). And while there are charters scatted throughout the city, even the highest concentration of those schools is in areas with marginally lower pre-Katrina black populations. There are generally more schools and more larger schools in those neighborhoods.

Again, circle size indicates enrollment size, and if the circle has a yellow triangle over it, the school is a charter school.  Further, I’ve kept the size scaling of circles on the same scale in this map as in the previous one. So, if a circle is smaller, it’s enrollment is smaller.

Map 2. Year 2010 distribution of traditional public and charter schools in New Orleans

Now, it is indeed hard to untangle supply from demand here. One can make the argument that the population didn’t return, therefore there is no demand for schools in those areas previously inhabited by the city’s lowest income black populations. Alternatively, one can as reasonably (and more so after reading Buras) argue that the dearth of available public services may provide some explanation for why families have not returned, or have not been able to return.

One might argue that because there exist so many “schools of choice” throughout the city, that geographic location doesn’t really matter. Ya’ just got to travel a bit. Sign up for one of those great schools over there! But research has consistently shown that even in “choice’ models geographic location/proximity is central to enrollment decisions.  Location matters. And having quality options nearby is important. In fact, parents will often favor location over publicly available “quality” measures, continuing enrollment in schools identified as persistently failing if/when other options are simply not geographically accessible. Then again, those “quality” measures aren’t always particularly meaningful.

This population density map for individuals 18 and under suggests comparable population densities in those areas where school density (especially charter school density) has remained much lower: http://www.gnocdc.org/LossOfChildrenInNewOrleansNeighborhoods/Map3.html

Authors such as Henry Levin have explained on numerous occasions that for a choice model to yield equitable distribution of opportunity, consumers must have equitable access to information on schools and equitable mobility among options. Clearly, equitable geographic access is out the window in Post-Katrina New Orleans. Yeah, I think we already knew this from various media reports. But sometimes I have to play with the data and map them myself for it to really sink in. Whether driven by geographic assignment or by choice enrollment, the distribution of educational opportunities in Map 2 above is troublesome.

Far more troublesome is that so many have publicly pitched this New Orleans mixed delivery model as the key to the future of urban education.

Like Buras, I’m pretty damn skeptical that an education system that has redistributed educational opportunity in the ways seen between Map 1 and Map 2 above is all that.  Just pondering and mapping on a Friday afternoon as the sun finally emerges in the rain-soaked Northeast.

Related maps on school aged population loss here: http://www.gnocdc.org/LossOfChildrenInNewOrleansNeighborhoods/index.html

I’ve previously written about the growing number of rigorous peer reviewed and other studies which tend to show positive effects of state school finance reforms. But what about all of those accounts to the contrary? The accounts that seem so dominant in the policy conversations on the topic. What is that vast body of research that suggests that school finance reforms don’t matter? That it’s all money down the rat-hole. That in fact, judicial orders to increase funding for schools actually hurt children?

Beyond utterly absurd graphs and tables like Bill Gates’ “turn the curve upside down” graph, and Dropout Nation’s even more absurd graph, there have been a handful of recent studies and entire books dedicated to proving that court ordered school finance reforms simply have no positive effect on children. Some do appear in peer reviewed journals, despite egregious (and really obvious) methodological flaws. And yes, some really do go so far as to claim that court ordered school finance reforms “harm our children.”[1]

The premise that additional funding for schools often leveraged toward class size reduction, additional course offerings or increased teacher salaries, causes harm to children is, on its face, absurd. Further, no rigorous empirical study of which I am aware actually validates that increased funding for schools in general or targeted to specific populations has led to any substantive, measured reduction in student outcomes or other “harm.”

But questions regarding measurement and validation of positive effects versus non-effects are complex. That said, while designing good research analyses can be quite complex, the flaws of bad analyses are often absurdly simple. As simple as asking three questions: a) whether the reform in question actually happened? b) when it happened and for how long? and c) who was to be affected by the reform?

• Whether: Many analyses argue to show that school funding reforms had no positive effects on outcomes, but fail to measure whether substantive school funding reforms were ever implemented or whether they were sustained. Studies of this type often simply look at student outcome data in the years following a school funding related ruling, creating crude classifications of who won or lost the ruling. Yet, the question at hand is not whether a ruling in-and-of-itself leads to changes in outcomes, but whether reforms implemented in response to a ruling do. One must, at the very least, measure whether reform actually happened!
• When: Many analyses simply pick two end points, or a handful of points of student achievement to cast as a window, or envelop around a supposed occurrence of school finance reform or court order, often combining this strategy with the first (not ever measuring the reform itself). For example, one might take NAEP scores from 1992 and 2007 on a handful of states, and indicate that sometime in that window, each state implemented a reform or had a court order. Then one might compare the changes in outcomes from 1992 to 2007 for those states to other states that supposedly did not implement reforms or have court orders. This, of course provides no guarantee that states from the non-reform group (a non-controlled control group?) didn’t actually do something more substantive than the reform group. But, that aside, the casting of a large time window and the same time window across states ignores the fact that reforms may come and go within that window, or may be sufficiently scaled up only during the latter portion of the window. It makes little sense, for example to evaluate the effects of New Jersey’s school finance reforms which experienced their most significant scaling up between 1998 and 2003, by also including 6 years prior to any scaling up of reform. Similarly, some states which may have aggressively implemented reforms at the beginning of the window may have seen those reforms fade within the first few years. When matters!
• Who: Many analyses also address imprecisely the questions of “who” is expected to benefit from the reforms. Back to the “whether” question, if there was no reform, then the answer to this question is no-one. No-one is expected to benefit from a reform that didn’t ever happen. Further, no-one is expected to benefit today from a reform that may happen tomorrow, nor is it likely that individuals will benefit twenty years from now from a reform that is implemented this year, and gone within the next three years. Beyond these concerns, it is also relevant to consider whether the school finance reform in question, if and when it did happen, benefited specific school districts or specific children. Reforms that benefit poorly funded school districts may not also uniformly benefit low income children who may be distributed, albeit unevenly, across well-funded and poorly-funded districts. Not all achievement data are organized for appropriate alignment with funding reform data. And if they are not, we cannot know if we are measuring the outcomes of who we would actually expect to benefit.

In 2011, Kevin G. Welner of the University of Colorado and I published an extensive review of the good, the bad and the ugly of research on the effectiveness of state school finance reforms.[2] In our article we identify several specific examples of empirical studies claiming to find (not just “find” but prove outright) that school funding reforms and judicial orders simply don’t matter. That is, they don’t have any positive effects on measured student outcomes. But, as noted above, many of those studies suffer from basic flaws of logic in their research design, which center on questions of whether, when and who.

As one example of a whether problem, consider an article published by Greene and Trivett (2008). Greene and Trivitt claim to have found “no evidence that court ordered school spending improves student achievement” (p. 224).  The problem is that the authors never actually measured “spending” and instead only measured whether there had been a court order. Kevin Welner and I explain:

The Greene and Trivitt article, published in a special issue of the Peabody Journal of Education, proclaimed that the authors had empirically estimated “the effect of judicial intervention on student achievement using standardized test scores and graduation rates in 48 states from 1992 to 2005” and had found “no evidence that court ordered school spending improves student achievement” (p. 224, emphasis added). The authors claim to have tested for a direct link between judicial orders regarding state school funding systems and any changes in the level or distribution of student outcomes that are statistically associated with those orders. That is, the authors asked whether a declaration of unconstitutionality (nominally on either equity or adequacy grounds) alone is sufficient to induce change in student outcomes. The study simply offers a rough indication of whether the court order itself, not “court-ordered school spending,” affects outcomes. It certainly includes no direct test of the effects of any spending reforms that might have been implemented in response to one or more of the court orders.

Kevin Welner and I also raise questions regarding “who” would have benefited from specific reforms and “when” specific reforms were implemented and/or faded out. In our article, much of our attention regarding who and when questions focused on Chapter 6, The Effectiveness of Judicial Remedies of Eric Hanushek and Alfred Lindseth’s book Courting Failure.[3] A downloadable version of the same graphs and arguments can be found here: http://edpro.stanford.edu/Hanushek/admin/pages/files/uploads/06_EduO_Hanushek_g.pdf.  Specifically, Hanushek and Lindseth identify four states, Kentucky, Massachusetts, New Jersey and Wyoming as states which have by order of their court systems, (supposedly) infused large sums of money into school finance reforms over the past 20 years. Given this simple classification, Hanushek and Lindseth take the National Assessment (NAEP) Scores for these states, including scores for low income children, and racial subgroups, and plot those scores against national averages from 1992 to 2007.

No statistical tests are performed, but graphs are presented to illustrate that there would appear to be no difference in growth of scores in these states relative to national averages. Of course, there is also no measure of whether and how funding changed in these states compared to others. Additionally, there is no consideration of the fact that in Wyoming, for example, per pupil spending increased largely as a function of enrollment decline and less as a function of infused resources (the denominator shrunk more than the numerator grew).

Setting these other major concerns aside, which alone undermine entirely the thesis of Hanushek and Lindseth’s chapter, Kevin Welner and I explain the problem of using a wide time window to evaluate school finance reforms which may ebb and flow throughout that window:

As noted earlier, the appropriate outcome measure also depends on identifying the appropriate time frame for linking reforms to outcomes. For example, a researcher would be careless if he or she merely analyzed average gains for a group of states that implemented reforms over an arbitrary set of years. If a state included in a study looking at years 1992 and 2007 had implemented its most substantial reforms from 1998 to 2003, the overall average gains would be watered down by the six pre-reform years – even assuming that the reforms had immediate effects (showing up in 1998, in this example). And, as noted earlier, such an “open window” approach may be particularly problematic for evaluating litigation-induced reforms, given the inequitable and inadequate pre-reform conditions that likely led to the litigation and judicial decree.

There also exist logical, identifiable, time-lagged effects for specific reforms. For example, the post-1998 reforms in New Jersey included implementation of universal pre-school in plaintiff districts. Assuming the first relatively large cohorts of preschoolers passed through in the first few years of those reforms, a researcher could not expect to see resulting differences in 3rd or 4th grade assessment scores until four to five years later.

Further, as noted previously, simply disaggregating NAEP scores by race or low income status does not guarantee by any stretch that one has identified the population expected to benefit from specific reforms. That is, race and poverty subgroups in the NAEP sample are woefully imprecise proxies for students attending districts most likely to have received additional resources. Kevin Welner and I explain:

This need to disaggregate outcomes according to distributional effects of school funding reforms deserves particular emphasis since it severely limits the use of the National Assessment of Educational Progress – the approach used in the recent book by Hanushek and Lindseth. The limitation arises as a result of the matrix sampling design used for NAEP. While accurate when aggregated for all students across states or even large districts, NAEP scores can only be disaggregated by a constrained set of student characteristics, and those characteristics may not be well-aligned to the district-level distribution of the students of interest in a given study.

Consider, for example, New Jersey – one of the four states analyzed in the recent book. It might initially seem logical to use NAEP scores to evaluate the effectiveness of New Jersey’s Abbott litigation, to examine the average performance trends of economically disadvantaged children. However, only about half (54%) of New Jersey children who receive free or reduced-price lunch – a cutoff set at 185% of the poverty threshold – attend the Abbott districts. The other half do not, meaning that they were not direct beneficiaries of the Abbott remedies. While effects of the Abbott reforms might, and likely should, be seen for economically disadvantaged children given that sizeable shares are served in Abbott districts, the limited overlap between economic disadvantage and Abbott districts makes NAEP an exceptionally crude measurement instrument for the effects of the court-ordered reform.¹⁶

Hanushek and Lindseth are not alone in making bold assertions based on insufficient analyses, though Chapter 6 of their recent book goes to new lengths in this regard. Kevin Welner and I address numerous comparably problematic studies with more subtle whether, who and when problems, including the Greene and Trivitt study noted above.  Another example is a study by Florence Neymotin of Kansas State University, which purports to find that the substantial infusion of funding into Kansas school districts which supposedly occurred between 1997 and 2006 as a function of the Montoy rulings never led to substantive changes in student outcomes. I blogged about this study when it was first reported. But, the most relevant court orders in Montoy did not come until January of 2005, June of 2005 and eventually July of 2006. Remedy legislation may be argued to have begun as early as 2005-06, but primarily from 2006-07 on, before its dismantling from 2008 on. Regarding the Neymotin study, Kevin Welner and I explain:

A comparable weakness undermines a 2009 report written by a Kansas State University economics professor, which contends that judicially mandated school finance reform in Kansas failed to improve student outcomes from 1997 to 2006 (Neymotin, 2009).¹³ This report was particularly egregious in that it did not acknowledge that the key judicial mandate was issued in 2005 and thus had little or no effect on the level or distribution of resources across Kansas schools until 2007-08. In fact, funding for Kansas schools had fallen behind and become less equitable from 1997 through 2005.¹⁴ Consequently, an article purporting to measure the effects of a mandate for increased and more equitable spending was actually, in a very real way, measuring the opposite.[4]

Kevin Welner and I also review several studies applying more rigorous and appropriate methods for evaluating the influence of state school finance reforms. I have discussed those studies previously here. On balance, it is safe to say that a significant body of rigorous empirical literature, conscious of whether, who and when concerns, validates that state school finance reforms can have substantive positive effects on student outcomes including reduction of outcome disparities or increased overall outcome level.

Further, it is even safer to say that analyses provided in sources like the book chapter by Hanushek and Lindseth (2009), or research articles by Neymotin (2009), Greene and Trivett, provide no credible evidence to the contrary, due to significant methodological omissions. Finally, even the boldest, most negative publications regarding state school finance reforms provide no support for the contention that school finance reforms actually “harm our children,” as indicated in the title of a 2006 volume by Eric Hanushek.

Sometimes, even when a research report or article seems really complicated, relatively simple questions like when, whether and who allow the less geeky reader to quickly evaluate and possibly debunk the study entirely.  Sometimes, the errors of reasoning regarding when, whether and who, are so absurd that it’s hard to believe that anyone would actually present such an absurd analysis. But these days, I’m rarely shocked. My personal favorite “when” error remains the Reason Foundation’s claim that numerous current reforms positively affected past results! http://nepc.colorado.edu/bunkum/2010/time-machine-award. It just never ends!

Further reading:

B. Baker, K.G. Welner (2011) Do School Finance Reforms Matter and How Can We Tell. Teachers College Record. http://www.tcrecord.org/content.asp?contentid=16106

Card, D., and Payne, A. A. (2002). School Finance Reform, the Distribution of School Spending, and the Distribution of Student Test Scores. Journal of Public Economics, 83(1), 49-82.

Roy, J. (2003). Impact of School Finance Reform on Resource Equalization and Academic Performance: Evidence from Michigan. Princeton University, Education Research Section Working Paper No. 8. Retrieved October 23, 2009 from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=630121(Forthcoming in Education Finance and Policy.)

Papke, L. (2005). The effects of spending on test pass rates: evidence from Michigan. Journal of Public Economics, 89(5-6). 821-839.

Downes, T. A., Zabel, J., and Ansel, D. (2009). Incomplete Grade: Massachusetts Education Reform at 15. Boston, MA. MassINC.

Guryan, J. (2003). Does Money Matter? Estimates from Education Finance Reform in Massachusetts. Working Paper No. 8269. Cambridge, MA: National Bureau of Economic Research.

Deke, J. (2003). A study of the impact of public school spending on postsecondary educational attainment using statewide school district refinancing in Kansas, Economics of Education Review, 22(3), 275-284.

Downes, T. A. (2004). School Finance Reform and School Quality: Lessons from Vermont. In Yinger, J. (ed), Helping Children Left Behind: State Aid and the Pursuit of Educational Equity. Cambridge, MA: MIT Press.

Resch, A. M. (2008). Three Essays on Resources in Education (dissertation). Ann Arbor: University of Michigan, Department of Economics. Retrieved October 28, 2009, from http://deepblue.lib.umich.edu/bitstream/2027.42/61592/1/aresch_1.pdf

Goertz, M., and Weiss, M. (2009). Assessing Success in School Finance Litigation: The Case of New Jersey. New York City: The Campaign for Educational Equity, Teachers College, Columbia University.

When Paul Peterson asks “Do we really need to spend more on schools?” we already know what he thinks the answer is – an unequivocal NO!  Knowing the answer you desire always makes it easier to frame the questions, and like previous years, this year’s Education Next survey of attitudes toward public education provides few surprises.

Before I even gained full access to Peterson’s most recent WSJ Op-ed (e-mailed to me by a family member), I was able to guess pretty much where he was going with it.  Here’s how Peterson explains the Ed Next public opinion survey findings:

At first glance, the public seems to agree with this position. In a survey released this week by Education Next, an education research journal, my colleagues and I reported that 65% of the public wants to spend more on our schools. The remaining 35% think spending should either be cut or remain at current levels. That’s the kind of polling data that the president’s political advisers undoubtedly rely upon when they decide to appeal for more education spending.

Yet the political reality is more complex than those numbers suggest. When the people we surveyed were told how much is actually spent in our schools—$12,922 per student annually, according to the most recent government report—then only 49% said they want to pony up more dollars. We discovered this by randomly splitting our sample in half, asking one half the spending question cold turkey, while giving the other half accurate information about current expenditure.

Later in the same survey, we rephrased the question to bring out the fact that more spending means higher taxes. Specifically, we asked: “Do you think that taxes to fund public schools around the nation should increase, decrease or stay about the same?” When asked about spending in this way, which addresses the tax issue frankly, we found that only 35% support an increase. Sixty-five percent oppose the idea, saying instead that spending should either decrease or stay about the same. The majority also doesn’t want to pay more taxes to support their local schools. Only 28% think that’s a good idea.

So there is the nation’s debt crisis in a nutshell. If people aren’t told that nearly $13,000 is currently being spent per pupil, or if they aren’t reminded that there is no such thing as a free lunch, they can be persuaded to think schools should be spending still more.

In other words… yeah… the ignorant general public thinks they want to spend more on schools, but only because they don’t realize how much we are already wasting on public schools! When we clue them into the egregious… no… outrageous… exorbitant spending already going on … and hold a gun to their head… and phrase our question just right… pointing out to them just how stupid we think they are… and how smart we are… then the fix their answer… and become much, much more reasonable!

This explanation is problematic at a number of levels.  First, let’s explore the basic model of local voter preferences for spending on local public schools – specifically the information on price and quality that informs those preferences. First, local public school revenue comes from two primary sources – local property taxes paid on various types of properties within school districts and state general funds derived largely from state sales and income taxes. The mix varies widely from state to state. Residential property owners frequently pay their property taxes embedded in monthly mortgage payments and renters pay their landlords’ property taxes embedded in rent prices. Homeowners and renters have at least some feel for the reasonableness of their aggregate monthly housing payments, and some feel for the quality of public services they receive (schools, fire, police, parks, etc.) for the aggregate price they pay. They also have some feel for a) whether they would like those services improved and b) whether they are willing to pay a bit more to support those improvements. In short, a typical taxpayer/survey respondent has a reasonable gut feel regarding their “tax price” paid for the quality of public service provided.

The local taxpayer/voter/survey respondent sufficiently involved with local public schools (having children in the schools, working in the schools, having children who are recent graduates of the schools, or having recently graduated themselves) probably has some indicators of schooling quality in his/her head that guide his/her preference to pay more (or less). Has class size risen, or does it just seem too large? Has the district cut visible programs like music, arts or athletics of late, or has the district increased fees to cover the costs of these programs? As a result, the respondent is at least somewhat able to piece together whether they wish to spend a little more to decrease class sizes, expand programs or reinstate programs previously cut.

But, the typical taxpayer/voter/survey respondent likely a) doesn’t give a damn about and b) is generally unable to contextualize the meaning of the Total per Pupil Expenditures for a local public school district. It’s an abstract concept. A number that relates in a meaningful sense only to those who really spend their days steeped in such numbers. A number most likely to do little more than bias a response in this case, and it seems to, though it is hard to know precisely why.

Even worse is when those numbers are used totally out of context, as in Peterson’s argument above. Peterson’s description above is actually even worse than the methods description provided at Ed Next (Interestingly, Peterson also adds over $600 per pupil to the average spending figure, and then rounds it up to $13,000 by the end of his op-ed, compared to the information in the paragraph below from Ed Next):

A segment of those surveyed were asked the same ques­tion except that they were first told the level of per-pupil expenditure in their community, which averaged $12,300 for the respondents in our sample. For every subgroup con­sidered, this single piece of information dampened public enthusiasm for increased spending. Support for more spend­ing fell from 59 percent to 46 percent of those surveyed. Among the well-to-do, the level of support dropped dramati­cally, from 52 percent to 36 percent. Among teachers, sup­port for expenditure increases fell even more sharply—from 71 percent to 53 percent (see Figure 7).

Surely, it would be completely absurd to ask (as implied by Peterson’s op ed) the average person in Tennessee if their schools should spend more, after telling that person what the average district spends nationally – implying to the respondent that the figure represents Tennessee spending (as seemingly implied by Peterson’s Op-Ed, and as in the online survey at Ed Next).  It is only marginally more useful, however to ask the average respondent in Tennessee whether they should spend more or less, given a completely out of context representation of their local spending per pupil.

Here’s how the 2008-09 actual national mean per pupil spending compares to the distribution of per pupil spending across Tennessee districts:

(national mean current spending per pupil in 2008-09 was $10,209.83 [w/outliers excluded])

Now, it might be interesting to show the average voter respondent in Tennessee this graph and then ask him/her whether they think more should be spent in Tennessee? This graph provides some context. Context that is completely absent when informing a Tennessee respondent either of their own local district spending WITH NO OTHER CONTEXT AVAILABLE or of the national spending WITH NO OTHER CONTEXT AVAILABLE.

Put very simply, a per pupil spending figure out of context is meaningless.  $17,000 I say! $17,000… an abomination I say. It’s  a huge number! Why would we ever consider spending more than that per pupil in New York City? Well, what if it just happened to turn out that in the same year, that $17,000 per pupil was lower, on average, than most of the surrounding districts with much less needy student populations? What if that $17,000 was only approximately 50% of what was being spent in private independent schools operating within the city?  It doesn’t sound so big any more does it?  How would survey respondents in New York City change their answer if this information was provided?

The Ed Next survey, while fun to ponder each year, isn’t particularly helpful for really understanding voter’s preferences or awareness regarding spending on public schools or perceived quality.

Actual data on local budget votes, including those involving tax increases (increasing the more voter-distasteful local property tax) tend to be a much more useful barometer and even in the worst of economic times, local voter support – especially where voters have the financial capacity to provide that support – remains overwhelmingly positive  (Example NY State Data & previous NJ Blog Post [over 70% pass rate in wealthy districts in worst year]).  Matt Di    Carlo provides further discussion of this topic here, explaining the general voter preferences. It is also worth noting that even the most poorly constructed and phrased polls do not find significant shares (if any) responding that less should be spent.  Yet that is precisely the argument advanced by many pundits in response to these surveys.

Earlier today I stumbled across this brilliant post by RiShawn Biddle over at Dropout Nation.

Biddle boldly claims:

Despite the arguments (and the pretty charts) of such defenders as Rutgers’ Bruce Baker, there is no evidence that spending more on American public education will lead to better results for children.

Now, regarding the “no evidence” claim, I would recommend reading this article from Teachers College Record, this year, which summarizes a multitude of rigorous empirical studies of state school finance reforms finding generally that increased funding levels have been associated with improved outcomes and that more equitable distributions of resources have been associated with more equitable distributions of outcomes.

In fact, even the Spring 2011 issue of the journal Education Finance and Policy includes an article by Joydeep Roy supporting the positive results of state school finance reforms (using Michigan data).

Proposal A was quite successful in reducing interdistrict spending disparities. There was also a significant positive effect on student performance in the lowest-spending districts as measured in state tests.(from abstract)

As Kevin Welner and I point out in our article, this study is not unique in its findings. Here are a few others:

Card & Payne (2002)

Using micro samples of SAT scores from this same period, we then test whether changes in spending inequality affect the gap in achievement between different family background groups. We find evidence that equalization of spending leads to a narrowing of test score outcomes across family background groups. (p. 49)

Deke (2003)

Using panel models that, if biased, are likely biased downward, I have a conservative estimate of the impact of a 20% increase in spending on the probability of going on to postsecondary education. The regression results show that such a spending increase raises that probability by approximately 5% (p. 275).

Papke (2001)

Focusing on pass rates for fourth-grade and seventh grade math tests (the most complete and consistent data available for Michigan), I find that increases in spending have nontrivial, statistically significant effects on math test pass rates, and the effects are largest for schools with initially poor performance. (Papke, 2001, p. 821.)

Downes (2004) on VT

All of the evidence cited in this paper supports the conclusion that Act 60 has dramatically reduced dispersion in education spending and has done this by weakening the link between spending and property wealth. Further, the regressions presented in this paper offer some evidence that student performance has become more equal in the post–Act 60 period. And no results support the conclusion that Act 60 has contributed to increased dispersion in performance. (p. 312)

Downes, Zabel & Ansel (2009) on Mass

The achievement gap notwithstanding, this research provides new evidence that the state’s investment has had a clear and significant impact. Specifically, some of the research findings show how education reform has been successful in raising the achievement of students in the previously low-spending districts. Quite simply, this comprehensive analysis documents that without Ed Reform the achievement gap would be larger than it is today. (p. 5)

Guryan (2003) on Mass

Using state aid formulas as instruments, I find that increases in per-pupil spending led to significant increases in math, reading, science, and social studies test scores for 4th- and 8th-grade students. The magnitudes imply a $1,000 increase in per pupil spending leads to about a third to a half of a standard-deviation increase in average test scores. It is noted that the state aid driving the estimates is targeted to under-funded school districts, which may have atypical returns to additional expenditures. (p. 1)

Goertz & Weiss (2009) on NJ

State Assessments: In 1999 the gap between the Abbott districts and all other districts in the state was over 30 points. By 2007 the gap was down to 19 points, a reduction of 11 points or 0.39 standard deviation units. The gap between the Abbott districts and the high-wealth districts fell from 35 to 22 points. Meanwhile performance in the low-, middle-, and high-wealth districts essentially remained parallel during this eight-year period (Figure 3, p. 23).

I could go on. But that’s a fair share of evidence right there.

And what does Biddle provide as counter evidence to this – apparent lack of evidence I summarize above (I’ve sent the article link to Biddle on more than one occasion, but he apparently doesn’t read this kind of academic stuff)?

Biddle counters with a link to this graph – a true gem (I’ve added some annotation, not in his original)!

Yes, Biddle’s entire counter to the body of research he has not and will not read, is to use this graph of “promoting power” by student race group for Jersey City, NJ in 2004 and 2009. Note that the infusion of additional funds in NJ occurred mainly from 1998 to 2003, leveling off thereafter. But that’s a tangential point (not really).  So, Biddle’s absolute verification that more money doesn’t matter is to simply assert without verification that Jersey City got a whole lot more money and then to use this graph to argue that nothing improved!

First of all, that analysis wouldn’t pass muster in as a master’s degree level assignment (I teach a class on this stuff at that level), no less major research conclusions. From a graphing standpoint, I often criticize my students’ work for what I refer to as gratuitous use of 3d – especially where the use of 3d bars actually obscures the comparisons by making it hard to see where they align on the axis.

But, the really funny if not warped part of this graph is that there appear to be significant gains for black males between 2004 and 2009, but those gains are obscured by hiding the 2009 black male score behind the 2004 black female score.

Note that the graph also contains no information regarding the actual shares of the student population that fall into each group? Not very useful. Pretty damn amateur. Certainly fails to make any particular point, and certainly doesn’t refute the various citations above – all of which employ more rigorous analytic methods, apply to more than a single district, and most of which appear in rigorous peer reviewed journals.

References:

Card, D., and Payne, A. A. (2002). School Finance Reform, the Distribution of School Spending, and the Distribution of Student Test Scores. Journal of Public Economics, 83(1), 49-82.

Deke, J. (2003). A study of the impact of public school spending on postsecondary educational attainment using statewide school district refinancing in Kansas, Economics of Education Review, 22(3), 275-284.

Downes, T. A. (2004). School Finance Reform and School Quality: Lessons from Vermont. In Yinger, J. (ed), Helping Children Left Behind: State Aid and the Pursuit of Educational Equity. Cambridge, MA: MIT Press.

Downes, T. A., Zabel, J., and Ansel, D. (2009). Incomplete Grade: Massachusetts Education Reform at 15. Boston, MA. MassINC.

Goertz, M., and Weiss, M. (2009). Assessing Success in School Finance Litigation: The Case of New Jersey. New York City: The Campaign for Educational Equity, Teachers College, Columbia University.

Guryan, J. (2003). Does Money Matter? Estimates from Education Finance Reform in Massachusetts. Working Paper No. 8269. Cambridge, MA: National Bureau of Economic Research.

Papke, L. (2005). The effects of spending on test pass rates: evidence from Michigan. Journal of Public Economics, 89(5-6). 821-839.

Roy, J. (2003). Impact of School Finance Reform on Resource Equalization and Academic Performance: Evidence from Michigan. Princeton University, Education Research Section Working Paper No. 8. Retrieved October 23, 2009 from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=630121(Forthcoming in Education Finance and Policy.)

I don’t spend much if any time talking about my personal decisions and preferences on this blog. It’s mostly about data and policy.  There’s been much talk lately about whether a Governor’s or President’s choice to send their children to elite private schools, or where Bill Gates, Mark Zuckerberg or prominent “ed reformers” attended school are at all relevant to the current policy conversation around  “reforming” public schools.  When those choices have been questioned publicly, they’ve often been met with the backlash that those are personal choices of no relevance to the current policy debate – just dirty personal attacks about personal, rational choices.

I have no problem with these personal choices. But, these personal choices may, in fact be relevant to the current policy debate.  I do keep in mind my own personal choices and preferences as I evaluate what I believe to be good policy for the children of others. And, I try to keep in mind what I know from my background in research and policy when I make my personal choices.   Like these prominent politicos and pundits, I too choose private independent schools – relatively expensive ones – for my children, and I have my reasons for doing so. As I’ve noted on my blog on a number of occasions, I taught at an exceptional private independent school in New York City, and have relatives and friends who continue to be involved in (and with) high quality private independent schools as teachers, administrators and parents. I did not, however, attend private school. I attended public school in Vermont, followed by private college (Lafayette College).

Why do I personally prefer private independent schools, which often come with a high price tag?  Here are a few reasons:

1. The responsiveness that comes from a close-knit small community with not only small class sizes but also lower total student load for teachers (at middle and secondary level in particular)
2. The depth and breadth of curricular offerings ranging from Latin in the middle school, to a diverse array of social science, advanced science and math courses at the high school level and a plethora of opportunities in the arts and athletics.
3. The lack of emphasis on standardized testing – bubble tests and overemphasis on tested curricular areas and state standards.

Yes, I do consider it important that these schools are not test-whipped, specifically that they are not obsessed with basic reading and math bubble tests alone, or even more disturbing, tests of science and social studies content where the balance (or absence) of content is a function of partisan preferences of ill-informed politically motivated elected officials (e.g. Kansas science standards, or Texas social studies/history standards – thankfully, I’m not in KS anymore).

These days, I consider it especially important that my children not be in a school where teachers have to hang their hopes of achieving a living wage (or getting a bonus to afford cosmetic surgery as in “Bad Teacher”[hope to see that one soon!]) on whether or not my child gains X+Y points on those reading or math tests. In fact, these may now be my main reasons for opting out.

So yes, you might try to call me a hypocrite for preferring private schools for my own children while apparently being such a staunch defender and supporter of the public system (including voting yes on local district budgets, even when encouraged to vote no by public officials). But that would be a dreadful oversimplification and misrepresentation of my position.

I have worked in both public and private schools – one good and one bad of each – over a 10+ year period prior to my life in higher education.  I’ve studied and compared public and private schools in various locations and of various types for over 15 years and published numerous articles, papers and reports. What I’ve learned most from these studies is that private and/or less regulated markets are simply more varied than public and/or more regulated markets. Neither better nor worse on average – simply more varied.

Top notch private schools spend much more, and many financially strapped, relatively average to very low academic quality private schools do spend much less. Much more and much less than one another, and much more and much less than nearby public schools.  It is a massive bait and switch to suggest – look how great Sidwell Friends (DC),  Dalton or Fieldston (NYC) are compared to public schools, and look how much the average Catholic parish elementary school spends compared to the urban public district?  Of course, it’s never as obviously phrased as a bait and switch – suggesting that you can get a Sidwell or Dalton education at an urban Catholic elementary school price.  You can’t! Yes, the average Catholic parish elementary school likely spends less per pupil than the public district. But that school is no Sidwell, Dalton or Fieldston, which spend closer to and in excess of double the public schools in their area.

Private schools do not, as many assume, spend only about half what public schools do. This is urban legend, drawn from dated analyses that were misrepresented to begin with (over 10 years ago).  My extensive report on private school supply and spending covers these issues quite extensively.

To reiterate a major finding from my study of private school costs, private independent schools of the type I am talking about here (members of NAIS or NIPSA), spend ON AVERAGE, 1.96 times the average per pupil amount of public schools in the same labor market! (and have half the pupil to teacher ratio)

I am quite convinced that many of the policy makers who choose elite private schools for their own and advocate for scaling back the public system, really don’t understand the difference. They really don’t know that their private schools outspend nearby traditional public schools – by a lot – despite serving more advantaged student populations. Heck, I’ve talked to administrators in private independent schools who feel that their own budgets are tight (legitimately so), and assume that the public schools around them spend much more per child. But they are simply naïve in this regard (while wise in many other ways). No intent to harm. They’ve simply bought into the misguided rhetoric that private schools spend less and get more and they’ve never double-checked the facts. But even a few minutes of pondering their own budgets and looking up local public school spending brings them around. (Part of this perception is likely driven by differences in access to funding for capital projects, where heads of private schools recognize the heavy lifting of major fundraising campaigns, and envy the taxing authority of public school districts for these purposes).

In my view, the hypocrisy lies in what those who choose elite private schools for their own argue are the best solutions for public education for the children of others.  If the preferences are the same, there is no hypocrisy. The problem is when those preferences are vastly different – completely at odds – as they tend to be in the present “ed reform” and “new normal” debate.

It is hypocritical for pundits who favor for their own children, expensive schooling with diverse curriculum, small class size and little standardized testing (freeing teachers to be professionals), to argue for less money, class size increases and increased standardized testing (and teacher evaluation based on those tests) when it comes to other peoples’ children.

Yes, I too personally favor expensive private schooling for the reasons I’ve indicated above. And yes, my private school significantly outspends both the elite suburban public school district where I live and New Jersey’s reasonably well funded urban districts (compared to other states, see: http://www.schoolfundingfairness.org).   The way I see it, I would not just be a hypocrite, but a complete a-hole if I used my pulpit (what little pulpit I have) as a school finance expert to argue that we should be spending less on others, advocating different policies for others than I desire for myself.  But it’s precisely because I spend my day buried in data on school finance and education policy that I see this glaring hypocrisy.

The difference is that I believe that other children – those whose parents are not able to make this expensive choice – should have access to well-funded schools that also provide small class sizes, diverse curriculum, and for that matter, place less emphasis on standardized tests, and treat teachers as responsible, knowledgeable professionals (not script reading stand-ins and test proctors).

To clarify, this is not a criticism of individuals with personal preferences for high quality education for their own children who are otherwise unconcerned with (or oblivious to) the broader public policy questions pertaining to the children of others. Rather, this is a direct criticism of those public officials and vocal “ed reformers” who prefer high quality, well funded education for their own and then loudly and publicly advocate for a very different quality (and type) of education for the children of others.

If we could actually close the gap between public school resources and resource levels of elite private schools, there might be less demand for those elite private schools (though some would indeed respond with an arms race to outpace public schools).  Presently, however, elite private schools stand to benefit significantly from the “ed reform” and “new normal” movement which will likely make more public schools – including those in more affluent ‘burbs – even less desirable for parents currently on the fence.

So, here’s my challenge to all those policymakers who also prefer elite private independent schools for their children.  I urge you to make a list of all of the reasons why you chose a private independent school. Notably, many if not most parents list class size as a major factor (and most schools advertise class size as a major benefit).  Make a list of the specific attributes of your private school including:

1. Average class size
2. Teacher education levels
3. Numbers and types of elective and advanced course offerings
4. Numbers and types of extracurricular activities
5. Whether they pay more experienced teachers more than less experienced ones (or more for teachers holding advanced degrees?)
6. Whether they emphasize student test scores when evaluating or compensating teachers?

and whatever else you might think of. (here are a few sample NJ private schools)

Get a copy of the school’s IRS 990 tax filing from the school (or from:  http://foundationcenter.org/, or http://www.guidestar.org) to find out roughly how much your school spends each year, and divide that by the number of total enrolled pupils.

Then, gather similar information on surrounding public schools. Make your own comparisons. And after you’ve done so, let me know if you’re still comfortable making bold public proclamations that we need to reign in the absurd spending of public schools, increase class sizes and slash all of those frivolous extracurricular programs for other people’s children, but certainly not our own!

Video Extra:

And a Song:

I’ve reached a point after these past few years where I feel that I’ve spent way too much time  critiquing poorly constructed arguments and shoddy analyses that seem to be playing far too large a role in influencing state and federal (especially federal) education policy. I find this frustrating not because I wish that my own work got more recognition. I actually think my own work gets too much recognition as well, simply because I’ve become more “media savvy” than some of my peers in recent years.

I find it frustrating because there are numerous exceptional scholars doing exceptional work in school finance and the economics of education whose entire body of rigorous disciplined research seems drowned out by a few prolific hacks with connections in the current policy debate.It may come as a surprise to readers of popular media, but individuals like Mike Petrilli, Eric Osberg, Rick Hess (all listed on the USDOE resource web site) or Bryan Hassel wouldn’t generally be considered credible scholars in school finance or economics of education. I’d perhaps have less concern – and be able to blow this off – if many of the assertions being made by these individuals – and others – weren’t so often completely unsupported by reasonable analysis and if those assertions didn’t lead to potentially dangerous and damaging policies.

This post is specifically about the body of methodologically flimsy research produced in recent years by Marguerite Roza, previously of the Center on Reinventing Public Education and currently an advisor to the Gates Foundation.

Why this post now? I’ve simply lost my patience.

This post is in part a response to the recent unveiling of the U.S. Dept. of Education web site on improving educational productivity http://www.ed.gov/oii-news/resources-framing-educational-productivity. Amazingly, this site lists primarily non-peer reviewed, shoddy work by Marguerite Roza and colleagues and bypasses entirely more serious research on educational productivity or methods for evaluating it.  The quality of some of the examples on this site is particularly abysmal. Yet it is presented as “the work of leading thinkers in the field.” (interesting that “thinkers” is used in place of “researchers.”) Among the worst examples, this site lists as a credible resource the Center for American Progress Return on Investment analysis. (by Ulrich Boser, a great writer on the topic of art theft, but in this case, a bit out of field).

I don’t mind so much that this stuff exists. But it certainly doesn’t belong in a serious policy conversation, nor does it represent “the work of leading thinkers in the field.”

Let’s start with a few common attributes of the worst-of-the-worst types of policy research floating around out there and warping and misguiding the education policy debates in general and school finance debates in particular. For lack of a better term, let’s just call it “hack research.”

Perhaps most importantly, hack research fails to recognize all of the credible work that’s already been done on a topic, typically because the research hack who produced it lacks entirely the discipline to bother to understand that body of work and how to build on it in order to come to new, credible findings and conclusions.

Further, hack research displays little regard for the connection between rigorous analysis and conclusions that may be drawn from it. This stems in part from the lack of discipline to actually conduct rigorous analyses.

Particularly effective hacks will not just ignore the body of existing scholarship but will do so belligerently, proclaiming that no good work has ever been done, no credible methods of analysis do exist, and therefore the time is right for their own creative and new perspective! The hack research method substitute is usually some seemingly intuitive, completely shallow, poorly conceived back-of-the-napkin approach. In other words, the hack research motto is that we must think outside the box, because it’s just too much work to open and unpack that box!

Many of us start as hacks, but eventually grow out of it as we realize that there’s a lot of great stuff out there to read and exceptional scholars from which to learn. And, some non-hacky researchers will occasionally hack. Hack happens. It’s only really problematic when it’s a persistent pattern of hackyness or even gets worse over time.

The most dangerous hacks use their shtick to influence policy with catchy anecdotes, convincing policymakers and major players that they need look no further (at real research, for example) than their own hacky “research.”  And the most effective hacks can spin findings that never were into pure urban legend – well-accepted myths turned realities – with serious policy implications!

Let’s take a look at a number of mythical findings from shoddy research produced by Marguerite Roza in recent years, including a few sources cited on the USDOE resources page.

Myth #1: States have largely solved between district funding disparities and within district disparities are the remaining problem of the day.

Sources of the myth: See references in Baker/Welner article (cited below)

A now common myth in school finance reiterated in numerous sources produced by the Education Trust, Center for American Progress and other DC think tanks and pundits is that states have largely resolved disparities in funding between districts and that persistent disparities are primarily within districts, between schools – a function of illogical district allocation formulas.

In a recent article Kevin Welner and I tackle this argument and dig deeply into the sources behind this argument, which invariably find their way back to Marguerite Roza, then of the Center on Inventing Research Findings – excuse me – Center for Reinventing Public Education (CRPE).

Kevin and I conclude in our article:

 Two interlocking claims are being increasingly made around school finance: that states have largely met their obligations to resolve disparities between local public school districts and that the bulk of remaining disparities are those that persist within school districts. These local decisions are described as irrational and unfair school district practices in the allocation of resources between individual schools. In this article, we accept the basic contention of within-district inequities. But we offer a critique of the empirical basis for the claims that within-district gaps are the dominant form of persistent disparities in school finance, finding instead that claims to this effect are largely based on one or a handful of deeply flawed analyses.

Kevin Welner and I dissect in detail the problematic, “non-traditional” methods Roza and colleagues use for conducting their analyses (ignoring real methods used by real researchers in real publications), but perhaps more interesting are those cases where a narrow, measured finding pertaining to one specific estimate in one specific context becomes a national trend, a dominant reality soon thereafter. Op-Ed columns by Roza on the topic of within versus between district funding disparities include particularly egregious examples. Kevin Welner and I explain:

Following a state high court decision in New York mandating increased funding to New York City schools, Roza and Hill (2005) opined: “So, the real problem is not that New York City spends some $4,000 less per pupil than Westchester County, but that some schools in New York [City] spend $10,000 more per pupil than others in the same city.” That is, the state has fixed its end of the system enough.

This statement by Roza and Hill is even more problematic when one dissects it more carefully. What they are saying is that the average of per pupil spending in suburban districts is only $4,000 greater than spending per pupil in New York City but that the difference between maximum and minimum spending across schools in New York City is about $10,000 per pupil. Note the rather misleading apples-and-oranges issue. They are comparing the average in one case to the extremes in another.

In fact, among downstate suburban[1] New York State districts, the range of between-district differences in 2005 was an astounding $50,000 per pupil (between the small, wealthy Bridgehampton district at $69,772 and Franklin Square at $13,979). In that same year, New York City as a district spent $16,616 per pupil, while nine downstate suburban districts spent more than $26,616 (that is, more than $10,000 beyond the average for New York City). Pocantico Hills and Greenburgh, both in Westchester County (the comparison County used by Roza and Hill), spent over $30,000 per pupil in 2005.[2] These numbers dwarf even the purported $10,000 range within New York City (a range that we agree is presumptively problematic); our conclusion based on this cursory analysis is that the bigger problem likely remains the between-district disparity in funding.

For the full take down, see:

Baker, B. D., & Welner, K. G. (2010). “Premature celebrations: The persistence of interdistrict funding disparities” Educational Policy Analysis Archives, 18(9). Retrieved [date] from http://epaa.asu.edu/ojs/article/view/718

Myth #2: America’s public school system suffers from something called Baumol’s disease, therefore the only solutions must be found outside of public education

Source: Curing Baumol’s Disease: In Search of Productivity Gains in K–12 Schooling Paul Hill, Marguerite Roza

While I don’t think this one really ever caught on, it’s so absurd that it must be addressed. Further, it’s actually cited on the USDOE resources in educational productivity page despite the fact that it offers no useful guidance whatsoever on the topic.

The objective of this policy brief by Paul Hill and Marguerite Roza of CRPE is to explain how American public education suffers from Baumol’s disease, or “the tendency of labor-intensive organizations to become more expensive over time but not any more productive.” Hill and Roza’s attempt at empirical validation that American public education suffers from Baumol’s disease is presented in two oversimplified figures, a graph showing increased number of staff who are not core teachers (Figure 1) and a graph showing that student test scores on the National Assessment of Educational Progress have remained flat over time (Figure 2).  The latter claim that we’ve seen no improvement in NAEP scores over time is contested.[1] And the former claim, when aggregated nationally is not particularly meaningful. The authors provide no empirically rigorous link between the two.

Rather, the casual reader is simply to assume that public schools have added a lot of non-teaching staff and have, on average, nationally seen no yield for that increase costs. Hill and Roza posit:

“While these indicators clearly point to increased costs for education, efforts to quantify productivity changes have been hampered by measurement challenges on the outputs side of the equation. By most accounts, key indicators of outcomes have not shown comparable gains. A thirty-year look at NAEP performance for seventeen year-olds, for instance, suggests that test scores have changed very little.” (p. 3)

While this may, in fact, not be entirely untrue, the authors provide no rigorous validation that “Baumol’s Disease” is a persistent problem of American public schools.

However, without a disease with a catchy name, there would be little reason for their proposed cure. But the proposed cure is no more thoroughly vetted or precisely articulated than the disease.  A central assumption in the Baumol’s disease policy brief is that American public education systems take on one single form, as represented by national averages in the TWO graphs provided, that there is little or no variation within the public education system in terms of resource use or outcomes achieved (e.g. that it all suffers Baumol’s disease), and that therefore the only possible cures are those that come from outside the public education system or at its fringes. That is, that we have nothing to learn from variation within the public education system itself, because there is no such variation. Instead, for example, the authors suggest a closer look at “home schooling, distance learning systems, foreign language learning, franchise tutoring programs, summer content camps, parent-paid instructional programs (music, swimming lessons, etc.), armed services training, industry training/development, apprentice programs, education systems abroad.” (p. 10)

Numerous more credible researchers have spent a great deal of time learning from the heterogeneity of how schools, school districts, and charter schools operate, as well as across states, including studying the relative efficiency of schools that either operate differently or change how they operate. The assumption that the only solutions must come from outside the system is patently absurd, when the “system” consists of 51 policy contexts, over 100,000 schools, 5,000 charter schools and about 15,000 public districts. And it’s just lazy, hack thinking.

While one might gain insights from other labor-intensive industries, or education at the fringes of the current public system, it would be foolish to ignore the extent of variation within the current American public education system, and across traditional public, magnet, charter and private schooling. Arguably, the authors present the view that there is little or nothing to learn from the current system specifically in order to avoid the need for conducting rigorous analysis of it. Further, while such policy briefs may be generously considered as useful conversation starters, we take serious issue with the U.S. Department of Education’s identification of sources of this type, which are purely speculative, and severely lacking in intellectual or empirical rigor, as “Key Readings on Educational Productivity.”

Myth #3: Poor, failing school districts have plenty of money but are squandering too much on Cheerleading and Ceramics when they need to be spending on basics!

Original Source of (the anecdote behind the) myth: “Now is a Great Time to Consider the Per Unit Cost of Everything in Education.”

As I explain in my recent conference paper:

Authors including Marguerite Roza and colleagues of the Center for Reinventing Public Education encourage public outrage that any school district not presently meeting state outcome standards would dare to allocate resources to courses like ceramics or activities like cheerleading. To support their argument, the authors provide anecdotes of per pupil expense on cheerleading being far greater than per pupil expense on core academic subjects like math or English.

• Imagine a high school that spends $328 per student for math courses and $1,348 per cheerleader for cheerleading activities. Or a school where the average per student cost of offering ceramics was $1,608; cosmetology, $1,997; and such core subjects as science, $739.1

These shocking anecdotes, however, are unhelpful for truly understanding resource allocation differences and reallocation options, and are an unfortunate and unnecessary distraction. For example, the major reason why cheerleading or ceramics expenses per pupil are seemingly high is the relatively small class sizes, compared to those in English or Math. In total, the funds allocated to either cheerleading of ceramics are unlikely to have much if any effect if redistributed to reading or math.

Now, this myth is a rather strange one, because the source from which it comes, which is authored by Marguerite alone, really isn’t totally unreasonable. It’s not useful in any way shape or form, but it’s not unreasonable either. This wacky anecdote about cheerleading and ceramics spending comes from a piece in which Roza is trying to explain the importance of comparing unit costs of providing specific programs/opportunities. This is a rather “no duh” idea, and the working paper and eventual book chapter is built on uninteresting anecdotes, at best. The original point of the paper is that if administrators look at the per unit cost of everything, they might find some things that stand out, and some things that might be reasonably reorganized to be offered at a lower unit cost (for example, the cost of cheerleading was reduced by moving it from a class period drawing on salaried time, to an after school activity, paid by small stipend).

But, the spin from this piece has been that this is all that low performing, poor urban districts need to do. They’ve all got enough. They themselves are responsible for the most persistent inequities – not the states. And they are the ones wasting way too much on things like cheerleading and ceramics. Given that this spin has had far more traction than the more reasonable paper behind it, one might assert that this is precisely what Roza intended.

In my paper, I conclude:

Rather, the emergent story from the data in both states was the contrast between high spending, high outcome districts, and low spending low outcome districts and their respective high schools. On average, high spending, high outcome districts were as one might expect much lower in student poverty concentration and low spending, low outcome districts much higher in poverty. That is, after applying thorough cost adjustment including adjustments for differences in student needs. Interestingly, the most striking differences between these groups of districts were not in the availability of assigned teachers or courses in the arts, but rather in the distribution of advanced versus basic course offerings in curricular areas such as math and physical science.

Note that to begin with, low spending, low outcome schools had fewer teacher main assignments and fewer course assignments per pupil. As such, they were, from the outset, more constrained in their allocation options. Further, there is at least some evidence that when evaluating district wide resource allocation, low resource, low outcome districts see greater necessity or feel greater pressure to allocate a larger overall share of resources to elementary classrooms (based on Illinois findings).

More thorough analyses of this issue see:

Baker, B.D. (2011) Cheerleading, Ceramics and the Non-Productive Use of Educational Resources in High Need Districts: Really? Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA 2011

Myth #4: High schools in Washington State pay math and science teachers less than other teacher despite public interest and state policies which encourage paying them more

Source: Washington State High Schools Pay Less for Math and Science Teachers than for Teachers in Other Subjects Jim Simpkins, Marguerite Roza, Cristina Sepe

This is one that suffers from both major issues identified at the beginning of this rant. First, the disconnect between the “study” and the press release:

The Press Release
http://www.crpe.org/cs/crpe/view/news/111

The analysis finds that in twenty-five of the thirty largest districts, math and science teachers had fewer years of teaching experience due to higher turnover—an indication that labor market forces do indeed vary with subject matter expertise. The subject-neutral salary schedule works to ignore these differences.

The Study
http://www.crpe.org/cs/crpe/download/csr_files/rr_crpe_STEM_Aug10.pdf

That said, the lower teacher experience levels are indicative of greater turnover among the math and science teaching ranks, lending support to the hypothesis that math and science teachers may have access to more compelling non-teaching opportunities than do their peers. (p. 5)

That is, the conclusions of the study itself and the press release are, well, not consistent. But this pattern of behavior is entirely consistent for Roza and CRPE.

In a previous post I address just how ridiculous the methods in this analysis are, in which she compares STEM teacher salaries with non-STEM teacher salaries without any controls for other factors that affect salaries (on the argument that salaries shouldn’t be based on those things – experience and degree level – anyway).

All that Roza really found in this paper was that STEM teachers tend to be younger and as a result have lower average salaries than non-STEM teachers. From that, she spun the argument that because STEM teachers don’t earn more than other teachers, but STEM fields are more competitive, STEM teachers must be leaving teaching at a higher rate, leading to a less experienced pool and lower average salaries (a vicious cycle indeed! But one that’s never validated by the ridiculous analysis).

In my post, I actually evaluate several years of teacher level data on all teachers in Washington State, finding most of her conclusions to be flat out wrong. Here’s the figure on mean STEM and non-STEM teacher salaries over time: https://schoolfinance101.com/wp-content/uploads/2010/08/slide42.jpg

I also point out that credible researchers like Lori Taylor of Texas A&M have actually done better analyses of Washington teacher wages and addressed variations in labor market competitiveness by field:

Report on Taylor Study:

http://www.wsipp.wa.gov/rptfiles/08-12-2201.pdf

Taylor Study:

http://www.leg.wa.gov/JointCommittees/BEF/Documents/Mtg11-10_11-08/WAWagesDraftRpt.pdf

Somehow, not surprisingly, Roza was unaware of either this better research or the more credible methods used in this research.

For the full take down, see: https://schoolfinance101.wordpress.com/2010/08/20/new-from-the-center-on-inventing-research-findings/

Myth #5: With our handy-dandy basket of reformy fixes, we can cut significant funding from American public schools and dramatically increase productivity!

Source: Petrilli and Roza

Stretching the School Dollar (Brief)

http://www.edexcellence.net/publications-issues/publications/stretching-the-school-dollar-policy-brief.html

In their policy brief on Stretching the School Dollar, Mike Petrilli of Thomas B. Fordham Institute and Marguerite Roza of the Gates Foundation provide a lengthy laundry list of strategies by which school districts and states might arguably increase their productivity at lower expense, or “stretch the dollar” so to speak.  This policy brief is an extension of the Frederick Hess (American Enterprise Institute) and Eric Osberg (Fordham Institute) edited book by the same title.  We highlight this source because of repeated specific references to this source in Secretary Duncan’s “New Normal” speeches during the Fall of 2010.[2]

Because this policy brief and book specifically list strategies that are intended to improve productivity at comparable or lower expense, it would be particularly relevant for the book or brief to either provide directly or summarize from other sources, rigorous cost-effectiveness analysis of these options, or relative efficiency comparisons of schools and districts employing these options.   But that is apparently asking way too much of Roza or Petrilli. I’ll cut Mike some slack here, because he isn’t the one actually presenting himself as a school finance expert/scholar. That’s Roza’s role in this partnership, therefore the burden falls on her.  But after reading enough work by Roza and colleagues, I’m no-longer convinced that she is even aware that there is a body of research out there on Cost-effectiveness analysis or relative efficiency (more on this later). I certainly encourage her to go buy a copy of Hank Levin and Patrick McEwan’s book, not so subtly titled Cost-Effectiveness Analysis: Methods and Applications. It’s a relatively easy, non-academic read.

I’ll offer a primer on these methods and their application to these questions in a future post. There’s no need to beat a dead horse on this topic. I’ve taken down Roza and Petrilli’s reformy gift basket in two previous posts to which you can refer.

For the full take down, see:

Part 1 – Stretching the Truth, Not Dollars: School Finance in a Can: Unproven and Unsubstantiated Dollar-Stretching State Policies

Part 2 – Stretching the Truth, Not Dollars: Considering the Application of Cost-Benefit Analysis to Teacher Layoff Alternatives