Category: Race to the Top

Thought for the day…

Many will consider this blasphemy, but, I’ve been pondering lately:

If our best public and private schools are pretty good (perhaps even better than Finland?),

And, if the majority (not all, but most) of our best AND our worst public (and private) schools use salary schedules which base teacher compensation primarily on degrees/credits/credentials obtained and years of experience or service…

Can we really attribute the failures of our worst schools  to these features of teacher compensation?

Yeah… there might be a better (more efficient and effective way), but is this really the main problem?

Teacher “effectiveness” ratings Freedom of Information Requests

Andy Rotherham over at Eduwonk posted an Irony Alert yesterday as many media outlets poised themselves to start “outing” ineffective teachers by posting publicly those teacher’s value-added effectiveness scores. Rotherham argued:

In light of this blow up about value-added in New York City, in a lot of places if the teachers unions would actually get serious about actually using value-add data as part of teacher evaluations it could be shielded from “Freedom of Information”requests that identify teachers, just as many aspects of personnel evaluations are.   They’re caught in their own mousetrap here.  My take on the larger issue from a few weeks ago and LA.

I thought…. hmmm… really? That doesn’t seem right. Is this just a clever argument intended to dupe teachers into getting those scores into their evaluations on some false assumption that the information would then be protected? Are these issues even transferable from state to state? Is the raw data used for generating the teacher effectiveness ratings actually considered part of the personnel file? I’m somewhat of an amateur on this school law stuff, but have enough background to start asking these questions when such arguments are tossed out there. So I did. I asked a handful of legal scholars in education policy, each of whom deals regularly with legal questions over personnel records under state law and with student record information.

Justin Bathon over at Ed Jurist has now posted his conversation starter for the legal community.

This is good stuff, and the very kind of conversation we should be having when such questions are raised. Ask the experts. Much of the argument hinges on when the raw data is translated into a measure that actually becomes part of the personnel file (at least with regard to the “shield” issue posed by Rotherham). Here’s Justin Bathon’s summary:

Anyway, summarizing, I think the raw data is generally going to be made publicly open following FOIA requests. I think New York City is currently correct in their assessment that no exemption exists under New York’s Freedom of Information Law. However, this is just my analysis after considering this issue for a single day and I want to caution against over reliance on my initial assumptions. A thorough analysis needs to be conducted of all 50 state policies, interpreting regulations, attorney general opinions, and previous case-law. Further, data experts such as Bruce must assist the analysis with a complete understanding of each state’s dataset and the possible links to both teachers and their evaluations within the datasets. Thus, there is still a lot of work left to be done.

This is a legal frontier (another one of those enabled by technology) that most legislatures would not have contemplated as possible in enacting their open records laws. Thus, it is a great topic for us to debate further to inform future policy actions on open records personnel evaluation exemptions.

Please, read the rest of his well thought out, albeit preliminary post.

Here are my follow-up comments (cross-posted at edjurist) on data/data structures and their link to teacher evaluations:

Here are some data scenarios:

A. The district has individual student test score data that are linkable to individual teachers but the district doesn’t use those data to generate any estimates of individual teacher “effectiveness,” has not adopted any statistical method for doing so and therefore does not include any such estimates as part of personnel records. The individual students’ identity can be masked but with matched ID over time and specific characteristics attached (race, low-income status)
B. The district has individual student test score data that are linkable to individual teachers just as above, and the district does a) have an adopted statistical model/method for generating teacher value added “effectiveness” scores, but uses those estimates only for district level evaluation/analysis and not for individual teacher evaluation.
C. The district has individual student test score data that are linkable to individual teachers as above, and a) the district has an adopted statistical method/model for generating teacher value-added “effectiveness” scores and has negotiated a contractual agreement with teachers (or is operating under a state policy framework) which requires inclusion of the “effectiveness” scores in the formal evaluation of the teacher.

Under option C above, sufficient technical documentation should be available such that “effectiveness” estimates could be checked/replicated/audited by an outside source.  That is, while there should be materials that provide sufficiently understandable explanations such that teachers can understand their own evaluations and extent to which their “effectiveness” ratings are, or are not under their own control, there should also be a detailed explanation of the exact variables used in the model, the scaling of those variables, etc. and the specification of the regression equation that is used to estimate teacher effects. There should be sufficient detail to replicate district generated teacher effectiveness scores.

That aside, a few different scenarios arise.

1. The LA Times scenario, as I understand it, falls under the first conditions above. The data existed in raw form. The district was not using those data for “effectiveness” rating. The LAT got the data and handed them over to Richard Buddin of RAND. Buddin then estimated the most reasonable regression equation he could with the available data and, for that matter, produced a sufficiently detailed technical report – such that anyone accessing the same data could replicate his findings. I suspect that individual student names were masked, but the students were clearly matched to identifiable teachers, and student data included specific identifiers of race, poverty, etc. and participation in programs such as gifted programs (indicator on child that he/she labeled as gifted). Not sure what, if any, issues are raised by detailed descriptive information on child level data. In this case, the data requested by LAT and handed over to Buddin were not linked to teacher evaluation by the district itself, in any way, as I understand it.

2. As I understood the recent NYC media flap, the city itself was looking to report/release the value-added ratings and the city itself also intends to use those “value added” ratings for personnel evaluation. It sounded to me that Charleston, SC was proposing roughly the same. Each teacher would have a nifty little report card showing his her “relative” effectiveness rating compared to other teachers. This effectiveness rating is essentially a “category” labeling a teacher as “better than average” or “worse than average.” These categories are derived from more specific “estimates” which come from a statistical model, which generates a coefficient for each teacher’s “effect” on the students who have passed through that teacher’s classroom (these coefficients having substantial uncertainty and embedded bias which I have discussed previously… but that’s not the point here). So, the effectiveness profile of the teacher is an aggregation of these “effects” into larger categories – but is nonetheless directly drawn from these effect estimates generate by the district itself for teacher evaluation purposes (even if subcontracted by the district to a statistician). I would expect that the specific estimate and the profile aggregation would be part of the teacher’s personnel record.

So, now that the city’s official release of effectiveness profiles is on hold, what if a local newspaper requested a) the raw student data linkable to teachers, with student names masked but with sufficient demographic detail on each student and with identifiable information on teachers, and b) the detailed technical documentation on the statistical model and specific variables used in that model?  The newspaper could then contract a competent statistician to generate his/her own estimates of teacher effectiveness using the same data used by the district and the same method. These would not be “official” effectiveness estimates, nor could the media outlet claim them to be. But they would be a best attempt at a replication. Heck, it might be more fun if they used a slightly different model, because the ratings might end up substantially different from the district’s own estimates.  Replicating or not, the districts own methods, and producing roughly the same or very different ratings for teachers, these estimates would still not be the official ones. Given the noise and variation in such estimates at the teacher level, it might actually be pretty hard to get estimates that correlate substantially with the district’s own estimates – and one would never know, because the district official effectiveness estimates for teachers would still be private.

I would assume under these circumstances, partly because the “official” personnel file estimates would remain unknown, and because it’s highly probable that the independent estimates produced by the media outlet – even if trying to replicate district estimates – might vary wildly from the district estimates – that the media outlet could get the data, estimate the model and report their results – their unofficial results. On the one hand, the media outlet could rely on the uncertainty of the estimates to justify that what they produce should not be considered “official” estimates. And on the other hand… in bold print in the paper… they could argue as the LA Times Jasons have … that these estimates are good and reliable estimates of actual teacher effectiveness!

The conversation continues over at EdJurist: http://www.edjurist.com/blog/value-added-evaluation-data-and-foia-state-versions-that-is.html?lastPage=true#comment10260419

Interesting follow-up point from Scott Bauries over at Ed Juris:

Thus, from the legal perspective, I am left with one question: if the data and conclusions are being used as reflected in option “c,” but the media only gets the conclusions and not the raw data, then does the law allow a teacher to protect his or her reputation from unfair damage due to the publishing of a conclusion based on a noisy equation?

This is a very complicated question, involving both defamation law and the First Amendment. For example, is a public school teacher a “public official” or “public figure” for First Amendment purposes, such that the standard for proving defamation per se is increased? If so, then is the relevant statistical analysis illustrating the noisy nature of the conclusion enough to show falsehood for the purposes of a defamation claim? I think probably not in both instances, but I don’t think this precise issue has ever come up.

When reformy ideologies clash…

(note: lots of ideas here that I wanted to start writing about… but not yet well organized or articulated. It will come, with time, I hope.)

Summary of Reformy Ideology

Bluntly stated, the two major components of education reform ideology are as follows:

  • Reformy Ideology #1: Teacher quality is the one single factor that has the greatest effect on a child’s life chances. Get a bad teacher or two in a row, and you’re screwed for life. The “best possible” way to measure teacher quality is by estimating the teacher’s influence on student test scores (value-added). Hiring, retention and dismissal decisions must, that is, MUST be based primarily on this information. This information may be supplemented, but value-added must play the dominant single role.
  • Reformy Ideology #2: Charter schools are the answer to most of the problems of poor urban school districts. Take any poor, failing urban school district, close its dreadfully failing schools and replace them as quickly as possible with charter schools and children in the urban core will have greatly expanded high-quality educational opportunities.

Now, let me do a bit of clarification here. These are the representations of reform ideology at the extremes – but these views are not uncommon in the “reform” community. Let me also clarify that item #2 above isn’t about the broader issue of charter schooling, the origins of charter schooling, purposes of individual charter schools or research on the effectiveness of specific charter school models. Item #2 above is specifically about the argument that large urban districts can and should be replaced with an open market of charter schools – that charter schools should not just be used to try out new and interesting ideas which may be replicated in other schools – charter or not – but rather that charters should become dominant providers of urban education.

In my framing of item #1 above, I do not by any means intend to discredit the importance of high quality teachers. I’m with the “reformers” on that idea. But, it is certainly an overstatement to attribute all of student gains and life chances to teacher quality alone. And, as I have discussed previously on many occasions on this blog, it is very problematic to assume that we presently have sufficient tools for measuring precisely and accurately the true effectiveness of any single teacher.

So, this brings me to recent completely unrelated events and media on education reform issues that raise some interesting points of conflict.

Part I: Ideology Clashing with… Ideology

The first example of the clash of reformy ideologies comes from upstate New York as the state begins the process of implementing the “reforms” that got that state Race to the Top funding. In short, charter school operators really don’t seem to want to be compelled to adopt the first prong of reformy ideology. What? Charters don’t want to be compelled to use student test scores as the primary, or even a major basis for personnel decisions? Blasphemy!

In recent weeks, in casual conversations and at symposia, I’ve actually heard a number of charter school operators raise serious questions about being compelled to adopt Reform Ideology #1 above. Charter operators appreciate their autonomy, and while most do enjoy wider latitude over personnel decisions than large urban school districts that serve as their hosts, most do not necessarily base the majority of their hiring, firing and compensation decisions on student test scores and test scores alone. And most don’t seem very interested in being compelled to do so – adopting a one size fits all evaluation model. Apparently, they are also relatively uninterested in disclosing how they evaluate faculty. Here’s a quote from the Albany Times Union.

Carroll, one of the most prominent education reformers in the state, helped write the state’s original charter laws. He said if the charter schools accepted the money, they would lose their current flexibility in the firing and hiring of teachers. He also said charter schools would be forced to disclose their teacher evaluation process, which is now confidential, and that it could become harder to fire an educator deemed ineffective.

http://www.timesunion.com/default/article/No-to-cash-with-a-catch-714008.php

So then, if expanding charters is a major component of reform, and making sure teachers are evaluated by student test scores is a major component of reform, how can this apparent clash be reconciled? It can’t! It seems hypocritical at best to force public school districts to play by a seriously flawed set of teacher evaluation rules and then let charters off the hook? This is especially true if one of the supposed benefits of charter schools is to experiment with creative strategies that may be emulated by traditional public schools and if traditional public schools are expected to improve by competing with charters on a level playing field. I’m with the charter leaders on this one.

UPDATE: Tom Carrol has clarified his comments here: http://www.nyfera.org/?p=2827, where he attempts to explain that charters are opposed to the teacher evaluation requirements not because charters oppose the ideas behind using data to evaluate teachers, but that charters oppose having the state education department mandate how that data should be used in evaluations:

SED simply has no authority to set thresholds for the use of data in teacher evaluations in charter schools.  Nor do they have the authority to require us to group teachers by four categories, or require such annual evaluations to be “a significant factor” for “promotion, retention, tenure determination and supplemental compensation.”  Nor do they have the authority to require charters to pursue “the removal of teachers and principals receiving two consecutive annual ratings of ‘ineffective’ after receiving supports from improvement plans.”

Carrol’s clarification coupled with his unsubstantiated claim that charters are already doing these things really doesn’t change my point above – that it remains hypocritical to hoist these deeply problematic policies on traditional public schools while letting charters off the hook on the basis that charters should be given the flexibility to experiment with alternative strategies and should be exempt from full disclosure regarding those strategies.

Part II: Ideology Clashing with Research

Now, this one is really not an obvious major clash, but rather a more subtle clash between ideology and research embedded in Eric Hanushek’s WSJ editorial the other day.  Embedded in the editorial were comments/claims that I found at least a little disingenuous.

The point of Hanushek’s OpEd was to explain that there is no war on teachers, and that this is really about getting teacher’s unions to negotiate for reasonable changes in contracts that would allow for more expeditious dismissal of the worst teachers – the bottom 5% or so. I’ll admit that I really haven’t seen Hanushek himself outright attacking teachers in his work of late, especially in his actual research on teacher labor markets. Much of it is very good and very useful stuff. That said, it’s hard to deny that many major public figures – talking head tweeters and bloggers for think tanks, etc. – have actually engaged in an all out attack on teachers, the teaching profession and teachers unions.

Setting that broader issue aside, Hanushek’s Op Ed rubbed me the wrong way because of examples he chose to advance his argument, and the extent to which his examples in the Op Ed clash – and clash quite significantly – with his own best recent research. Hanushek summarized his arguments about the non-war on teachers as follows:

What’s really going on is different. President Obama states that we can’t tolerate bad teachers in classrooms, and he has promoted rewarding the most effective teachers so they stay in the classroom. The Los Angeles Times published data identifying both effective and ineffective teachers. And “Waiting for ‘Superman'” (in which I provide commentary) highlighted exceptional teachers and pointed out that teachers unions don’t focus enough on teacher quality.

This is not a war on teachers en masse. It is recognition of what every parent knows: Some teachers are exceptional, but a small number are dreadful. And if that is the case, we should think of ways to change the balance.

http://online.wsj.com/article_email/SB10001424052748703794104575546502615802206-lMyQjAxMTAwMDEwODExNDgyWj.html

So, part of his claim is that it was unjustified for teachers unions – not teachers mind you – but their unions to object so loudly when the LA Times merely – in the public interest – revealed data which validly (implied above, by absence of disclaimers) identified, labeled and named effective and ineffective teachers.

Wait… isn’t it Eric Hanushek’s own research and writing that highlights problems with using value-added measurement to evaluate teachers where non-random student assignment occurs (which is pretty much anywhere)? For me, it was my familiarity with his work that led me to explore the biases in the LAT model that I’ve written about previously on this blog.  In that same post, I explain why I am more inclined to accept Jesse Rothstein’s concerns over the problems of non-random assignment of students than to brush those concerns aside based on the findings of Kane and Staiger.  Hanushek provides a compelling explanation for why he too places more weight on Rothstein’s findings.

Correct me if I’m wrong, but isn’t it possible that teachers and their union were in an uproar at least partly because the LA Times released highly suspect, potentially error ridden and extremely biased estimates of teacher quality? And that the LA Times misrepresented those estimates to the general public as good, real estimates of actual teacher effectiveness?

Yes, much of Eric Hanushek’s recent writing does advocate for some reasonable use of value-added estimates for determining teacher effectiveness, but he usually does so while giving appropriate attention to the various caveats and while emphasizing that value-added estimates should likely not be a single determining factor. He notes:

Potential problems certainly suggest that statistical estimates of quality based on student achievement in reading and mathematics should not constitute the sole component of any evaluation system.

http://edpro.stanford.edu/hanushek/admin/pages/files/uploads/HanushekRivkin%20AEA2010.CALDER.pdf

Yet, that’s just what the LA Times did, and without even mentioning the caveats!

Isn’t it a bit of an unfair assertion, given Hanushek’s own research and writing on value-added estimates, to claim that LA teachers and their union were completely unjustified in their response to the LA Times?

Part III: Ideology Clashing with Reality

There exists at least one segment of the truly reformy crowd that believes deeply in second major ideology laid out at the beginning of this post – that if we can simply close failing urban schools (the whole district if we have to!) and let charters proliferate, children in the urban core will have many more opportunities to attend truly good schools. Yes, these reformers throw in the caveat that we must let only “good” charters, “high performing” charters start-up in place of the failing urban schools. And when viewing the situation retrospectively, these same reformy types will point out that if we look only at the upper half of the charters, they are doing better than average. Yeah… Yeah… whatever.

One long-term research project that has interested me of late is to look in-depth at those “failing” urban school districts that over the past decade have had the largest shares of their student population shift to charter schools – that is, the largest charter market share districts. Here is link to the Charter Market share report from the National Alliance for Public Charter Schools: http://www.publiccharters.org/Market_Share_09

It would seem that if we adopt the reformy ideology above, that if we identify those districts with the largest charter market shares, those districts should now be models for high quality, equitably distributed educational opportunities. We should eventually see sizeable effects in the achievement and attainment of children growing up in these cities, we should see quality of life increasing dramatically, housing values improving with an influx of families with school-aged children – a variety of interesting, empirically testable hypotheses, which I hope to explore in the future.

In the mean-time, however, we have new and interesting descriptive information from a report from the Ewing and Marion Kauffman Foundation in Kansas City, focused on educational opportunities in Kansas City. Kansas City is #4 on charter market share, according to the National Alliance report, and rose to that position much earlier in the charter proliferation era than other cities. As a result, by reformy logic, Kansas City should be a hotbed for educational opportunity for school-aged children – after years of previously throwing money down the drain in the Kansas City Missouri Public School District (many of these claims actually being Urban Legend).

In Kansas City, the reality of charter expansion has clashed substantially with the reformy ideology. Arthur Benson in a recent Kansas City Star Op Ed, noted:

Charters have subtle means for selecting or de-selecting students to fit their school’s model. The Kansas City School District keeps its doors open to non-English speakers and all those kids sent back from the charter schools. In spite of those hurdles, Kansas City district schools across the board out-perform charter schools. That is not saying much. We have until recently failed 80 percent of our kids, but most charters fail more.

I was initially curious about Benson’s (a district board member and attorney) claims that charters have done so poorly in Kansas City. Could it really be that the massive expansion of charter schools in Kansas City has done little to improve and may have aided in the erosion of high quality educational opportunities for Kansas City children?

The recent Kauffman Foundation report draws some similar conclusions, and Kauffman Foundation has generally been an advocate for charter schools. The report classifies district and charter schools into groups by performance, with level 4 being the lowest, and level 1 being the only acceptable group.

  • Level I- A school that met or exceeded the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level II- A school that scored between 75 and 99 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level III– A school that scored between 50 and 74 percent of the state standard on the MAP Communication Arts and Mathematics exams in 2008-2009.
  • Level IV– A school that scored below 50 percent of the state standard on the MAP Communication

Among other things, the report found that charter operators had avoided opening schools in the neediest neighborhoods. Rather, they set up shop in lower need neighborhoods, potentially exacerbating disparities in opportunities across the city’s zip codes. The report recommended:

A strategy for charter school growth should be developed by Kansas City education leaders. Charter schools should only be approved by DESE if they can demonstrate how they intend to fill a geographic need or a specific void in the communities they intend to serve.

Regarding charter performance more generally, the report noted:

In many communities charter schools are a model that increases students’ access to better public schools, but the majority of charter school students (5,490 or 64.7 percent) are in a Level IV school. Many of Kansas City’s charters have existed for 10 years and are still not able to reach even half of state standard.

Now, I’m not sure I accept their premise that in many communities this actually works – and that it just went awry for some strange reason in Kansas City. That said, the reality in Kansas City, by the authors own acknowledgment is in sharp contrast with the reality the authors believe exists in other cities.

One implication (not tested directly) of this report is that the massive charter school expansion that occurred in Kansas City may have done little or nothing to improve the overall availability or distribution of educational opportunities for children in that city and may have actually made things worse.

Isn’t it strange how we hear so little about these things as we look to replicate these models of great reformy success in other cities of comparable scale such as Newark, NJ?

On False Dichotomies and Warped Reformy Logic

Pundit Claim 1 – Value added modeling is necessarily better than the “status quo”

There exists this strange perspective that we are faced with a simple choice in teacher evaluation – a choice between using student test scores and value-added modeling, or continuing with the status quo. This is a false dichotomy, false dilemma or logical fallacy. In other words, it’s a really stupid argument in which we are forced to assume that there are only two choices that exist. This argument is usually coupled with an implicit assumption that one of the two must be superior.

“Reformers” continue to press the argument that current teacher evaluations are so bad, so unreliable, that anything is better than this “status quo.”

Expressed mathematically:

Anything > Status Quo

Bear with me while I use the “greater than” symbol to imply “really freakin’ better than… if not totally awesome… wicked awesome in fact,” but since it’s relative, it would have be “wicked awesomer.”

Because value-added modeling exists and purports to measure teacher effectiveness, it therefore counts as “something,” which is a subclass of “anything” and therefore it is better than the “status quo.” That is:

Value-added modeling = “something”

Something ⊆ Anything (something is a subset of anything)

Something > Status Quo

Value-added modeling > Current Teacher Evaluation

Again, where “>”  means “awesomer” even though we know that current teacher evaluation is anything but awesome.

It’s just that simple!

After all, you can’t even measure the error rate in current principal and supervisor evaluations of teachers can you? And if you can’t measure the error rate it must be higher than any error rate you can measure? More really basic reformy logic! That is, the unobserved error rate in one system is necessarily greater than the observed error rate of another – even if we have no way to quantify it – in fact, because we have no way to quantify it?

Unobserved error rate of ‘status quo’ > measured error rate of VAM

Let’s be really blunt here. Both are patently stupid arguments.

And both of these arguments bring to mind one of my favorite analogies related to this issue. If we were in a society that still walked pretty much everywhere, and some tech genius invented a new cool thing – called the automobile – but the automobile would burst into a superheated fireball on every fifth start, I think I’d keep walking until they worked out that little kink. If they never worked out that little kink, I’d probably still be walking. I’ve written previously about how this relates to likely error rates in teacher dismissal (misclassifying truly effective teachers as ineffective) as would occur when using typical value-added modeling approaches.

Pundit Claim 2 – If we get rid of the bad teachers, the system will necessarily be better

The assumption of many pundits is that replacing existing teachers necessarily improves the teaching workforce – that the average potential applicant for any/all available teaching jobs will be better than the average person already there, or at least better than the person we dismiss as ineffective. Now, recall that we have a pretty high chance of misclassifying truly effective teachers and dismissing them.

Now, the math here is similar to that above. The basic premise is that:

Anything > Status Quo

First of all, we know already that schools with more difficult working conditions have a much more difficult time recruiting and retaining quality teachers. Working conditions play a significant role in teacher sorting in initial job matches and in teacher moves over time.

We also know, just by looking at such information as the patterns of higher and lower “effectiveness” scores in the LA Times analysis, that if we dismiss teachers on the basis of their value added scores, we will be dismissing larger shares of teachers in higher poverty, higher minority schools. Or, we can just take the Central Falls, RI approach and declare the entire school failing based on its average performance over time (setting aside demographics and resources) and just fire everyone. Surely the replacements will be better. How could we do worse? Right?

Here’s the thing – even if we assume that some of the lower performance of teachers in poorer LA schools or the lower performance of Central Falls HS is a function of a weaker, less effective teacher workforce, we can only make things “better” by replacing that workforce with “better” teachers.

It is completely arrogant to take the reformy attitude of “how can we possibly do worse?” How could we possibly get a worse pool of teachers than the lazy slugs already in the system?

If the teacher pool in these schools is in fact less effective, and don’t just look that way statistically because of other factors, it may just be that these schools had a difficult time recruiting and retaining teachers to begin with. If we introduce our “game changing” policies – firing all of the teachers for low school performance, or firing individual teachers for bad effectiveness ratings – we will likely make things even worse.

Any teacher wishing to step in line to replace the previous cohort of “failures,” will have to not only consider the difficult working conditions but also the disproportionate likelihood that she/he will be fired a few years down the line, for factors well beyond his/her control (e.g. that pesky non-random assignment problem). That’s a significant change in working conditions – job risk. Without either changing other working conditions or substantially increasing compensation to offset this new risk, the applicant pool is not likely to get better – especially when risk is not increased similarly in other “more desirable” school districts. All else equal, the applicant pool is likely to get worse. The disparity in the quality of applicants for teaching positions is likely to increase dramatically, and the average quality of applicants to high poverty, high minority concentration districts may decline significantly.

Bonus video with thanks to Sherman Dorn:

If money doesn’t matter…

A) Then why do private independent schools, like those attended by our President’s children (Sidwell Friends in DC), or by Davis Guggenheim’s children (?), spend so much more than nearby traditional public schools?

Davis Guggenheim, producer of Waiting for Superman, frequently explains to the media these days that he feels uneasy that he has made a personal choice drive by his neighborhood school each day to bring his children to a private school. Now, I don’t know which private school his children attend, but I would suspect (though I may be wrong) that it is more likely to be an academically elite, private independent school than to be a conservative Christian or urban Catholic school. As I discuss in this previous report, the spending differences and resulting programmatic resources and teacher characteristics by type of private school are striking: http://epicpolicy.org/publication/private-schooling-US

I would see little problem with Guggenheim’s personal anecdote were it not for one of the central arguments of Superman being that money plays little or no role in fixing public education systems. Instead, tough-minded superintendents like Michelle Rhee, or charter schools are the solution – money or not.

Again, I’ll fess up to the fact that I am a former teacher at and big supporter of Private Independent Schools. Here’s the school in New York City where I used to teach www.ecfs.org, and here’s its page on tuition: http://www.ecfs.org/admission/tuition.aspx. It was then, and I suspect still is an outstanding example of what a school can be! But that outstanding-ness comes at a price!

(approximately $36,000 per year for middle school and up)

The problem with the assertion that “money wouldn’t help public schools anyway” is that many of those pitching the argument seem themselves to favor private schools that spend more – A LOT MORE – per child than the public schools they are criticizing as failing (speak nothing of the fact that the public schools are serving a much more diverse student population).

Here are some comparisons pulled from my 2009 study on private school expenditures.

First, here’s the per pupil spending in 2005-06 for a handful of major labor markets that had sufficient numbers of Private Independent Day Schools for calculating the averages. My original sample of IRS Tax filings covered about 75% of all Private Independent Day Schools (NAIS or NIPSA member schools), so these are not “outlier” schools.

FIGURE 1 (This figure is now the figure from my original report: http://epicpolicy.org/publication/private-schooling-US)

And here are the regional averages, adjusted for regional differences in competitive wages, using the NCES Education Comparable Wage Index.

FIGURE 2 (This figure is now the figure from my original report: http://epicpolicy.org/publication/private-schooling-US)

If money doesn’t matter when it comes to school quality, then why not pick one of those private schools that charges only $6,000 in tuition, and spends $8,000 per pupil? Clearly there is some basis for the decision to send a child to a more expensive private school? There is some “utility” placed on the differences in what those schools have to offer? In the complete report above, I discuss (in painful detail) those differences across private schools, but here, I quickly summarize some of the differences between private independent schools and traditional public school districts.

FIGURE 3 (This figure is now the figure from my original report: http://epicpolicy.org/publication/private-schooling-US)

Private independent schools a) spend a lot more per pupil, b) have much lower pupil to teacher ratios and c) have much higher shares of teachers who attended more competitive colleges. These seem like potentially substantive differences to me. And they are differences that come at a cost.

I am by no means criticizing the choice to provide your own child with a more expensive education. That is a rational choice, when more expensive is coupled with substantive, observable differences in what a school offers. I am criticizing the outright hypocritical argument that money wouldn’t/couldn’t possibly help public schools provide opportunities (breadth of high school course offerings, smaller class sizes) more similar to those of elite private independent day schools, when this argument is made by individuals who prefer private schools that spend double what nearby public schools spend.

Sidebar: I suspect there are few if any private independent day schools out there which currently evaluate their teachers based on student test score gains alone. Please let me know if you know of one? And, I should note that the private independent school where I worked in New York City was actually unionized and had a tenure system in place with a probationary period similar to that of public schools and a salary schedule tied to experience.

B) Then why do venture philanthropists continue to throw money at charter schools while throwing stones at traditional public schools?

Charter school backers like Whitney Tilson love to throw stones at public schools while throwing money at charter schools. Here’s one of his presentations:

http://www.tilsonfunds.com/Personal/TheCriticalNeedforGenuineSchoolReform.pptx

On Slide 13, Whitney Tilson opines that increased spending on public education has yielded no return to outcomes over time, and therefore, by extension, increased spending would not and could not help public schools in the future. Tilson is featured prominently in this New York Times article on affluent fund managers in NYC rallying for charter schools: http://www.nytimes.com/2010/05/10/nyregion/10charter.html?pagewanted=all

So, here we have one of many prominent New York City charter school supporters on the one hand arguing that throwing more money at the public school system could not possibly help that system, but on the other hand, providing substantial financial assistance to charter schools (or at least participating in and promoting groups that engage in such activity)?

A New York City Independent Budget Office report suggested that charter schools housed in public school facilities have comparable public subsidy to traditional NYC public schools, but charter schools not housed in public school facilities have to make up about $2,500 (per pupil) in difference. I will show in a future post, however, that student population differences (charters serving lower need populations) largely erase this differential.

Kim Gittleson points out here, that in 2008-08, NYC Charter schools raised an average of $1,654 per pupil through philanthropy. But, some raised as much as $8,000 per pupil. As a result, some charters – those most favored by venture philanthropists – spend on a per pupil basis much more than traditional NYC public schools (including KIPP schools). I will provide much more detail in this point in a future post.

One might argue that the Venture Philanthropists are trying to spend their way to success – To outspend the public schools in order to beat them! After all, it’s the New York Yankee, George Steinbrenner way? (spoken from the perspective of a Red Sox fan, who spent the last several years in Kansas City, supporting the underdog – low payroll – Royals).

But here’s the disconnect – These same Venture Philanthropists – like Tilson, who are committed to spending whatever it takes on charters in order to prove they can succeed, are arguing that public schools a) don’t need and b) could never use effectively any more money. They are trying to argue that charters are doing more with less, when some are doing more with more, others less with less, and some may be doing more with less, and others are actually doing less with more. Shouldn’t traditional public schools be given similar opportunity to do more with more? And don’t give me that … “we’ve already tried that and it didn’t work” claim. I’ll gladly provide the evidence to refute that one, much of which is in the article at the bottom of this post!

C) Then why do affluent – and/or low poverty – suburban school districts continue in many parts of the country to dramatically outspend their poorer urban neighbors?

Last but not least, why do affluent suburban school districts in many states continue to far outspend poor urban ones? If there is no utility to the additional dollar spent and/or no effect produced by that additional dollar then why spend it?

Here is the overall trend, over time in the relationship between community income and state and local revenues per pupil.

When the red line is above the green horizontal line, there exists a positive relationship between district income and state and local revenue. That is, higher income districts have more state and local revenue per pupil. The red line never drops below the green line. This graph, drawn from this article (http://epaa.asu.edu/ojs/article/view/718) shows that state and local revenues per pupil remain positively associated with income across school districts nationally, after controlling for a variety of factors (see article for full detail). Things improved somewhat in the 1990s, but then leveled off.

FIGURE 4 (from: http://epaa.asu.edu/ojs/article/view/718)

Here are the trends for mid-Atlantic states, where some including New York State improved, but remain strongly associated with income. New Jersey is the only state among these where the relationship between income and revenue is disrupted and ultimately reversed.

FIGURE 5 (from: http://epaa.asu.edu/ojs/article/view/718)

Here are the trends for the New England trend, where New Hampshire school district state and local revenues remain strongly tied to income.

FIGURE 6 (from: http://epaa.asu.edu/ojs/article/view/718)

Here are the trends for the Great Lakes are trend, where Illinois remains among the most regressively funded systems in the nation (along with New York).

FIGURE 7 (from: http://epaa.asu.edu/ojs/article/view/718)

Here’s a specific look at state and local revenues per pupil in New York State districts in the NY metropolitan area, with districts organized by U.S. Census Poverty rates.

FIGURE 8

Is there a reason why Westchester County and Long Island school districts choose to spend so much more than New York City on a per pupil basis? What about those North Shore Chicago area districts?

These communities demand higher expenditures per pupil for their schools on a presumption that the marginal dollar does not go entirely to waste – that there is some value, some return for that dollar, perhaps in the richness of supplemental programs offered or the smaller class sizes – much like the differences in private schools seen above.

Finally, I point you to this recently published article in Teachers College Record, where Kevin Welner and I try to set the record straight on the effectiveness of “reforms” involving state school finance systems. They’re not the “reformy” reforms, but school finance reforms are reforms nonetheless.

Baker, B.D., Welner, K. School Finance and Courts: Does Reform Matter, and How Can We Tell? Teachers College Record

http://www.tcrecord.org/content.asp?contentid=16106

DoReformsMatter.Baker.Welner

Value-Added and “Favoritism”

Kevin Carey from Ed Sector has done it again. He’s come up with yet another argument that fails to pass even the most basic smell test. A few weeks ago, I picked on Kevin for making the argument that while charter schools, on average, are average, really good charter schools are better than average. Or, as he himself phrased it:

reasonable people acknowledge that the best charter schools–let’s call them “high-quality” charter schools–are really good

I myself am reasonable on occasion and fully accept this premise. Some schools are really good, and some not so good. And that applies to charter schools and non-charters alike, as I show in my recent post Searching for Superguy.

Well, last week Kevin Carey did it again – made a claim that simply doesn’t even pass the most basic smell test.  In the New York Times Room for Debate series on value-added measurement of teachers, Carey argued that Value-added measures would protect teachers from favoritism. Principals would no-longer be able to go after certain teachers based on their own personal biases. Teachers would be able to back up their “real” performance with hard data. Here’s a quote:

“Value-added analysis can protect teachers from favoritism by using hard numbers and allow those with unorthodox methods to prove their worth.” (Kevin Carey, here)

The reality is that value-added measures simply create new opportunities to manipulate teacher evaluations through favoritism. In fact, it might even be easier to get a teacher fired by making sure the teacher has a weak value-added scorecard. Because value-added estimates are sensitive to non-random assignment of students, principals can easily manipulate the distributions of disruptive students, students with special needs, students with weak prior growth and other factors, which, if not fully accounted for by the VA model will bias teacher ratings. And some factors – like disruptive students, or those who simply don’t give a $#*! won’t (and can’t) be addressed in the VA models. That is, a clever principal can use the VA non-random assignment bias to create a statistical illusion that a teacher is a bad teacher. One might argue that some principals likely already engage in a practice of assigning more “difficult” students to certain teachers – those less favored by the principal. So, even if the principal is less clever and merely spiteful, the same effect can occur.

I wrote in an earlier post about the types of contractual protections teachers should argue for, in order to protect against such practices:

The language in the class size/random assignment clause will have to be pretty precise to guarantee that each teacher is treated fairly – in a purely statistical sense. Teachers should negotiate for a system that guarantees “comparable class size across teachers – not to deviate more than X” and that year to year student assignment to classes should be managed through a “stratified randomized lottery system with independent auditors to oversee that system.” Stratified by disability classification, poverty status, language proficiency, neighborhood context, number of books in each child’s home setting, etc. That is, each class must be equally balanced with a randomly (lottery) selected set of children by each relevant classification.

This may all sound absurd, but sadly, under policies requiring high stakes decisions such as dismissal to be based on value added measures, this stuff will likely become necessary. And, it will severely constrain principals who wish to work closely with teachers on making thoughtful, individualized classroom assignments for students. I address the new incentives of teachers to avoid taking on the “tough” cases in this post: https://schoolfinance101.wordpress.com/2010/09/01/kids-who-don%E2%80%99t-give-a-sht/

Technical follow-up: I noticed that Kevin Carey claims that VA measures “level the playing field for teachers who are assigned students of different ability.” This statement, as a general conclusion, is wrong.

a) VA measures do account for the initial performance level of individual students, or they would not be VA measures. Even this becomes problematic when measures are annual rather than fall/spring, so that summer learning loss is included in the year to year gain. An even more thorough approach for reducing model bias is to have multiple years of lagged scores on each child in order to estimate the extent to which a teacher can change a child’s trajectory (growth curve). That makes it more difficult to evaluate 3rd or 4th grade teachers, where many lagged scores aren’t yet available. The LAT model may have had multiple years of data on each teacher, but didn’t have multiple lagged scores on each child. All that the LAT approach does is to generate a more stable measure for a teacher, even if it is merely a stable measure of the bias of which students that teacher typically has assigned to him/her.

b) VA measures might crudely account for socio-economic status, disability status or language proficiency status, which may also  affect learning gains. But, typical VA models, like the LA Times model by Buddin tend to use relatively crude, dichotomous proxies/indicators for these things. They don’t effectively capture the range of differences among kids. They don’t capture numerous potentially important, unmeasured differences.  Nor do they typically capture classroom composition – peer group – effect which has been shown to be significant in many studies, whether measured by racial/ethnic/socioeconomic composition of the peer group or by average performance of the peer group.

c) For students who have more than one teacher across subjects (and/or teaching aides/assistants), each teacher’s VA measures may be influenced by the other teachers serving the same students.

I could go on, but recommend revisiting my previous posts on the topic where I have already addressed most of these concerns.

Searching for Superguy in Gotham

Who is Superguy? By most popular accounts, Superguy is a figure of mythical proportion (urban legend proportion) capable of swooping down into the poorest of urban neighborhoods of America’s largest cities, gaining immediate access to schooling facilities, rounding up unthinkable private contributions from wealthy philanthropists and quite simply saving the lives of low-income urban school children trapped in bleak, adult-centered, perpetually failing traditional public schools.

Superguy could be found anywhere in the U.S where urban charter schools have proliferated in the past decade – Kansas City, Washington D.C., Chicago, Dallas, Houston, or even more likely, New York City – Gotham itself (yeah… Gotham was a Batman thing, not Superman… but hang in there with me).  I’ve chosen to focus on urban locations here, because who has ever heard of a “rural legend?”

I’ve written on a number of occasions about my general skepticism that Superguy really exists, or that he necessarily exists in the form of an urban charter school operator. My skepticism is based on my own read of the balance of research on charter schools and my own casual analysis  of New York City and New Jersey Charter Schools. New Jersey Charter Schools in particular are pretty average and those that are better than average serve very few of the lowest income children, no special needs children and few or no limited English proficient children. Personally, I’d expect Superguy to be out there fighting for these kids in particular, not just setting up shop in their neighborhood and cream-skimming the less needy among the more needy. But hey, that’s just my notion of what Superguy should be.

For an exceptional review of charter school research, I would recommend Robert Bifulco and Katrina Bulkley’s chapter on Charter Schools in the Handbook of Research on Education Finance and Policy. Neither of these scholars are charter school naysayers, yet they conclude:

Research to date provides little evidence that the benefits envisioned in the original conceptions of charter schools – organizational and educational innovation, improved student achievement, and enhanced efficiency – have materialized.

Of course, the true believers in Superguy (as charter operator) will argue vehemently that the finding that charters, on average, are average does not shake their belief… because the “upper half of charter schools is really good!, better than average, in fact!” Skeptically, I respond – isn’t the upper half of all schools better than average? If so, might Superguy actually be found in any school that’s better than average? But who am I to nitpick?

The most compelling evidence that Superguy exists was provided in Caroline Hoxby’s finding regarding NYC charter schools that:

On average, a student who attended a charter school for all of grades kindergarten through eight would close about 86 percent of the “Scarsdale-Harlem achievement gap” in math and 66 percent of the achievement gap in English.

Who other than Superguy could close the Harlem-Scarsdale gap? However, Stanford University researcher Sean Reardon explains:

Because the report relies on an inappropriate set of statistical models to analyze the data, however, the results presented appear to overstate the cumulative effect of attending a charter school.

Superguy in Gotham is also assumed to have competitive effects, lifting entire neighborhoods wherever he may be present. This evidence is often cited to Marcus Winters’ (Manhattan Institute) finding that:

The analysis reveals that students benefit academically when their public school is exposed to competition from a charter.

But thwarting this Superguy sighting is Wellesley economist Patrick McEwan’s observation that:

The statistical analysis suggests that increasing competition has no statistically significant impact on math test scores, but that it has small positive effects on language scores. The report does not conclusively demonstrate that the results are explained by increasing competitive pressure on public school administrators; they may also be explained by shifting peer quality or declining short-run class sizes in public schools.

Those pesky, curmudgeonly,  academics are at it again… denying the true believers that Superguy comes in the singular form of a New York City charter school operator!

Then there’s the claim that Superguy himself may have been outed in Harlem (is Superguy really Geoffrey Canada?) – as evidenced by Dobbie and Fryer’s studies of the amazing success of Harlem Children’s Zone. But then Russ Whitehurst of Brookings stepped in to rain on this parade, finding the HCZ Promise academy to be relatively average as far as NYC charter schools go.

For several additional curmudgeonly critiques of Superguy sightings, see: http://epicpolicy.org/search/epicpolicy/charter

It is with these contradictory findings in mind that I present the following figures, and we begin our statistical search for Superguy. Now, this is not a really deep, statistically rigorous search for Superguy. The approach here is what some refer to as a “beating the odds” approach (BTO) and is similar to the adjusted performance approach used by Whitehurst in his Brooking critique of HCZ.

It seems that the logical place to start would be New York City, home to the greatest number of Superguy sightings. Let’s begin with a simple flyover of NYC schools, including traditional public schools and charter schools just to get a feel for the demographics of those schools.

Here is the % of children qualifying for Free Lunch across Harlem (and South Bronx) schools. This map does not indicate which schools are charters, but you can click the link below the map to see which ones are.

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS (indicated with an asterisk)

Here is the % of children who are limited in their English language proficiency. Again, this map doesn’t show us which schools are charters, but you can click the link below.

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS (indicated with an asterisk)

Now, here’s our first “Beating the Odds” scatterplot. The predicted performance values expressed on the vertical axis are from a regression equation that accounts for a) limited English proficiency shares, b) free lunch shares, c) mobility shares, d) borough and e) year (includes 2008 & 2009). These graphs look at the adjusted performance levels (not value-added) of NYC traditional public and NYC charter schools (standardized difference between actual and predicted values for 2009). These are illustrative. While outcome levels do go up (are inflated) in 2009, the distributions in these scatterplots don’t change a whole lot if I use 2008 or earlier. (here are the models)

The first Beating the Odds scatterplot looks at the average performance from 2009 (yes, the really inflated test score year) for NYC public schools. Charter schools are not identified in this graph. Schools are displayed from low to high poverty along the horizontal axis. Schools above the red horizontal line are beating the odds, or scoring higher than expected given their location and student population. Schools below the line are, well, not beating the odds. We would, of course, expect our Superguy operated schools to be flying high above the rest and certainly not falling well below… at the bottom of the scatter. So, where is Superguy?

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS

Now, here’s the average of the 4th and 5th grade outcomes. Same deal. A good ol’ BTO analysis (yeah… this isn’t really rigorous stuff, but it is illustrative). Again, charters are not identified in this picture. Yes, there are some high and some low flyers in this graph too. But are all of the high flyers charter schools? Is Superguy really here?

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS

While I don’t think we’ve found Superguy here, we are left with some potential clues about the conditions surrounding Superguy sightings – A) that superguy sightings seem more common in the presence of unexplainable deficits in the shares of children who qualify for Free Lunch, B) that Superguy sightings seem more common in the presence of unexplainable deficits in the shares of children with limited English proficiency. Other than that, it seems that Superguy is equally likely to be hiding in a traditional New York public school as it is that Superguy is secretly disguised as a charter school operator somewhere in Gotham.

Alternatively, there exists the depressing but real possibility that Superguy simply doesn’t exist – at least not in the expected form. That there just isn’t a charter school operator out there who can single-handedly swoop into poor urban neighborhoods and save childrens lives – creating results never seen before with a truly representative population of children. Or, at the very least, not all or even the average charter school operator qualifies as Superguy. Yes, some are better than others. And, some are quite good. But you know what I have to say about that argument (see above).

Yeah… I’d like to be a believer. I don’t mean to be that much of a curmudgeon. I’d like to sit and wait for Superguy – perhaps watch a movie while waiting (gee… what to watch?). But I think it would be a really long wait and we might be better off spending this time, effort and our resources investing in the improvement of the quality of the system as a whole. Yeah, we can still give Superguy a chance to show himself (or herself), but let’s not hold our breath, and let’s do our part on behalf of the masses (not just the few) in the meantime.

Value-added and the non-random sorting of kids who don’t give a sh^%t

Last week, this video from The Onion (asking whether tests are biased against kids who don’t give a sh^%t) was going viral among the education social networking geeks like me. At the same time, the conversations continued on the Los Angeles Times Value-Added story, with LAT releasing the scores for individual teachers.

I’ve written many blog posts in recent weeks on this topic. Lately, it seems that the emphasis on the conversation has turned toward finding a middle ground – discussing the appropriate role for VAM (Value Added Modeling) – if any, in teacher evaluation. But also, there is renewed rhetoric defending VAM. Most of that rhetoric seems to take on most directly the concern over the error rates in VAM – and lack of strong year to year correlation between which teachers are rated high or low.

The new rhetoric points out that we’re only having this conversation about VAM error rates because we can measure the error rate in VAM, but can’t even do that for peer or supervisor evaluation – which might be much worse (argue the pundits). The new rhetoric argues that VAM is still the “best available” method for evaluating teacher “performance.” Let me point out that if the “best available” automobile burst into flames on every fifth start, I think I’d walk or stay home instead. I’d take pretty significant steps to avoid driving. Now, we’re not talking about death by VAM here, but the idea that random error alone – under an inflexible VAM based policy structure – could lead to wrongfully firing a teacher is pretty significant.

Again, this current discussion pertains only to the “error rate” issue. Other major – perhaps even bigger issues include the problem that so few teachers could even have test scores attached to them – creating a whole separate sub-class (<20%) of teachers in each school system and increasing divisions among teachers – creating significant tension, for example between teachers under the VAM (math/reading) rating system, and teachers who might want to meet with some of their students for music, art or other enrichment endeavors.

Perhaps most significantly, there still exists that pesky little problem of VAM not being able to sufficiently account for the non-random sorting of students across schools and teachers. For those who wish to use Kane and Staiger as their out on this (without reference to broader research on this topic), see my previous post on the LAT analysis. Their findings are interesting, but not the single definitive source on this issue. Note also that the LAT analysis itself reveals some bias likely associated with non-random assignment (the topic of my post).

So then, what the heck does this have to do with The Onion video about testing and kids who don’t give a sh^%t?

I would argue that the non-random assignment of kids who don’t give a sh^%t presents a significant concern for VAM. Consider any typical upper elementary school. It is quite possible that kids who don’t give a sh^%t are more likely to be assigned to one fourth grade teacher year-after-year than to another. This may occur because that fourth grade teacher really wants to try to help these kids out, and has some, though limited success in doing so. This may also occur because the principal has it in for one teacher – and really wants to make his/her life difficult. Or, it may occur because all of the parents of kids who do give a sh^%t (in part because their parents give a sh^%t) consistently request the same teacher year after year.

In all likelihood, whether the kids give a sh^%t about doing well – and specifically doing well on the tests used for generating VA estimates – matters, and may matter a lot. Teachers with disproportionate numbers of kids who don’t give a sh^%t may, as a result receive systematically lower VA scores, and if the sorting mechanisms above are in place, this may occur year after year.

What incentive does this provide for the teacher who wanted to help – to help kids give a sh^%t? Statistically, even if that teacher made some progress in overcoming the give a sh^%t factor, the teacher would get a low rating because give a sh^%t factor would not be accounted for in the model. Buddin’s LAT model includes dummy variables for kids who are low income and kids who are limited in their English language proficiency. But, there’s no readily available indicator for kids who don’t give a sh^%t. So we can’t effectively compare one teacher with 10 (of 25) kids who don’t give a sh^%t to another with 5 (of 25) who don’t give a sh^%t. We can hope that giving a sh^%t , or not, is picked up by the child’s prior year performance, and even better, by the prior multiple years of value-added estimates on that child. But, do we really know whether giving a sh^%t is a stable student characteristic over time? Many VAM models like the LAT one don’t capture multiple prior years of value-added for each student.

I noted in previous posts that peer-effect is among those factors that compromises (biases) teacher VAM ratings. Buddin’s LAT model, as far as I can tell, doesn’t try to capture differences in peer group when attempting to “isolate” teacher effect (though this is very difficult to accomplish). Unlike racial characteristics or child poverty, whether 1 or 10 kids in a class give a sh^%t might rub off on others in the class. Or, the disruptive behavior of kids who don’t give a sh^%t might significantly compromise the learning (and value-added estimates) of others. Yet, all of this goes unmeasured in even the best VAMs.

Once again, just pondering…

NEW: BONUS VIDEO

http://www.youtube.com/watch?v=OMivkYJbcAk&feature=player_embedded

Why I’m not crying for Louisiana and Colorado

Many of the “reformers” out there are whining and fist-thumping about the surprise omission of Louisiana and Colorado as Race to the Top Winners. After all, Louisiana has been a heavy favorite from the outset of RttT, and Colorado… well Colorado took the amazingly bold leap of adopting legislation to mandate that a majority of teacher evaluation be based on value-added test scores. That’s got to count for something. Heck, these two states should have gotten the whole thing? Here’s Tom Vander Ark’s take on this huge surprise loss: http://edreformer.com/2010/08/co-la-surprise-losers/

Now here’s why I find it somewhat of a relief that these two states did not find themselves in the winners’ circle (not that I can identify a great deal of logic to support those who did… but…).

I’ve written numerous times about Louisiana’s public education system, and that state’s support or lack-thereof for providing a decent quality education for the children of Louisiana.

https://schoolfinance101.wordpress.com/2009/12/18/disg-race-to-the-top/

Here’s an excerpt from that previous post:

Let’s take a look at Louisiana’s education system. Yes, their system needs help, but the reality is that Louisiana politicians have never attempted to help their own system. In fact they’ve thrown it under the bus and now they want an award? Here’s the rundown:

  • 3rd lowest (behind Delaware & South Dakota) % of gross state product spent on elementary and secondary schools (American Community Survey of 2005, 2006, 2007)
  • 2nd lowest percent of 6 to 16 year old children attending the public system at about 80% (tied with Hawaii, behind Delaware) (American Community Survey of 2005, 2006, 2007). The national average is about 87%.
  • 2nd largest (behind Mississippi) racial gap between % white in private schools (82%) and % white in public schools (52%) (American Community Survey of 2005, 2006, 2007).  The national average is a 13% difference in whiteness, compared to 30% in Louisiana.
  • 3rd largest income gap between publicly and privately schooled children at about a 2 to 1 ratio. (American Community Survey of 2005, 2006, 2007)
  • 4th highest percent of teachers who attended non-competitive or less competitive (bottom 2 categories) undergraduate colleges based on Barrons’ ratings (NCES Schools and Staffing Survey of 2003-04). Almost half of Louisiana teachers attended less or non-competitive colleges, compared to 24% nationally.
  • Negative relationship between per pupil state and local revenues and district poverty rates, after controlling for regional wage variation, economies of scale, population density (poor get less).
  • 46th (of 52) on NAEP 8th Grade Math in 2009. 38th of 41 in 2000. http://nces.ed.gov/nationsreportcard/statecomparisons/
  • 49th (of 52) on NAEP 4th Grade Math in 2009. 35th of 42 in 2000.

So, this is a state where 20% abandon the public system and 82% of those who leave are white and have income twice that of those left in the public system, half of whom are non-white. While the racial gap is large in Mississippi, a much smaller share of Mississippi children abandon the public system and Mississippi is average on the percent of GSP allocated to public education. Mississippi simply lacks the capacity to do better. Louisiana doesn’t even try. And they deserve and award?

Quite honestly, I hadn’t really thought much about Colorado’s chances until today. I was certainly aware of their finalist status and aware of the reform crowd support for their new teacher evaluation legislation. But I hadn’t really reviewed their “indicators.” Here’s my summary of Colorado from earlier today:

Using 2007-08 data, Colorado ranks:

  • 45th in effort (% gross state product spent on schools)
  • 39th in funding level overall
  • 32nd in funding fairness (has a system whereby higher poverty districts have systematically less state and local revenue per pupil than lower poverty districts)

Yes, better than Louisiana, but nothin’ to brag about. And yes, both are marginally better than Round 1 winner Tennessee… but nearly every other state in the nation is.

So, where do these two states fit into those scatterplots I posted earlier today which identified Round 1 and Round 2 winners? Here they are – First, fiscal effort and overall spending level. Both states are very low effort states, and both are relatively low spending states.

And next, effort and funding fairness – or the extent to which funding is allocated in greater amounts to districts with greater needs.

In both cases, Louisiana and Colorado fall toward the lower left hand corner of the plot. Both are very low fiscal effort states. They have the capacity to provide more support for public education – BUT DON’T! Both states are also “regressive” – allocating systematically less funding per pupil to higher need districts, with Louisiana close to a flat distribution. And both are generally low spending despite their capacity to do better.

Improving state data systems – linking those data to teacher preparation institutions in order to impose sanctions on those institutions – banning teachers from obtaining tenure until they can achieve 3 consecutive years of positive value-added scores (error rates alone and year to year fluctuations may make this a low probability event) – and expanding charter schools are not likely to dig these states out of their current position. Doing so will require far greater investment than RttT could ever provide, especially in the case of Louisiana.  In fact, dramatically increasing job risk and career instability for individuals interested in entering teaching without also increasing the reward is likely to have significant negative effects. Unfortunately, it is about as likely that losing RttT will cause these states to reconsider their shortsighted reform agendas as it is that reading this blog post will get them to reconsider the persistent deprivation of their public education systems.

RttT Round 2 – Stuff that Doesn’t Matter!

Unlike many RttT enthusiasts, I have to say that I was pleased to see that Louisiana and Colorado were not among the winners. I’ve written extensively about Louisiana public schools in the past:

https://schoolfinance101.wordpress.com/2009/12/18/disg-race-to-the-top/

Although Colorado doesn’t look as bad as Louisiana on the indicators I often use on this blog, it ain’t pretty.

Using 2007-08 data, Colorado ranks:

  • 45th in effort (% gross state product spent on schools)
  • 39th in funding level overall
  • 32nd in funding fairness (has a system whereby higher poverty districts have systematically less state and local revenue per pupil than lower poverty districts)

Of course, these indicators – which I believe tell us a lot about state education systems – don’t really matter much when it comes to the big race, as I pointed out here:

https://schoolfinance101.wordpress.com/2010/03/29/and-the-rttt-winners-are/

Thankfully, while these indicators of actual effort to finance state school systems and participation rates in those systems didn’t matter in Round 2 either, the picks for Round 2 winners are somewhat – though not entirely – less offensive. I’ve highlighted in yellow with red type any cases where a Round 1 or 2 winner comes in 40th or lower on an indicator – Bottom 10. I’ve indicated in green with blue type, cases where states are in the Top 10. Sadly, there are far more bottom 10 cases than top 10 cases.

I would consider EFFORT and FAIRNESS to be the two key indicators here over which states have greatest control. A poor  state could put up significant effort and still not raise significant funding (Mississippi). The only Round 2 winner state with no “bad” marks and many good ones is Massachusetts. Massachusetts scores well on fairness and overall funding level. Tennessee, from Round 1 is simply a disgrace! North Carolina is perhaps the weakest link in Round 2, along with Florida which ranks poorly but avoids the bottom ten on any measure, and Hawaii which makes the bottom 10 on measures less within the control of the state – coverage. But, Hawaii has inflicted significant damage on its already struggling public schooling system in recent years.

And here are a few interesting two-dimensional views of RttT Round 1 and Round 2 states. First, here’s a two-dimensional view of educational effort and spending level – spending for high poverty districts. The two are reasonably related. Effort explains about half of the variation in spending levels. States like North Carolina and Tennessee are low on effort and low on spending. States like Massachusetts are relatively high on spending, but average on effort.  Rhode Island, Maryland, New York, and Ohio are above average on spending and effort. But spending level doesn’t guarantee that it’s spent – or distributed – fairly across wealthier and poorer districts.

Here’s a look at “fairness” and spending level. New York is high on spending level, but not so good on fairness. In New York State, wealthy districts in Westchester County and Long Island outspend much of the rest of the nation. But, poorer districts including New York City are largely left out, spending significantly less than the affluent suburbs.  Then there are those wonderful states where higher poverty districts have slightly higher revenue per pupil than lower poverty ones, but for the most part – everyone is similarly deprived. These are the “you get nothing!” (reference to Willy Wonka in previous post) states and Tennessee tops that list! Even more depressing are the states where “you get nothing” in general, and you get less if you are poor. Those states include RttT Round 2 standout North Carolina … and Florida sits on the margin of this group. Massachusetts is the “good” standout in this figure.


And here’s effort and coverage – or the share of 6 to 16 year olds attending the public school system.  New York, Maryland and Ohio (on the margin) do poorly on coverage, but have reasonable overall effort. Delaware is the real outlier here… with very low effort and very, very low coverage. Thankfully, none in Round 2 can match Delaware!

Finally, here is the state and local revenue predicted level for high poverty districts, and NAEP mean 2009, grade 4 reading and math scores (combined).  It’s always fun to throw the outcome data in there. And in this case, the RttT Round 1 and Round 2 winners are distributed across the range. Again, Tennessee from Round 1 is the biggest “bad” outlier, but one could say that Massachusetts from Round 2 is a positive counterbalance. Clearly, the demography and economy of these two states differs significantly. My complaint with Tennessee is not that it performs poorly partly because it has a large, low-income population. Rather, my problem with Tennessee, as I’ve noted many previous times is that TN puts up little effort and spends little, and barely spends even that paltry amount equitably. In addition, as I’ve discussed previously, TN has consistently had the lowest outcome standards.

For more on Rhode Island school funding, see: https://schoolfinance101.wordpress.com/2010/07/01/the-gist-twists-rhode-island-school-finance/

For more on Hawaii, see: https://schoolfinance101.wordpress.com/2009/11/06/hawaiis-funding-mess-my-thoughts-on-why/

As I noted on my previous post regarding Round 1 winners:

So then, who cares? or why should we? Many have criticized me for raising these issues, arguing “that’s not the point of RttT.  It’s (RttT)not about equity or adequacy of funding, or how many kids get that funding. That’s old school – stuff of the past – get over it! This…  This is about INNOVATION! And RttT is based on the ‘best’ measures of states’ effort to innovate… to make change… to reach the top!”

My response is that the above indicators measure Essential Pre-Conditions! One cannot expect successful innovation without first meeting these essential preconditions.  If you want to buy the “business-minded” rhetoric of innovation, which I wrote about here , you also need to buy into the reality that the way in which businesses achieve innovation also involves investment in both R&D and production (coupled with monitoring production quality). You can have all of the R&D and quality monitoring systems in the world, but if you go cheap on production and make a crappy product – you haven’t gotten very far.  On average, it does cost more to produce higher quality products.

This also relates to my post on common standards and the capacity to achieve them. It’s great to set high standards, but if don’t allocate the resources to achieve those standards, you haven’t gotten very far! It costs more to achieve high standards than low ones. Tennessee provides a striking example in the maps from this post! (their low spending seems generally sufficient to achieve their even lower outcome standards!)

That in mind, should states automatically be disqualified from RttT for doing so poorly on these Essential Preconditions? Perhaps not. After all, these are states which may need to race to the top more than others (assuming the proposed RttT strategies actually have anything to do with improving schools). But, for states doing so poorly on key indicators like effort and overall resources, or even the share of kids using the public school system, those states should at least have to explain themselves – and show how they will do their part to rectify these concerns.

For a video version of my comments on the big race, see: