Rightsize this! When simple, ignorant solutions & simulations just don’t cut it

Recently, TB Fordham Institute released a report by AIR researcher Michael Hansen on “Rightsizing” the classroom. Hansen based his analysis on data from the state of North Carolina, using distributions of teacher value added scores and class sizes to derive conclusions about how “great” teachers could be given larger classes, thus reducing students exposed to “bad” teachers, leading to overall benefits in terms of student outcomes. This dreadfully oversimplified, a-contextual (even taken out of the constraints of its actual context) extrapolation has since made the rounds across reformy outlets.

The solution to all of our woes is simple and elegant. Just follow these steps.

• Step 1: Identify “really great” teachers (using your best VAM or SGP) who happen to be currently teaching inefficiently small classes of 14 to 17 students.
• Step 2: Re-assign to those “really great” teachers another 12 or so students, because whatever losses might occur in relation to increased class size, the benefits of the “really great” teacher will far outweigh those losses.
• Step 3: Enter underpants Gnomes.
• Step 4. Test Score Awesomeness!

The research assumption based on the North Carolina data is that the negative effects of increased class size are small, especially for 8th graders.

For most students above the third grade, the evidence points to at most a small class-size effect, if any at all.15 (Using the North Carolina data, I likewise estimate small class-size effects in fifth and eighth grades.)16 Thus in effect, it would take an increase of at least ten to twenty additional students in a good teacher’s class to dilute his productivity to that of an average teacher.17 Put another way, assigning a few extra students to the class of an effective teacher can translate to big gains for these students, while making only very small reductions in that teacher’s performance for everyone else in the class. (page 9)

Further, that the impact of “great teachers” is far more important. Thus, as the report puts it:

Intensively reallocating eighth-grade students—so that the most effective teachers have up to twelve more pupils than the average classroom—may produce gains equivalent to adding roughly two-and-a-half extra weeks of school (see figure ES-1).

While this is a fun/playful thought exercise… simulation, etc… much like the Chetty study extrapolation of the great teacher increasing a classroom full of 3rd grader’s lifetime income by $250k, this simulation ignores so many layers of reality that it’s just mind boggling.

While it certainly makes sense that we’d want to be able to assign more students to our “best” teachers (heck, why would we want to have anything but good teachers on our staff?), the practical constraints to implementing this elegant and oh-so-obvious solution are many:

1. successful implementation requires that within our school we can actually identify with some consistency, those teachers who are measurably more effective? (and that we have some of each…??)
2. that those “great” teachers have small enough class sizes for us to add those students without significant consequence and within room size/space constraints?
3. and that their effectiveness is not particularly sensitive to the size of classes they’ve been teaching (which likely varies across teachers)
4. that adding 12 students to each of 5 or 6 sections of daily workload for a teacher will not have some cumulative negative effect (on grading/quality of feedback they provide/retention of “great” teachers). That’s 60 to 72 more students. Individual classes are not the only relevant unit of analysis here! Total workload matters. At even 10 minutes of grading per week for each student, we’ve added 10+ hours of weekly work.

What’s been fun to follow about this report is the assertion that it is somehow broadly applicable to any/all policy settings.

Let’s consider above constraints in the context of New York City.

First, can we figure out who those “great” teachers in 7th/8th grade are… even in Math where value-added scores tend to be more stable, and in the school with the largest number of them in the value-added data released a few years back.

Here are the 8th grade math teachers and 7th grade math teachers with their year over year value-added percentiles. For example, we see that in 2008-09 Dorothy is below average (left of vertical line) but in 2009-10, Dorothy is above average. The same is true  for Natalie. Donna is above average both years and two (overlapping) are below average both years. Looking back an additional year, we only have one carry over teacher, Dorothy, who is above average again.

Donna does show up for 7th grade (below), and is above average there as well, but only average back in 2005-06. Otherwise, a) we don’t have that many teachers who  even persist in the school from year to year, and b) those who do have percentile ranks that jump all over the place.

So, it’s not really so easy to find those persistently excellent teachers.

And then what of that class size issue? Do we really think that Donna is going to have an inefficiently small class into which we can shove 12 more students?

The likelihood of that occurring in New York City is not great. Here are school average class sizes in 2010, 2011 and 2012.

Here’s a statewide look at the percent of classes already over certain thresholds.

In higher poverty settings, most 8th grade class sizes already exceed 23 students and most in New York City far exceed that. It would be utterly foolish to extrapolate the assertion of minimal downside to increasing an NYC 8th grade math class from, say, 32 up to 44 students (if the room could even hold them).

One might assert that affluent suburban Westchester and Long Island districts with much smaller average class sizes should give more serious consideration to this proposal, that is, if they are a) willing to accept the assertion that they have both “bad” and “good” teachers and b) that parents in their districts are really willing to permit such experimentation with their children? I remain unconvinced.

As for leading private independent schools which continue to use small class size as a major selling point (& differentiator from public districts), I’m currently pondering the construction of the double-decker Harkness table, to accommodate 12 students sitting on the backs of 12 others.  This will be a disruptive innovation like no other!