Why you can’t compare simple achievement gaps across states! So don’t!

Consider this post the second in my series of basic data issues in education policy analysis.

This is a topic on which I’ve written numerous previous posts. In most previous posts I’ve focused specifically on the issue of problems with poverty measurement across contexts and how those problems lead to common misinterpretations of achievement gaps. For example, if we simply determine achievement gaps by taking the average test scores of children above and below some arbitrary income threshold, like those qualifying or not for the federally subsidized school lunch program, any comparisons we make across states will be severely compromised by the fact that a) the income threshold we use may provide very different quality of life from Texas to New Jersey and b) the average incomes and quality of life of those above that threshold versus those below it may be totally different in New Jersey than in Texas.

For example, the histogram below presents the New Jersey and Texas poverty income distributions for families of children between the ages of 5 and 17. The Poverty Index is the ratio of family income to the poverty income level (which is fixed national). The histograms are generated using 2011 American Community Survey data extracted from http://www.ipums.org (one of my favorite sites!). The vertical line is set at 185% poverty, or the federal “reduced price lunch” threshold, a common threshold used in comparing low income to non-low income student achievement gaps.

As we can see, the income distribution in New Jersey is simply higher than that of Texas. It’s also more dispersed. And, as it turns out the ratio of income for those above, versus those below the 185% threshold is much greater in New Jersey.

In New Jersey, the income ratio is about 6:1 for those above versus those below the 185% threshold. In Texas, the ratio is about 4.5:1. And that matters when comparing achievement gaps!

Figure 1. Poverty Income Distributions for Texas and New Jersey


Figure 2 illustrates the relationship between the income ratios for non-low income to low–income children’s families and outcome gaps for NAEP grade 4 Math in 2011. Put simply, states with larger income gaps also have larger outcome gaps. States with the largest income gaps, like Connecticut, have particularly large outcome gaps. Clearly, it would be inappropriate to compare directly, the income achievement gap of Idaho, for example to New Jersey. New Jersey does have a larger outcome gap. But New Jersey also has a larger income gap. Both states fall below the trendline, indicating (that if we assume the relationship to be linear) that their outcome gaps are both smaller than expected, and in fact, quite comparable.

Figure 2: Relating Income Gaps to Outcome Gaps (NAEP Grade 4 Math)




Table 1 summarizes the correlations between income and outcome gaps for each NAEP math and reading assessment, over several years.

Table 1. Correlations between Income and Outcome GapsSlide3

It stands to reason that if the income differences between low income and non-low income families affect the income achievement gaps, then so too would the income differences between racial groups affect the outcome differences between racial groups. Therefore, it is equally illogical to compare directly racial achievement gaps across states.

Figure 3a shows the black and white family income distributions in Texas and 3b shows the income distributions in Connecticut.

In Texas, the ratio of family income for white families to black families is about 1.5 to 1.

In Connecticut, that ratio is over 2.3:1.

Figure 3a. Black and White Income Distributions in Texas


Figure 3b. Black and White Income Distributions in Connecticut


Thus, as expected, Figure 4 shows that the income gaps between black and white families are quite strongly correlated with the outcome gaps of their children in Math (Grade 4).

Figure 4. Income Gaps between Black and White Students and Outcome Gaps


Table 2 shows the correlations between black-white family income gaps and black-white child outcome gaps for NAEP assessments since 2000.

Table 2. Correlations between Black-White Income Gaps and Black-White Test Score GapsSlide7

Why is this important? Well, it’s important because state officials and data naïve education reporters love to make a big deal about which states have the biggest achievement gaps and by extension assert that the primary reason for these gaps is state lack of attention to the gaps in policy.

Connecticut reporters and politicos love to point to that state’s “biggest in the nation” achievement gap, with absolutely no cognizance of the fact that their achievement gaps, both income and race related, are driven substantially by the vast income disparity of the state. That said, Connecticut consistently shows larger gaps than would even be expected for its level of income disparity.

Black-white achievement gaps are similarly a hot topic in Wisconsin, but will little acknowledgement that Wisconsin also has the largest income gap (other than DC) between the families of black and white children.

New Jersey officials love to downplay the state’s high average performance by lambasting defenders of public schools with evidence of largest in the nation achievement gaps.

“The dissonance in that is if you get beneath the numbers, beneath the aggregates, you’ll see that we have one of the largest achievement gaps in the nation.” (former Commissioner Christopher Cerf)

Years ago, politicos and education writers might have argued that these gaps persist because of funding and related resource gaps. Nowadays, the same groups might argue that these gaps persist because employment protections for “bad teachers” in high poverty, high minority concentration schools, and that where the gaps are bigger, those protections must be somehow most responsible.

But these assertions – both the old and the new – presume that comparisons of achievement gaps, either by race or income, between states are valid. That is, they validly reflect policy/practice differences across states and not some other factor.

Quite simply, as most commonly measured, they do not. They largely reflect differences in income distributions across states, a nuance I suspect will continue to be overlooked in public discourse and the media. But one can hope.



15 thoughts on “Why you can’t compare simple achievement gaps across states! So don’t!

    1. I think I recall seeing these posts of yours. I have actually done similar comparisons with TUDA in the past. The point you raise does relate to a different issue – alignment of the data & spaces/schools w/poverty measures.

  1. To spell it out a little bit more, the minimum possible income is about the same in every state. So, it’s not a surprise that states with more higher income people also have a larger gap. There’s no upper bound on the scale, but there is a lower bound, and the lower bound doesn’t, in our economy, increase just because of the upper incomes increasing.

    However, states with a higher median income are also usually higher cost states – real estate costs more, and that added demand percolates into every expense from food to child care in addition to rents. So, lower income people in those states would generally have less cash flow and fewer discretionary resources.

      1. Could you estimate the “livable” income (income necessary to afford x, y, and z) for each metro area and then transform into z scores centered on the livable wage? You would still not have a normal distribution, but might capture this better than what is typically used. Might make comparisons more reasonable as well. Just a thought off the top of my head.

      2. working on this type of thing. Lori Taylor, Jay Chambers, Jesse Levin and I have a piece in EFP which applies adjusted poverty estimates. I’m trying to figure out how to better blend that with outcome gap measurement.

  2. I’m so glad that common sense is backed by properly constructed statistics. It’s definitely not one size fits all, and anyone who has taught at various types of schools throughout their career (even within a given radius) knows this intuitively via their experiences.

  3. Thanks, Bruce.
    Re NY, am I correct in reading this to indicate that, despite large funding inequities (F in distribution), NY manages to be on the favorable side of outcome gaps compared to it’s income gaps?
    Best, Molly
    (MA and NJ seem to do that, too, but they’ve had more equitable funding over the years.)

    Molly A. Hunter, Esq.

    Standing Up for Public School Children
    (973) 624-1815, x 19
    http://www.educationjustice.org CONFIDENTIALITY NOTICE
    The information in this email may be confidential and /or privileged. This email is intended to be viewed only by the individual or organization named above. If you are not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any review, dissemination or copying of this email and its attachments, if any, or the information contained herein, is prohibited. If you have received this email in error, please immediately notify the sender by return email and delete this email from your system. Thank you.
    >>> School Finance 101 4/16/2014 11:57 AM >>>

    schoolfinance101 posted: “Consider this post the second in my series of basic data issues in education policy analysis. This is a topic on which I’ve written numerous previous posts. In most previous posts I’ve focused specifically on the issue of problems with poverty measurem”

    1. While it certainly appears that NY’s achievement gap is “smaller than would be expected given their income gap,” it’s still pretty hard to figure out the contributing forces. Demographically (and geographically) NY is just so different from, say, MA, CT and NJ. I suspect that NYC plays a unique role in influencing the income gap measures… and perhaps the distribution of outcome gaps as well. So, I’m not sure I’d make too much of this one snapshot which considers only math grade 4 in 2011 and income gaps in 2011.

      1. I might be wrong, but doesn’t NYS have the largest diversity population in the country? Or at least one of the largest. And even though there are pockets of poor throughout the state, there are also affluent segments of the population, many in the suburbs, who bring up the scores. This in addition to the unique circumstances of NYC. NJ is similarly situated.

        Of course, this is just an attuitive response. I just wonder if the statistical data backs it up.

        Then the next question, is it more difficult to get valid data when comparing states like these with more homogeneously populated states like Vermont?

      2. It’s just more difficult to make any valid comparisons between such geographically and demographically different states. We might, however, better compare the area from just north of Albany to the Canadian boarder to Vermont as a whole.

      3. Just leave out Buffalo and perhaps Rochester and Syracuse (the cities, not the surrounding areas).

      4. yep… actually just the area east of Syracuse, above I-90… excluding the cities (Utica, Albany-Schenectady-Troy)

Comments are closed.