Every time I have access to state by state education data, I look up North Dakota (the state in which I am employed). I expect to see that North Dakota has done well. My confidence is not based in an assumption of high quality, but because I assume North Dakota does not have to face some of the challenges that face educators in other states. This perspective actually prompted part of my reaction to the implications in comments I address here.
For example, this post regarding reading performance indicates North Dakota (and Iowa) scored toward the bottom. I started to think about this. My biases may be at work here, but I think first impressions may be misleading.
Careful reading indicates the chart shows gains in reading performance among students receiving free or reduced lunch (used as a measure of poverty). Two issues to consider:
- This is not a chart showing levels of performance – gain scores are a difference score.
- This is a chart based on a subset of all students and the size of this subset is not constant by state.
Researchers know that change scores are very difficult to interpret and I think this could be the case here. A key issue in this case is likely what the scores were before the intervention (NCLB). If the scores reflect a “ceiling effect” improvement would imply something very different from a situation in which the beginning scores were originally low (and little changed occurred). I think it likely there is some of each possibility at work here and it would be important to know which low improvement states within this group fall into the low or high start category.
The second possibility is that a state with lower population and a lower proportion of students from low income families may find more it challenging to address the needs of these students. I do not know that this is the case, but I believe it to be the case with other educational challenges – e.g., ESL students.
It is possible I am completely wrong and North Dakota is doing a poor job of instruction. However, critical thinking is important any time data are presented and causality is difficult to tie down in correlational rather than manipulative research.