AERA 2007

We spent the last week in Chicago and in the journey getting from here to there and back. The event was the American Educational Research Association convention (2007). The one presentation I attended that may be of interest to those who read this blog concerned the evaluation of math and reading software I just described last week. A presenter from Mathematica and a discussant from Stanford considered the results of the first phase of the study. Here are a couple of comments that I found helpful.

Even though the study involved many schools, teachers, and students, treating school and teacher as units of analysis required, based on a power analysis, that the researchers expect an effect size of .15 to regard an effect as significant. In this situation, it was not possible to evaluate the impact of the 16 individual instructional packages that were used and the overall effect of reading or math software did not achieve statistical significance.

Based on tracking functions present in some of the software programs, the researchers estimated that use of the software replaced approximately 10% of traditional instruction. The discussant noted that for approximately $25 per seat (the average cost of the software) it might be argued that 10% of teacher time was freed to attend to the needs of individual students. This “spin” on the results was kind of interesting and at first I thought this to be a positive statement and quite optimistic given the very guarded approach taken by the researchers. However, while potentially true, it would also seem that this flexible time must not have been used productively or achievement gains would have been generated.

The data from the second phase of the study is still under review. The second phase will allow a comment on individual software packages.

Representatives from a couple of software companies were in the audience and noted that the software was implemented with the minimal amount of technical support and inservice preparation. The presenters accepted this description, but countered that this was the level of support that schools normally purchased.

The presenters were careful to stay away from speculating how the results of the research would be interpreted by politicians and policy analysts. The presenters argued that the results could be spun in different ways.

We did have the opportunity for a little recreation. The Sunday before the conference we were able to watch the White Sox and Twins. It was cold, but Santana was pitching for Minnesota and we are becoming big Twins fans. Twins won.

Blogged with Flock