Preliminary evaluations of 1:1 initiatives
Why do we always come back to questions related to data? I admit that I do this and I have an uneasy feeling if I cannot. I have no easy explanation for this self-imposed requirement. I was trained to think in this fashion and it is part of my method for thinking critically.
Here is one perspective to consider. You are probably familiar with those lists of recommendations educators used to create when preparing students to critically evaluate web content.
Those lists typically contained items such as:
In addition to such guidelines, my personal internal list also contains:
In my opinion, some of us must take this approach and others should at least be supportive. In addition, I would suggest that it is important for stakeholders to invest in the evaluation of demonstration projects. By stakeholders, I mean the federal and state government and private foundations. Resources - time and money - are always so limited we cannot afford to having educators chasing one idea after another without first investing in studies that give direction to new initiatives. There are many perspectives and personal values so disagreements are inevitable. While I do not assume measurement and evaluation practices will end controversy, I at least feel more comfortable when the appeal to data is part of the process.
Early Returns
Some states have instituted major 1:1 initiatives and experiences of these early adopters might offer guidance to those states or school districts willing to follow. I like state-level approaches over demonstration projects because with larger projects you get a better feel for "what would work in most places".
Initiatives in Texas and Maine have received a lot of attention. There is a good deal of anecdotal information concerning the trials and tribulations of such major initiatives that may interest administrators charged with guiding such large implementations. Our interest is more focused on what has been learned regarding student achievement.
Let me begin my comments on the research associated with these programs by making an observation. Applied research on this scale is messy and must contend with the practical interests of schools and educators who must deal with agendas far different from the priorities of researchers. Those of us who look for research in "high end" research journals typically do not encounter articles offering concise conclusions on such topics as whether 1:1 initiatives improve student academic performance. The methodologies that are acceptable to such journals are seldom used in the situations that interest us or the results generated in such situations are typically mixed or unpersuasive. We must patch together our conclusions from what might be called technical reports, i.e., evaluation documents prepared to meet the expectations of funding agencies, or from secondary sources authored by other individuals who have read such technical reports. Some may be offended by this generalization, but I think it is at least close to the most common state of affairs.
I have attempted to review what I can find resulting from evaluations of the Texas and Maine initiatives. I identify my sources at the conclusion of this section and will revise what I have written here if I encounter new information.
1:1 initiatives benefit writing skills
Before researching this segment, I was aware of positive claims made regarding improved writing performance among students studied as part of the Maine 1:1 initiative. Reference is made to writing as the academic area showing consistent positive gains in the prestigious journal Science (Zucker & Light, 2009). The Science article cites as the source a review (Penuel, 2006) which includes a few published studies but also the technical reports I mention previously. A careful reading of Penuel also indicates that the author notes the methodological shortcomings of the 4 studies finding positive benefits in the area of writing (mostly the lack of pretests). My purpose here is to note these things in anticipation of others who will claim that there is no data from studies enforcing a rigorous methodology demonstrating benefits for such initiatives.
Performance of 7th and 8th grade students on the Maine Educational Assessment writing assessment from 2000 (before implementation) was compared with performance in 2005 (after beginning the laptop initiative) demonstrating a 1/3 standard deviation improvement in performance. This improvement can be translated as the average student in 2005 scored at the level of 67th percentile from 2000 (Silvernail & Gritter, 2007). The authors do note that overall performance on Maine's Educational Assessment did not improve. The authors speculate that writing would be an area most likely to show the advantage of the laptop initiative arguing in combination that other 21st century skills were likely not evaluated by the state method of assessment and also that it takes some time to learn to take advantage of the new tools.
The idea that writing would be an area likely to be impacted by 1:1 initiatives is certainly consistent with the logical case we have attempted to offer here. Writing and other forms of "authoring to learn" have broad applicability across content areas and are within the scope of learning tasks likely to be assigned by many teachers. It makes sense that an initiative that makes it more convenient for students to research and engage in authoring tasks would impact writing proficiency.
Some links to descriptions of the Maine initiative are included at the end of this segment.
The Texas immersion study
The Texas immersion study is interesting because of the methodology. While the results regarding achievement were mixed, the approach involved an attempt to contrast what went on in immersion and control schools.
The hypothetical impact model by which the commitment to the heavier investment in equipment was to impact student achievement was proposed. It is useful to consider this model because it makes assumptions more obvious.
1) Students are provided convenient access to technology
2) With the availability of more resources, it is likely teachers will become more proficient and students will experience more use of technology in classes
3) Greater student use of technology will generate changes in learning experiences in school and out
4) Changed learning experiences will result in achievement advantages
I do not mean to imply that moving from step 1 to 2, etc. occur automatically simply as a function of meeting the conditions of step 1. Faculty development and support have been consistently identified as an important requirement in analysis of 1:1 initiatives (Penuel, 2006). Perhaps a better way of thinking about the flow here is to understand each preceding step as providing the opportunity for the following step.
The Texas immersion (their term) study is methodologically quite complex. There are multiple cohorts (a new group of students started the process each year for multiple years) and multiple content areas were being assessed, but not necessarily each year. The treatment and control schools were not exactly identical - the treatment schools volunteered and then control school were matched as closely as possible. Results were not always consistent for the cohorts.
I read the results as indicating some limited achievement benefits. The findings demonstrated little difference in reading, an advantage for immersion students in the area of mathematics, and a limited disadvantage for immersion students in the area of writing. Note that the effect sizes were small and the writing outcome contradicts the findings from the Maine study. A couple of other findings caught my attention. First, home use of technology was a significant predictor of math and reading achievement. Again, this is an outcome with multiple possible interpretations. Perhaps, this is really not that exciting and is simply a way to identify differences in motivation or some other individual difference predicting higher achievement. However, perhaps, this finding demonstrates the advantage of expanding opportunities to learn. Finally, there appeared to be what I would describe as slippage in the commitment to project goals. Classroom laptop use and home laptop use declined somewhat over time. The researchers noted that teachers may revert to more traditional activities when given the opportunity.
So, perhaps you thought my summary would be more optimisitic and encouraging. I can only attempt to summarize what the research appears to suggest. We might note that the Texas immersion study did not necessarily focus on participatory web applications. It is a major evaluation effort that takes as a starting point the provision of hardware and related faculty support. There are so many complex issues here and it would be inappropriate to claim I have the answers or even can suggest the most important questions. For me, the down turn in use after the end of the formal project, was one of the more interesting phenomena demonstrated. The authors of the study commented on factors felt to influence continuation. Administrator support and teacher buy-in were essential suggesting that a state-wide model may not be feasible. Teacher beliefs about how learning is best accomplished also appeared to be significant. In this conclusion, the findings of the Texas immersion study seem very reminiscent of an earlier study (Windschitl & Sahl, 2002) concluding that access to new opportunities will lead to change but only among teachers with specific belief systems about teaching and learning.
References:
Penuel, W.R. (2006). Implementation and effects of one-to-one computing initiatives. Journal of Research on Technology in Education, 38, 329-348.
Windschitl, M. & Sahl, K. (2002). Tracing teachers' use of technology in a laptop computer school: The interplay of teachers beliefs, social dynamics, and institutional culture. American Educational Research Journal, 39, 165-205.
Zucker, A.A. & Light, D. (2009). Laptop programs for students. Science, 323, 82-85.
Silvernail, D.L. & Gritter, A.K. (2007) Maine's Middle School Laptop Program: Creating Better Writers - Research Brief http://usm.maine.edu/cepare/impact_on_student_writing_brief.pdf
Texas Center for Educational Research (2009). eTxTiP - Evaluation of the Texas Technology Immersion Pilot: Final outcomes for a four-year study. Available - http://www.tcer.org/research/etxtip/index.aspx