Tag: Research

Quest For Truth

One of the benefits of blogging is that it is possible to follow-up on previous posts. I suppose this is similar to a news program doing a kind of “where are they now” segment.

A year or so ago, I wrote an entry that described the Department of Education’s effort to do a better job of evaluating the effectiveness of instructional software. This post described a research program to be initiated in 2004-2005 to evaluate 16 commercial reading and mathematics programs available to K-12 schools. Mathematica Policy Research and SRI were awarded the grant providing the resources to finally conduct quality research. Conducting educational research in a way that avoids confoundings and methodological flaws is extremely difficult so I have been attempting to follow the success of this project.

I have been able to locate a brief update provided by Mathematica in mid 2004. This update promises:

During summer and fall 2005, the team will analyze the data and prepare a report to be submitted to Congress. Consideration is being given to adding another year of data collection with the same teachers and new students. These data would be gathered in 2005-2006.

I am unable to locate a report resulting from this research. Online searches generate plenty of descriptions and announcements by the companies involved, but I have been unable to locate study results. I would think the initial results described above should be available. If you are aware of published results, I would appreciate an email.

Disregard my request for information. My emails generated a reply:

I expect ED will release our report on findings within six weeks or
thereabouts. When it is released, it will be posted at the Institute of
Education Sciences web site and also at edtech.mathematica-mpr.com.

Mark Dynarski

Stay tuned.

Loading

A Picture About Several Thousand Words

One of my general frustrations with the individuals I consider the movers and shakers in the educ blog community is that they seldom collect or rely on research and data. Innovators trained in the tradition I come from have vision and beliefs just like anyone else, but we start to feel guilty when we promote practices without the benefit of some form of evidence. I hope this “built-in” guilt mechanism is a hedge against becoming too caught up in the thrill of being a promoter.

I decided when there was a connection to the general themes of this blog I would make an effort to share data I collect. Here is a simple finding that may be relevent to those interested in podcasting educational content.

A topic I study as a researcher is the impact of providing beginning college students “lecture supplements.” I consider such questions as –

  • Will students make voluntary use of lecture supplements?
  • What are the characteristics of students who use lecture supplements?
  • Is there a connection between the use of lecture supplements and attendance?
  • Etc.

I have been evaluating student use of lecture outlines (available before lectures) and lecture summaries for some time. For the past couple of years, I have added audio.

The following chart (a picture about student use of lecture resources – several thousand words) indicates the % of outlines, summaries, and audio files (from left to right) “downloaded” by students.

Yahoo Time Capsule

An impression – within the context of text-based resources, students are not that interested in audio content. Why – I am guessing the efficiency of lecture summaries is much greater and lecture summaries prepared by a qualified individual have been “preprocessed” and are more helpful as a supplement.

So – I still think there may be some specific opportunities for “podcast” content, but the DATA I have regarding represenations of entire lectures suggest that college students would prefer a quality text summary. When students are investing their time, students appear to prefer well written and complete information summaries. I am assuming there would be specific exceptions that would involve the unique capabilities of audio – language development, music, bird calls, etc. Perhaps a collaborative set of master notes generates by students via a wiki should be the focus of our attention.

Loading

The Cuban Interview

EdTechLive provides the audio of interviews with prominent individuals in the field. A recent addition is a session with Larry Cuban (author of Oversold and Underused – the PDF of which is also available from the site) and this conversation has generated some buzz in the blogs I follow. For those who have followed some of the core issues, most of the arguments in the interview are pretty much a rehash of themes from previous publications. Cuban’s 2001 book investigated a wide variety of issues concerning the use of technology in schools. Two of the major themes are apparent from the title – schools are pressued to purchase technology and the amount of time students spend using technology is very limited. The lack of reliable evidence that technology has a meaningful impact on achievement, limited opportunities for students to use the resources that are purchased, and the cost of the purchases are blended into a rather pessimistic picture. While pessimistic, this book and a journal article (much shorter – American Educational Research Journal, 38(4), 813–834) should be read by those interested in this field. The interview provides a nice overview and may generate in some the motivation necessary to locate the book in the library or online.

Some of these same issues have been raised again by the recent Wall Street Journal article criticizing 1:1 laptop initiatives (accessible without cost from other services). This topic comes up in the interview.

Addressing and interpreting all of the concerns raised in the Cuban documents would require another book. Many of the observations (e.g., students spend little time with technology, teachers with adequate personal technology skills fail to avail their students of the same tool opportunities) have been made by others and the causes and related recommendations vary. The issue of “proof” in education is troublesome to all of us attempting to make good decisions about the field and also attempting to keep our personal interests from clouding the practices we advocate.

I would suggest that a careful reading of the research literature would bring into question several “costly” and time consuming educational practices (one of which Dr. Cuban seemed to mention in a positive way in this interview – general liberal arts education). One of may favorite examples (perhaps because of my personal academic training) is the science laboratory. The cost of laboratory experiences in introductory chemistry is extremely expensive (partly because the safety requirements that are imposed) and the demonstrated contribution to the understanding of the content taught in such courses would be hard to justify based on the literature I have read. I think some of the same factors that Cuban mentions in his book apply to the way we think about science laboratories. Instead of pressures exerted by parents to involve their children with computers and the promotions of companies interested in selling technology to schools, pressure comes from other directions. The assumptions regarding what constitutes sound scientific training promoted by organizations such as the NSF and the politics asssuming that general math and science preparation are somehow essential to international competitiveness prompt schools to spend money in certain ways. I am not against spending the money – my point is that finding costly academic ventures that lack a strong empirical basis is not that difficult.

Listen to the interview, parse the various arguments, and determine for yourself how you would interpret many of the observations that are provided.

Blogged with Flock

Loading

The Case For Direct Instruction

I have a personal interest in the benefits of student multimedia authoring (e.g., web design, blogging, video production). The idea is that students author products associated with their content area study and I assume they benefit from such activity. To me, this idea is not a huge leap of faith. Educators have been promoting and researchers have been evaluating writing to learn and writing across the curriculum for years. Large scale multimedia projects might be viewed as an alternative to a term paper. Smaller scale projects are more the equivalent of other less utilized writing to learn tasks.

Our interest in student authoring is not unique – many interested in K-12 technology applications have encouraged multimedia projects. Often, this approach has been identified as constructivist. The difficulty I have with this label is that it appears to mean different things to different people. When I think of a learning theory, I assume the theory is an effort to describe how learning happens. To me, the idea that a theory is about a technique is misguided. It seems possible that different technqiues might result in better or poorer learning, but inappropriate to label some technigues as constructivist and some not. So I see theories as constructivist (i.e., learning requires the integration of what is known with new experiences) or not, but not experiences. My professional training involved a focus on information processing theory – I understand info processing theory as an attempt to explain some of the details of constructivism (which to me lacks attention to the specifics of learning).

Anyway, my wife sent me a reference to a blog entry summarizing an Educational Psychologist article (Kirschner, Sweller, Clark – 2006) critical of constructivism, discovery learning, experiential learning, etc. A draft of the Kirschner, et al. article can be found using a link from the blog (the journal publication date listed on the draft is incorrect and should be 41(2) – I suppose the journal got a little behind). The article argues that learners inexperienced in an area of study do poorly when allowed too much freedom in their approach to learning (hence the references to some of the very early critics of discovery learniong – e.g., Ausubel). Perhaps recognition of this reality is why some constructivists are careful to recognize the need for scaffolding. This is a good analysis and understanding such issues is important.Some of these topics remind me of political controversies – i.e., your approach is boring and meaningless, students can’t learn about topics for which they have no background.Comments on the Kirschner, et al paper seem to be rippling through the educational blogs. The day after the original version of this post the Connectivism blog commented on the same article. If the distinction between constructionism and constructivism is of interest to you the connectivism post has much more detail (too many isms for me to explain).

What I like about Kirschner’s approach is the connection (pardon my use of this term) with the empirical literature. Those taking a contrasting position seems to rely on philosophy, anecdote, and personal reflection. The Kirschner article, Mayer’s Ed Psychologist article, and a historical series of similar complaints (e.g., Ausubel) seem to beg for engagement in a “scientific” debate. If no data exists to counter their concerns, perhaps the first step would be admit this problem and then at least outline the types of demonstrations and the data types that could be used to argue for a different position.

Loading

Dev Psych – Internet and Achievement

Several weeks ago, I reported on an issue of Developmental Psychology focused on children and the Internet. The article I highlighted suggested a possible relationship between Internet Use and Achievement. In response to my post, a comment was submitted noting the difficulty of properly attributing a causal relationship (it was reported that use led to improved achievement) based on nonmanipulated environments (i.e., correlational research). In other words, it would seem just as possible that “bright” kids read more online as that reading more online leads to changes in reading skill.

I finally had time to read the paper more carefully (Jackson, et al, 2006) and the causality issue favoring the influence of online experience on achievement is more solid than might be assumed from my original sketchy report.

One technique developmental researchers sometimes employ to address the “causality issue” when variables cannot be manipulated is to take advantage of the passage of time (longitudinal research). If experiences and characteristics (Internet use and reading skill) can be measured at multiple points in time and the correlation of the experience to later measurement of the skill is greater than the correlation of the skill to a later measurement of the experience, the argument for use causing changes in achievement is stronger. This is what the paper in Dev. Psych reports.

I think the research on out of school use of the Internet is potentially of great significance because the experiences of children vary greatly and potential consequences may be easier to observe. Schools would not in good conscience allow the differences in experience that exist in the “real world” and are thus in a weaker position to demonstrate the benefits of technology. It is somewhat ironic that educators must on one hand demonstrate that value of the methods they employ and on the other avoid not providing all students pretty much the same “opportunities”. If methodological challenges can be addressed, the use of these naturalistic data may be quite convincing.

Loading