Home

Evaluating ALN: What Works, Who's Learning?

by Sloan-C

This forum consists of the following threads:

• What Is ALN
• What Indicators Shall We Use?
• What to Evaluate
• Impacts of Anytime/Anyplace Learning
• How to Evaluate: Methods and Strategies

I. What Is ALN?

Several approaches to describing ALN arose in our discussions:

Interaction mode

John Sener noted Frank Mayadas' description of ALN in terms of its mix of self-study (e.g., books, broadcast TV); synchronous interaction (e.g., interactive TV, audio conferencing); and asynchronous interaction (e.g., groupware and Internet bulletin boards).

Olin Campbell described ALN as a people network for learning that is largely asynchronous. It combines self-study with substantial, rapid, asynchronous interactivity with others. In ALN learners use computer and communications technologies to work with remote learning resources, including coaches and other learners, but without the requirement to be online at the same time.

By this definition, a web-based workshop that requires frequent online conferencing and collaboration with others is ALN. So is a text- or computer-based training course that requires learners to use email to discuss assignments with each other and with the coach. ALN also encompasses a proctored examination at a specified time and place, or occasional synchronous chat or lab sessions for near-campus learners, or an in-person kickoff meeting. Also by this definition, distance education based primarily on a synchronous audio or video presentation or conference is not ALN because these constantly require learners and instructors to be available at the same time. A videotaped course or mail-based correspondence course or computer-based training is not ALN because these do not include substantial and rapid interactivity with others, even though the learner might mail in a paper or test and receive a reply days later.

Distance from campus

Mayadas also describes ALN in terms of on-campus, near campus (e.g., learners come to campus for labs), and far from campus (learners may never meet in one place). Sener notes that the mix of delivery modes for a particular course may be itself a range, depending on how much flexibility learners have in choosing the degree and type of interaction they use to complete their coursework.

Delivery mode

ALN includes a technology-mediated interactive or collaborative component, plus substantial technology-mediated content resources, and activities and resources that are structured to support learning (not just communication). Sener articulated this view, pointing out that ALN requires more than just communication of information like announcements and course syllabi.

Management

Another outlook is management oriented - we need to consider the payoff from institutional investment in technology, and learner investment in ALN courses, as part of Steve Ehrmann's Triple Challenge of access, quality, and costs framework. Jaishree Odin added this perspective. Chet Skinner also observed that our focus should to be on what the learners want and are looking for, since ALN can be viewed as a retail business that will benefit from high volumes.

II. What Indicators Shall We Use?

The discussion began with Ehrmann's Triple Challenge. Frank Mayadas at Sloan has suggested that we also consider learner and faculty satisfaction. Olin Campbell suggested that learners would want to know about the organization that's providing the learning opportunities -- "What's the fit between me, the school, and my prospects after finishing the school?" If they are looking for a specific course they may also want course ratings. Campbell then offered a draft set of indicators related to a course. Other indicators relate to a school and a program within the school.

John Bourne observed that acquiring and validating this information could be difficult, especially if an unbiased body of examiners did the work vs. potentially biased respondents at the institutions. He wondered if graduates would be more reliable sources of information. Is it feasible for an institution to provide detailed information about each course when different instructors teach it using different methods and materials, and might want to keep their instructional practices private? Jaishree Odin raised this issue and noted that ratings might lead to standardization and stifle the creativity of instructors who want to experiment with new materials in their teaching.

One of the indicators is course workload (hours of study required). Course providers only infrequently tell the prospective learner the workload involved in taking a course. John Gundry observed that this might be expressed as typical hours per week required, times the number of weeks in the course. He could not see how a learner is expected to work out whether the course workload will fit with their life without this information. John Sener observed that even when faculty put this information on the syllabus, learners only see it after registering for the course.

There are potential problems with putting workload information in the course catalog. One is fear of reducing enrollments in schools where enrollments drive revenue (and faculty positions). ALN courses would stand out in a catalog where classroom courses did not include this information, and ALN courses showed 15-20 hours of work per week - the equivalent of a part time job. On the other hand, Sener observed that many of those who sign up in a community college for ALN courses do so because they are working, and need to know the course workload to plan their time. In such cases, publishing the course workloads could be a marketing tool.

III. What to Evaluate

Kirkpatrick (1994) proposes 4 levels of evaluation:

1. Learner reaction (e.g., end of course evaluation sheets)
2. Learning (e.g., learner performance at end of course)
3. Behavior (ability to use what is learned in realistic situation or job)
4. Results (impact on the learner's organization)

Observing an outcome does not mean we can attribute it to the learning situation, as Steve Ehrmann noted. We may document a learning change but not understand what actually happened or how learners and coaches actually used the options we provided. Because of this, Project Flashlight focuses heavily on what faculty and students actually do. Lanny Arvan suggested a division of labor between evaluators who collect the outcome data and instructors who report on how they used the technology. Instructors may be inexperienced in evaluation, but they need an outlet for discussing their approach. Learner choices in ALN may not be visible to faculty, and many of the most important outcomes in education result from teaching patterns that culminate over many courses. From these observations, Ehrmann suggested that evaluators work with faculty to find patterns of choices by faculty and students that interfere with or cancel out each other, but to leave responsibility with faculty to learn to measure what they care about.

So what should we evaluate? David Brigham suggested a focus on process: what influences facilitate and impede student perceived progress toward student course goals, how are these influences related, and what is their relative importance? He, like Richard Clark (1991), pointed out that it's not the delivery method that makes the difference - it's the learning strategies employed within the delivery method, and implementation of those strategies, that matter most. Campbell added that new delivery vehicles (like computers) can enable strategies like powerful simulations. He also observed that learner and coach self reports can be helpful but are just one cue to causality (see Einhorn and Hogarth, 1986). He prefers a combination of perceptions through a qualitative study and outside/more objective rating via job performance, projects, tests and expert opinion.

Evaluation of examinations and grades may be too limited. Instead we need to identify specific and meaningful learning outcomes and compare student achievement of those outcomes using different methods, according to Edward Neal. That means mastery of higher-order learning rather than simple recall or application of information. He cites Bloom's Taxonomy of Educational Objectives as particularly helpful to identify precise and measurable definitions of learning outcomes. These include recall, comprehension, application, analysis, synthesis, and evaluation (judgment). In particular, Neal is concerned with undergraduate (not adult/professional) education, and suggests that many interchanges that occur in ALN do not improve higher order thinking abilities, but are simply sequential position statements.

Iv. Impacts of Anytime/Anyplace Learning

ALN might have negative impacts. Examples cited by Campbell are:

• taking time at home away from family life
• allowing third rate providers to appear to be at the same level as "name brand" universities
• disenfranchising the poor who lack Web access
• requiring corporate learners to study at their desks in competition with the priorities of their daily tasks.

John Gundry observed that if the workplace is too noisy or subject to interruption for ALN, it doesn't say much for "knowledge work" there either. Several working professionals from East Carolina University (ECU) who are studying for an MS in Industrial Technology with Dr. John Reid weighed in on this point. Reid noted that the learning environment must be conducive to good learning. In fact, by opening up an area for asynchronous learning, a local manufacturing plant expects to benefit from a tax break. Laurence Hornibrook said this type of program lends itself to his work very well, given that he travels and is a parent. The costs are comparable to any state university, but ALN avoids travel costs and may not require a textbook purchase. He added that, "To pursue knowledge requires a constant struggle and one that if you don't keep at it will cause you to fall behind quickly." Donald Sanders added that without ALN he could not complete a degree in his field because no colleges in his local area offer such courses. Because of ALN he spends more time home with his family. While some of that time is on the Web, he's not wasting travel time to and from school. Without his online team he would have been unable to complete some of his assignments.

Kevin Clay, who is disabled and finds it difficult to maneuver around a large campus, observed that were it not for the online courses at ECU he would have greater difficulty pursuing his M.S. degree. Thomas Weigand said that since he works rotating 12 hours shifts, he would be unable to continue his education were it not for the online program.

In the Air Force, which requires annual recertification on many tasks but has limited classes available, ALN can increase the availability of classes and save travel costs according to Jim Hollister. Another benefit would be to go through a learning module at your home base, then immediately walk out to the flight line and get hands on training on your own equipment. Sanders noted a practical problem with this: access to computers, and especially those connected to the Web, can be very limited. Moreover Air Force or corporate policy may preclude using their computers for ALN courses that are not directly duty related.

What is the relative impact of experience in the classroom and in ALN? Joseph Jordan prefers the classroom because he has access to the professor, instant feedback to questions (which allows clarification of the question), and more discussions with classmates. Clay observed that in the classroom learners may fear asking a "stupid question" or not be able to get their point across. This may apply especially to foreign students who are still learning the language. Other students may dominate the discussion, not allowing time for others. In ALN learners can think through their question before posting it, and must take initiative to find their own answers instead of relying on the faculty.

Impacts of ALN may be determined by what George Blakeslee terms the frequency function - how frequently learners are expected or required to take part in class discussions. It ranges from continuous chat to completely independent study. The less frequently learners must log on and take part in discussion, the more explicit and bounded the initial structure of the course must be, and the less lively will be discussions. This is because discussion and changes take longer and longer to propagate through the class and because long wait times between comment and reply produce an intellectual detachment leading to mini essays rather than personal discussion.

The number of postings may have optimal values. Too few, and there is little to talk about. Too many, and participants cannot reasonably read - much less respond to - them all. Skip Knox found that requiring three messages a week from each of 20 learners makes for a lively discussion. He thought that decreasing the messages per person and increasing the number of people (as with a successful listserv) could send the discussion off on tangents, proceeding by fits and starts. The discussion can help to create a learning community where those with similar interests (for example the ECU folks above) find each other, potentially learn from each other, and develop a social cohesion that could increase learning and completion rates.

How does ALN impact the role of learners and coaches? As the question suggests, learners may come first and faculty may become coaches more than tellers. Blakeslee stated that he must set specific expectations for learners to take leadership for portions of the class, but that shared responsibility for learning opens him to criticism that learners are not getting the instruction from him that they feel they paid for. He observed that, counter intuitively, with worldwide communication capability learners tend to avoid interacting with each other. He ends up in the middle of a "star" communications structure where he talks with every student but they talk only with him. The workload becomes overwhelming. On the other hand, Knox noted that he hated seminars that had group assignments, and that perhaps we are trying to force behavior onto an unwilling population.

V. How to Evaluate: Methods and Strategies

Evaluation is often used to find positive results to show a sponsor. For those doing academic research there may be little incentive to scale up and widely disseminate a new learning strategy after we've "shown it works." But to develop a discipline of research we need to know what we want out of the evaluation, according to Campbell. One way is to create outcome measures and then to determine appropriate methods and groups for comparison. Qualitative research (e.g., observations, interviews, and analysis of learner postings) helps us make sense of what is going on. Then we can turn to quantitative evaluations and comparisons when we have ideas about what to look for.

Steve Ehrmann noted that, for example, looking for comparable outcomes with lower costs has at least two weaknesses: weak evaluation methods that produce "no significant difference" (nsd) results, and finding "comparable" comparisons. The problem with nsd is that it's touted as showing equivalent outcomes, when the real reason is likely that the evaluation was too weak to detect the differences, according to Campbell. Experiments can be designed using a power analysis to estimate the number of learners required to find statistical significance, according to John Gundry, while James Salsman pointed to the USENET newsgroup sci.stat.edu for more information. This approach can lead to finding significant differences if there are any - which can often be more helpful than nsd results. Random assignment of learners to learning conditions decreases the problems with comparable groups. While this is sometimes viewed as impossible in an academic setting, Campbell has done it several times with informed consent and approval of the Institutional Review Board (that protects the interests of participants in experiments), as part of regular academic courses in both interpersonal and technical areas.

Gundry flagged a study by Gerald Schutte. It reports an average of 20 percent higher test scores for students randomly assigned to a Web learning environment, in relation to those assigned to a traditional classroom. Other reported results were significantly higher perceived peer contact, time spent on class work, perceived flexibility, understanding of the material and greater affect toward math. The study is at http://www.csun.edu/sociology/virexp.htm.

The Schutte study had not gone through the scholarly review process at the time of the discussion, though it appears to be one of the few experimental designs to evaluate ALN-like work. Edward Neal found the study seriously flawed because Schutte used different teaching methods for the treatment group and the control group, giving the "virtual class" more opportunities to be involved with each other, the instructor and the course material each week. Neal opined that to attribute the differences to technology, Schutte should have kept the treatments the same. Instead he ". . . simply demonstrated that cooperative learning methods and intensive engagement with the material yield improved performance - results that other researchers discovered years ago."

In reply, Gundry supposed that Schutte should indeed be seen as demonstrating that a technique works, as part of other projects starting to investigate a new technique for learning. Collaborative learning is a key pedagogical method for ALNs, and the study suggests that collaboration can be successful in ALN. Perhaps Schutte asked how he could compare a traditional class to one where technology affords the opportunity to decrease the costs of interactions. Campbell noted that it is not the technology itself, but that the technology affords us the opportunity to use what works, which we could not previously afford. Neal commented that Schutte, "used millions of dollars worth of technology to accomplish what teachers have done with no technology at many institutions." Ehrmann replied that Neal's comment ignores the enormous costs of the campus over decades, and that Schutte's study can expand our imaginations about what might be possible and desirable.

Another issue is exam security. Matt Insall wondered how we could ensure that students take the test themselves when there is no classroom and no proctor. Campbell observed that Sylvan Prometric centers are routinely used for high stakes testing for licensure and certification. Brigham said similar centers will eventually be available throughout the world.

VI. For more information

1. Clark, R. E. (1991). When researchers swim upstream: Reflections on an unpopular argument about learning from media. Educational Technology, 31(2), 34-40.
2. Einhorn, H. J. & Hogarth, R. M. (1986). Judging probable cause. Psychological Bulletin, 99, 3-19.
3. John Ory at University of Illinois is evaluating the extensive work of the Sloan Center for Asynchronous Learning Environments (SCALE), and points to their site at: http://w3.scale.uiuc.edu/scale/
4. Jim Salsman suggests the following Web site on the impact of technology in education: http://www.mcrel.org/connect/tech/impact.html