Educational Performance of ALN via Content Analysis

Download PDF version:

¹ Work supported in part by a grant from Multimedia Online Services and Technology Consortium, Inc., Israel.
² This work was done while the author was on sabbatical from the Open University-at the Department of Computing     Sciences, Victoria University of Wellington, New Zealand.

Abstract
Learning in an ALN mode is modeled by a set of educational processes. The group is modeled by an abstract entity that provides services to the learners via its group educational processes. The learners reciprocate by their corresponding educational processes. Following findings of the Social Interdependence Theory of Cooperative Learning, we conjecture that the ALN is Cooperative Learning enhanced by extended think time. If ALN is structured for effective cooperation then the group dynamics will regulate the high level reasoning and the interpersonal relationships of the learners towards their highest levels.

If this conjecture is found to be true, it identifies the maximization of reasoning and interpersonal relationships as one of the educational benefits of an ALN.

To test the conjecture, we developed a methodology for the evaluation of the performance profiles of the ALN educational processes. Performance profiles are calculated via content analysis of the information flows exchanged between the participants, and the results are tested for reproducibility. We use this methodology to analyze three weeks of asynchronous discussions embedded in an ALN course of the Open University of Israel (OUI). The results of this analysis indicate the plausibility of our conjecture.

I. INTRODUCTION

A. Open University of Israel
The OUI was established 25 years ago. It provides 400 courses to 40,000 (part-time) students via distance learning methods. The University was modeled after the British OU. Learning was a mix of individualized learning (based on pre-prepared packages of learning materials), with substantial support from face-to-face tutorials, telephone consultations, and one-way television broadcasting.

In 1992, this author suggested that the OUI explore ways of incorporating the (then) emerging telecommunication technologies into its teaching and learning environment. A University-wide task force was established and, following its recommendations an experimental project (directed by this author)-TEle LEarning Methodologies (TELEM)-was set up with the goal of identifying the benefits of asynchronous learning and methods of its implementation. This project was later expanded into The Center for Information Technologies in Distance Learning, of which TELEM is one department. The Center is in charge of exploring benefits of learning methodologies that employ a variety of information technologies, asynchronous and synchronous, and of introducing them (the technologies and the methodologies) into the courses of OUI. TELEM focuses on ALNs. Currently about 80 courses include Internet-based technology support (Websites, E-mail, and asynchronous forums). Ten courses include Synchronous Technology support (satellite-based interactive communication: one-way video, two-way audio) as well. The plan is to introduce asynchronous technology into most of the 400 courses in the next three years.

B. Goal of this Work
The primary difficulty in diffusing the usage of ALN and other technologies into the fabric of OUI courses was and still is the lack of clear understanding of the educational benefits of doing so. Theoretically, ALN enables a variety of cooperative activities, enhanced by extended think time. Theoretically, then, it enables students to acquire benefits that seem to be hard to acquire with other modes of distance learning. These include deep, time-consuming reasoning processes, for example. These ideas have to be developed, researched, and refined within a solid educational theory and tested, so that a cost-benefit analysis can be performed. This work is a first step. Its goal is to demonstrate the feasibility of

• Incorporating ALN into an existing educational theory
• Conjecturing a set of educational benefits
• Deducing the necessary conditions that ALN should satisfy in order that the educational benefits will be achieved
• Developing a methodology for quantitative evaluation of the actual levels of achievements of these benefits
• Demonstrating the plausibility of the conjecture by using the methodology to evaluate the performance of an experimental ALN course

This research is a feasibility study since its experimental findings are based on a single experiment. It does not include repeatability tests. (Other shortcomings of this research will be outlined in the Discussion section of this paper.)

C. The Course
The course that served as a test for this research was Computer Mediated Communication and Distance Learning. This course is a senior-year, small-group, seminar-type course, and an elective in the Bachelor of Arts in Computer Science program. It was specifically established with a view of using it as a test for research. The course was available in one integrated (mostly asynchronous) form, described in Section IV. The intended audience was Computer Science undergraduates nearing the end of their studies (a Computer Networks course was a prerequisite) who were specifically interested in computer-mediated communication from both technical and educational standpoints.

This paper is organized as follows: Section II, theoretical foundations; Sections III and IV, background information and the delivery method of the course that served as a test in this research, respectively; Section V, results; Section VI, discussion; and Section VII, summary.

II. RATIONALE

The primary motivation for creating this research was the wish to understand the educational benefits of ALN. Toward this end, we searched for suitable educational-benefits candidates and an appropriate theoretical framework and developed a methodology for evaluating the achieved benefits. In order to demonstrate the feasibility of the methodology, we designed and carried out an ALN course that served as a test. The theoretical foundation is described in Subsection A; an ALN process model in Subsection B; structuring of ALN in Subsection C; and the evaluation procedure in Subsection D.

A. Conceptual Framework: The Social Interdependence Theory of
Cooperative Learning
"Asynchronous Learning Networks (ALN) are people networks-learning anywhere" [1]. The key characteristic of the ALN model is the capability of its learners to be a part of a community, cooperating asynchronously to perform a common task-learning [1], [2]. Typical implementation of the ALN model is a group of participants (learners, tutors, experts, technicians, and lecturers) supported by a distributed computer system that includes information servers (WWW, ftp, library), their associated clients, and multi-way communication software such as electronic mail and conferencing system (For a detailed example, see [3]. For a recent discussion about the policy issues, functional requirements and practical issues see [4], [5], and [6], respectively).

Hiltz and Turoff [7] pointed out that the educational value of an ALN should measure the educational benefits to the learner. Moonen [8] noted that a critical factor in a cost-effectiveness analysis of ALN is the effective usage of the relatively long contact time of the learner with the ALN environment. The perceived effectiveness of ALN determines its usage by the learner. Earlier studies [9] show that learners perceive the content of communication as an information resource. But the benefits of ALN go beyond the accumulated knowledge. It is the process of construction of the shared body of knowledge that is valued in constructivist learning theory [10]. This process includes a rich set of cooperative educational communication patterns-questions and answers, guided tutoring, announcements, concurrent multithreaded discussions, voting, competitions, etc. [9]. The asynchronous nature of communication gives rise (at least theoretically) to extended learners' think time. Thus, the starting point of this project was the hypothesis that learners in the ALN mode gain additional educational benefits from ALN.

Focused studies identified some sets of these educational benefits. McReary [11], Newman et al. [12] and Henri [13] pointed out that certain critical-thinking processes such as reasoning benefit from ALN. Wegerif [14] emphasized the social-support benefit of ALN. Hiltz and Turoff [7], Ellis and McReary [15], Harasim [16] and others identified the peer-to-peer interactivity or the responsiveness benefit.

These observations fit well with the social interdependence theory of cooperative learning (see Johnson and Johnson [17] for a complete description). Briefly, this theory suggests that the way social interdependence (of group members) is structured into the learning environment determines the group dynamics, which, in turn, determines the learning outcomes. Learning is structured for effective cooperation when members' goal achievements are positively correlated, i.e., individuals perceive that they can reach their goals if and only if the others in the group also reach their goals. In this case, individuals seek outcomes that are beneficial to all those with whom they are cooperating. This type of interdependence (as opposed to competition or to individualized learning) results in promotive interactions-actions of individuals substitute for each other, participants positively invest energy to support each other's actions and there is a high degree of openness among participants. This results in a large variety of educational outcomes that are classified into the categories of High Effort to Achieve, Positive Interpersonal Relationships, and Psychological Health. These outcomes influence each other. Numerous experiments, performed in the last 100 years, provide a sound ground for the validity of this theory and its findings. For a comprehensive review, see [18].

Three findings of the social interdependence theory of cooperative learning are of particular importance for ALN studies. The first is in the High Efforts to Achieve category: "Cooperative learning promotes a greater use of higher level reasoning strategies and critical thinking than do competitive or individualistic learning strategies." This finding is correlated with the fact that cooperators spend more time on tasks then competitors or students working individualistically [18-21].

The other findings are in the positive interpersonal relationships category. Social support is the degree of exchange of resources intended to enhance mutual well-being and the existence and availability of people on whom one can rely for assistance, encouragement, acceptance, and caring [17]. By peer support (or responsiveness) individuals in the group learn from each other how they perform on a given effort. It was found that "Cooperative experiences tend to promote greater social support than did competitive or individualistic efforts", and in addition, "stronger [educational] effects were found for peer support than for superior [teacher] support" [18].

The three outcomes are correlated. Efforts to achieve high levels of reasoning and quality of interpersonal relationships are reciprocally related [18]. Caring and committed friendships come from a sense of mutual accomplishment, mutual pride in joint work, and the bonding that results from joint efforts. The more students care about each other, however, the harder they will work to carry out their mutual educational tasks.

Thus, the conjecture of this research is: ALN is cooperative learning enhanced by extended think time. If ALN is structured for effective cooperation then the group dynamics will regulate the high level reasoning and the interpersonal relationships of the learners towards their highest levels. This conjecture is indirectly supported by the preliminary finding of the Virtual Classroom project, presented by Hiltz [22]. These findings present learners' subjective perceptions of learning in the ALN mode relative to traditional face to face. These findings will be discussed again in Section VI of this paper.

The conjecture claims that one educational benefit of ALN is the maximization of reasoning and interpersonal relationships. This research aims at objective testing of the conjecture. To do this, several clarifications are in order: 

• What is the metric of strength used here, i.e. what are the meanings of "highest level" or "maximization" in the context of reasoning and interpersonal relationships? 
• How does one structure ALN for effective cooperation? 
• How does one evaluate, in practice, the performance of the abstract entities "high level reasoning" and "interpersonal relationships"? 

These issues are discussed in the next three subsections, respectively.

B. ALN Process Model
Constructivist learning theory values the processes of construction of the shared body of knowledge [10]. Hence, we model an N participants ALN as a collection of processes. The group of participants in an ALN can be thought of as an abstract entity, peer of the individual learner. This entity knows how to socialize, to respond to learner queries, and to provide reasoning. It provides Educational Services carried by group processes to the learners that encourage them to reciprocate by their corresponding input processes. Following the social interdependence theory of cooperative learning, we hypothesize the existence of three processes-sets. The Social Processes set, the Response Processes set, and the Reasoning Processes set. Each of these sets includes one service (group) process, and N input (individual) processes.

The Social and Response Processes model the creation and maintenance of the cohesive responsive cooperative group of participants. The Reasoning Processes model of learning uses a set of cognitive skills and knowledge of the participants in the ALN to carry out the educational tasks. This list of processes is not exhaustive. Educational activities other then reasoning, e.g., meta-cognitive reflections can be modeled by additional processes. Each of the processes is performed at a certain level and at any point in time. Maximizing a process is regulating it towards its highest possible performance level. These levels are described next.

The Social Processes provide group cohesiveness. These processes move the learner across the threshold from outsider to insider status [15]. Each can be performed at two levels-SocialValue or NoSocialValue. They either exist or not.

The Response Processes provide the content-relevant communication between the learner and the group. Winiecki [23] presented strategies for reconstituting conversational practice into ALN that can improve participants' ability to "keep the thread" of the discussion. The social interdependence theory of cooperative learning makes a distinction between peer responses (learner-to-learner) and learner-to-tutor responses. At the lowest level, NonResponsive, these processes do not relate to any learner's specific needs. Otherwise, the processes respond to the tutor (Response-To-Tutor level) or at the highest level (Response-to-Learner), to the learner.

The Reasoning Processes exhibit skills and knowledge that are on par with those at work in problem resolution [24]. Reasoning Processes perform, then, at five levels, described next. These levels correspond to the learning hierarchy suggested by Biggs [25]. We use the definitions suggested by Henri [13].

Reasoning at the SimpleClarification level means identifying problem elements and their linkages. DeepClarification provides more details about beliefs or assumptions that underlie the statement of the problem. Inference means problem solving via induction, deduction, or another problem-solving methodology, e.g., algorithm. Reasoning at the Judgement level means making decisions, appreciation, evaluations, and criticism of content-related issues. Reasoning at the Strategy level proposes a plan for attacking the problem. These educational processes and their possible performance levels are presented in Table 1.

The information flow created by a participant in a cooperative-learning scenario consists of three sub-flows contributed by his/her three input processes. The performance of each of these processes is described by a performance profile, which is the distribution of the process-levels exhibited in its sub-flow. The educational performance of a participant is, thus, described by a set of three performance profiles. For example, participant seven might be characterized with a performance profile of his/her input Reasoning Process equal to (0, 25%, 30%, 45%, 0); the input information flow of this participant provides a reasoning sub-flow consisting of 25% DeepClarifications, 30% Inferences, 45% Judgments, but neither Strategies nor SimpleClarifications. The Reasoning Input Process of participant seven created this information flow. Participant seven is characterized, in addition, by performance profiles of his/her input Social Process and the input Response Process.

Table 1: Educational Service and Input Processes

The performance of the ALN group is described by a similar set of performance profiles of the three service processes. These profiles are the distributions of the group process-levels in the corresponding sub-flows of the total ALN information flow.

The performance profiles achieved by the learners and the group depend on the group dynamics. The social interdependence theory of cooperative learning predicts that the profiles are correlated and that if the cooperative learning is structured for effective cooperation then the profiles will migrate towards their respective highest or best profiles. The best profile of the Reasoning Process is determined by the educational task. If, for example, the primary activities required to carry out the educational task are induction and deduction, then the reasoning profiles should be, theoretically, biased towards the Inference level. The exact forms of the best profiles are not known. The social interdependence theory of cooperative learning indicates that the best profile of the Response Process will be peaked around the ResponseToLearner level, and the best profile of the Social Process will include high values of SocialValue. How much is high-we cannot tell at present.

We can now rephrase the conjecture of this research: ALN is cooperative learning enhanced by extended think time. If ALN is structured for effective cooperation then the performance profiles of the educational processes will be regulated by the group dynamics towards their best profiles. In particular, the performance profile of the Reasoning Process will be biased towards the reasoning level associated with the educational task; the performance profile of the Response Process will be biased towards the Response-To-Learner level; and the performance profile of the Social Process will include high values of SocialValue.

C. Structuring Effective Cooperation into ALN
Structuring effective cooperation into ALN is laying the foundations for the group dynamics and, hence, for the educational processes. Successful structuring will shift the performance profiles toward their best profiles. The basic structural components of effective cooperation are positive interdependence, group reflection, individual accountability, promotive interaction, and social skills. Numerous studies demonstrated that these components are necessary conditions for high achievements [18].

Positive interdependence exists when each learner perceives that he (or she) succeeds if and only if all learners succeed, so that they all must coordinate their efforts. Among the methods for structuring positive interdependence into learning are

• Deliverable interdependence (producing a deliverable by the whole group)

• Task interdependence (division of labor)

• Resource interdependence (sharing learning materials, tools, and information created on the fly)

• Role interdependence (assigning roles to learners)

• Reward interdependence (rewarding individuals for group performance above some threshold)

Group reflection is a periodic process in which the group reflects upon its learning activities, analyzing what member actions were helpful or unhelpful and makes decisions about what actions to continue or change. This is a mechanism that provides learners with the feeling that they are on the right track. They get adequate service from the group, so their time is efficiently used. A good method to structure group reflection into ALN is dedicated face-to-face meetings. If this is not feasible, then one can enforce periodic assignments in which learners fill out evaluation forms monitored by the tutor.

Positive interdependence and group reflection are the low-level components of the required structure. Positive interdependence is the static infrastructure, whereas group reflection is an externally monitored dynamic force that directs the system, efficiently, on course.

Individual accountability is the sense of the personal responsibility for completing one's share of the work and contributing to the work of the group. Insisting on individual accountability discourages taking free rides on others' work. Structuring individual accountability into learning can be done by imposing periodic personal work-reports and testing individual members of the group who are selected at random.

Since individual accountability specifies the educational task, it provides the initial drive for the learning activities, which will be developed by the group dynamics into the Reasoning Process.

Promotive interaction is the encouragement and facilitation of each other's efforts to reach the learning goals-providing help, feedback, and resources, and advocating increased efforts to achieve. Initial
promotive interactions can be achieved by positive interdependence. Learners can be assigned the roles of helper, feedback provider, resource manager, and process reflector. The tutor should then closely monitor and regulate the workings of these promoters as their enthusiasm tends to decay. These roles should be rotated between learners. The promotive interaction component lays the foundation for the Response Processes.

For the purpose of forming a community, the learners are required to learn and practice four basic social skills-get to know and trust each other, communicate accurately and unambiguously, accept and support each other, and resolve conflicts constructively. The tutor should explain these skills and their importance and follow this with a set of structured, light-hearted, warm-up group-skills exercises. This component lays the foundation for the Social Processes.

D. Evaluating Performance by Content Analysis
As early as 1991 Mason [26] suggested that performance profiles of educational processes could be evaluated by content analysis of the messages exchanged between these processes. Henri [13] suggested an analytical method for extracting the performance levels of participants from the content of the information flows in ALN. Bonanno [27] evaluated communication patterns of participants in an ALN via content analysis of the messages. More recently, Newman, Webb and Cochrane [12] performed an extensive one-dimensional (the depth of critical thinking dimension) comparative content analysis of a face-to-face course versus an ALN course. They found that group learning in either of these environments provides a similar depth of critical thinking. Krippendorf provides a comprehensive description of the methodology of content analysis[28].

Our evaluation proceeds in four major steps-unitizing, coding, analysis, and reliability tests.

1. Unitization
The data in this research consist of the input information flows of all the participants in a set of asynchronous discussions. In the unitizing step we divide the data into units of analysis. Each unit will be analyzed independently in the analysis step. There are several candidates for units--physical units (messages), syntactical units (words or statements), referential units (messages sent by a particular person), propositional units (identified by having a predefined structure), and thematic units (identified by definitions of various contents). Thematic units are the most preferable in content analysis since they are related to the context in which the analysis will be performed. On the other hand, it is generally difficult to identify them reliably.

In this work we use a three-level hierarchical unitization. The data is divided into elementary information units that are aggregated into the input information sub-flows of the participants which are aggregated into the information flows of the participants. An elementary information unit is a statement or a continuous set of statements, which convey one identifiable idea (a simple thematic unit). An input information sub-flow of a participant is the collection of all the elementary information units sent by this participant, which exhibit one of the three educational processes. An input information flow of a participant is the collection of all his (or her) input information sub-flows. Actual identification of sub-flows and flows is done in the coding step.

2. Coding
In the coding step, each elementary information unit is coded with  (sub-flow, level) pairs. Sub-flow is one of the three processes; level is one of the levels of this process. This coding means that the elementary information unit exhibits that process at that level. Decisions about coding are based on the identification of appropriate identifier-statements in the elementary information unit. The identifier-statements are defined by a set of coding rules. Henri [13] provides details about these rules. These rules are listed in Tables 2-4.

Table 2: Coding Rules for Social Process Sub-flow

Table 3: Coding Rules for the Response Process Sub-flow

Table 4: Coding Rules for the Reasoning Process Sub-flow

3. Analysis
The coding step identifies the elementary information units associated with each input process of each participant (and hence of each service processes of the ALN). For each elementary information unit the associated process level is also recorded. In the analysis step we calculate for each process the relative frequencies of appearance of the performance levels exhibited in the codes assigned to its elementary information units. The set of the relative frequencies is the performance profile of that process. Performance profiles will serve as the basis for drawing conclusions about the educational performance of the ALN and the learners. This will be covered in Section V, Results.

4. Reliability Tests
The last step in content analysis is to conduct reliability tests. At a minimum, this step includes a two-coder reproducibility test-two data analysts perform content analysis on the same data using the same coding rules. Standard statistical tests are then performed to accept (or reject) the hypothesis that the differences between the resulting distributions of codes are due to chance. These tests can (and should) be done for the various distributions resulting from the analysis, including the distributions of codes among elementary information units as well as distribution of codes among aggregated data units (such as the full information flows or sub-flow data units). In this research we used t-tests to check for reproducibility of the distribution of information sub-flow levels among learners.

III. BACKGROUND INFORMATION FOR THE COURSE

At the end of 1994, there were virtually no ALN courses at the OUI. Several courses in Computer Science started to incorporate electronic mail for simple tutoring and delivery of assignments. The first course that was specifically designed with an ALN mode of learning as a primary mode was Computer-Mediated Communication and Distance Learning, which was designed and delivered by this author. This was a 17-week, one-semester course which included three, one-week asynchronous discussions which served as the test for this research. Ten learners, one tutor (this author), and one technician participated in the course and the discussions.

Learners were undergraduates majoring in Computer Science nearing the end of their academic programs. They all had a Computer Networks course-a typical, end-of-the program course before taking this course. As is usual in Israel and in particular at the OUI, these were mature students (all 25 and older; all were working in addition to their studies). Their grades were (this was found after the course) average and above. Two students-in addition to the ten-started the course but left right at the beginning. This was due to the intensive workload and the particular topic of this course. All learners were experienced Internet users, and, of course, very experienced in using computers. None had any previous experience with ALN. The course was an elective targeted to Computer Science students with special interest in education. There was no set goal for the number of learners, but as it was intended to serve as a test for researching cooperative learning, it was expected to undertake about 15 learners.

IV. METHOD

A. Technology and Infrastructure
1. Hardware/Software
Each learner used his PC from home. This included Windows software or UNIX with TCP/IP or Terminal Emulation over dial-up. We supported all four platforms. The OUI served as the Internet service provider. A UNIX workstation (SUN SPARC) at the Department of Computer Science provided a set of Internet server processes. No commercial applications were used. For those who did not have TCP/IP or Terminal Emulation software on their PCs, we provided a public domain version.

2. Electronic Delivery Mode
Most of the communications (E-mail and newsgroups) were text-based. No streaming media was used.

3. Management of Infrastructure
The network administrator of Computer Science managed the technical infrastructure-servers, networks, etc. in-house.

B. Content Delivery
1. Delivery
All course materials, definitions of assignments, the work schedule, and the descriptions of the asynchronous discussions were delivered to the learners at the beginning of the course. Learners handed in assignments and made all other transactions (questions, advice, work exchange, discussions, etc.) electronically via asynchronous communication.

A detailed, tight, intense work schedule was prepared and delivered to the learners at the beginning of the course. This covered the whole series of ALN activities, including the interleaved sets of series of individualized learning, face-to-face tutorials, Internet search and retrieve activities, team projects, and the asynchronous discussions. The work schedule is listed in the next paragraph.

2. Structure
The Computer-Mediated Communication and Distance Learning course was an integrated course. It included static information resources-a monograph [9], study guides, research papers, and dynamic resources from the Internet. It also included synchronous (face-to-face) meetings at the beginning, the middle-after the first asynchronous discussion, and at the end, but most of the communication was asynchronous-via electronic mail and newsgroups. It included individual work and cooperative projects, including three, one-week asynchronous discussions. Each of these had a predefined deliverable (a document). The workflow of the course is described in Box 1.

Box 1: Workflow of the Test Course

The course was structured for effective cooperation, as described in section II.C. The class was a formal cooperative learning group that lasted for 17 weeks. Positive interdependence was structured into the course via deliverable interdependence (joint reports), task interdependence (division of labor), resource interdependence (exchange of materials and information created on the fly in discussions and in assignments), role interdependence (panel members; see below), and reward interdependence (a participant's grade in the asynchronous discussion was determined by the degree of triggering others to provide learning inputs). Group reflection was structured into the course by the formal three face-to-face meetings in which the learning procedures were discussed. Evaluating learners' grades individually enforced individual accountability. Promotive interaction was enhanced directly in the face-to-face meeting and indirectly by monitoring the work of the panel members. Social skills were exercised by several simple group works at the beginning of the course.

3. The Asynchronous Discussions
Each of the three asynchronous discussions had the same format. Background literature (usually two papers) was read in the two weeks prior to the beginning of the actual discussion. These were summarized by the each student; copies were sent to the tutor before the discussion.

At the beginning of the asynchronous discussion (Saturday night) the tutor broadcast one or two focused questions on the specific newsgroup allocated to this discussion. From there on, the discussion continued on the newsgroup (and on other newsgroups allocated for technical or administrative issues).

At the first broadcast, the tutor also assigned three students as panel members. The role of the panel members was to start the discussion thread (first response) and to keep it going with questions, remarks, etc. At the end of the week (Friday afternoon) the panel members summarized the discussions and the tutor-who spent most of the week on-line, but with minor interventions- provided concluding remarks.

4. Evaluation of Learners' Performance in the Course

• 30% asynchronous discussions (level of triggering others)

• 30% assignments

• 30% project

• 10% presentations

5. Face-to-face Meetings

• First day of course: Introduction, technical issues, deliverables and tasks, introducing model of learning, assigning social-skills experiencing exercises

• After first asynchronous discussion: Reflecting about the learning methods

• Last day of course project presentations and reflection

C. Organization and Evolution
1. Development Responsibility
This author (a faculty member) was responsible for the development and delivery of the course.

2. Technical Support
Technical support was provided on-line as well as by telephone, and on the first day, in a face-to-face meeting. The system and network administrator of the Department of Computer Science provided this support. He was available at all times for asynchronous communications. Telephone consultation times were predetermined. This arrangement was needed only during the first weeks, with most problems being modem-related.

V. RESULTS

The conjecture of this research is that (see section II.B) ALN is cooperative learning enhanced by extended think time. If ALN is structured for effective cooperation then the performance profiles of the educational processes will be regulated by the group dynamics towards their best profiles. In particular, the performance profile of the Reasoning Process will be biased towards the reasoning level associated with the educational task, the performance profile of the Response Process will be biased towards the response-to-learner level, and the performance profile of the Social Process will include high values of social value.

The three-week asynchronous discussions of the Computer-Mediated Communication and Distance Learning course serve as a test for demonstrating the plausibility of this hypothesis. The asynchronous discussions were structured for effective cooperation (see Section IV.B). For the purpose of this research, the discussions were triggered by focused questions (one or two per discussion) aimed primarily at the judgment level and secondary at the inference level. Typically, learners were requested to judge mutual compatibility (or incompatibility) of structured ideas from two research papers. For this, they had first to infer the reciprocal implications. For example, in the third discussion, learners were requested to judge the compatibility and incompatibility between proposed learners' needs and capabilities and the technical specifications of architecture of the computerized communication system that supports them. This discussion was based on two papers [29, 30], one presenting the architecture and the other evaluating learners needs and capabilities. None of the questions required the learners to construct a system, so no developing of strategy was expected.

The performance profiles of the ALN group and the learners' processes were calculated by content analysis as presented in II.D. The analysis was done independently by two data analysts (Master of Science students in Educational Technology) and their results were then tested for reproducibility. Since our interest is in assessing the performance level of the educational processes, we checked for the reproducibility of the distributions of each of these processes' levels among the learners. This means that we did not check for the reproducibility of the coding of the elementary information units but for the reproducibility of the coding of the aggregated information sub-flows.

The performance profiles of the service (group) processes are presented in Section A. The performance profiles of individual learners are discussed in Section B, and the reliability tests are presented in Section C.

A. Educational Services of an ALN
Figure 1 presents the performance profiles of the ALN service processes.

 
Figure 1

In Figure 1 we see that the reasoning service process peaks at the judgment and the inference levels, with relatively low levels of strategy, and that the response service process peaks at the response-to-learner level. In other words, the reasoning service process was self-regulated by the group dynamics to its highest possible levels compatible with the educational task. Concurrently, the response service process is indeed strongly biased towards response-to-tutor. This means that the asynchronous discussions were learner-centered fruitful debates, not a series of monologues or "lectures".

From Figure 1, we see that the ALN group provided a relatively high level of social service process (30% of the information flow carried social value). This high level of social service of the discussions enabled most of the learners to pass the threshold and created a cohesive group of cooperating learners. These results indicate that the conjecture is, at least, plausible. Obviously, further research is needed to transform this conjecture into a testable assumption and the plausibility into a proof. We shall discuss the limitations of this research in Subsection VI.B

B. Educational Performances of Individual Learners
Figure 2 presents the performance profiles of the social input processes of the participants. These profiles were calculated by content analysis (Section II.D) using the coding rules listed in Table 2. We see that all the learners' inputs (except learner seven) provide relatively large, similar levels of social value, despite the fact that their actual participation in asynchronous discussions (Figure 3), vary. In other words, these learners considered the discussion as a social event, independent of their actual participation or total contribution to it. (The relative number of elementary information units contributed by each participant measured Participation).

 
Figure 2

 
Figure 3

The performance profiles of the response input and the reasoning input processes were calculated by content analysis (Section II.D) using the coding rules listed in Tables 3 and 4, respectively. These profiles are presented in Figures 4 and 5.

We identify four groups of learners. Learners 3 and 9 perform at low levels. Their responses have low levels of response-to-learner and their reasoning is mostly at the clarification level. We conclude that learners 3 and 9 are revealed as non-performers, technically as well as educationally.

Figure 4

The second group consists of students 1, 2, 6, 8, and 10. These students are assertive, socially connected, responsive, and communicate via higher-level reasoning. The high levels of educational performance of these students regulated the group dynamics and, hence, determined the overall service levels of the asynchronous discussions.

High-level educational performances are not necessarily correlated with active participation. The third group, learners 4 and 5, participated less then those of the second group, but performed well in their response and reasoning inputs.

Figure 5

These three groups performed well in their social input. Student seven did not participate very much and did not contribute at all to the social strength of the ALN. However, this student performed well in his (or her) reasoning. He was also very responsive to other learners. This is the smart guy (or gal) that occasionally communicates but other learners should be waiting to hear what he/she has to say.

C. Reliability
Distributions of the relative values among the N=10 learners for each process level were calculated by the two analysts. These distributions were t-tested for reproducibility. Specifically the null hypothesis is that the differences are due to chance fluctuation. T-ratios are presented in Table 5.

Table 5: T-ratios for Process Level Distributions (10 Learners)

A t-ratio of 2.26 or greater is required for significant disagreement (at the 0.05 level with 9 degrees of freedom). The results show that the differences in the results of the two data analysts-as far as the value distributions of process levels among learners are concerned-can be explained by fluctuation caused by the natural distributions (assuming they are normal) of these levels. The value distribution of the ResponseToTutor level is suspected to present non-reproducibility, which means that that there was some ambiguity in the coding rules for this level. Preliminary analysis shows that the analysts disagreed about coding indirect response to the tutor-whether it is ResponseToTutor (correct) or NonResponsive (incorrect).
Reliability analysis is essential for identifying ambiguities in coding rules. For example, in the preliminary phase of this research an attempt was made to identify meta-cognitive processes (such as self-regulation of one's learning procedure) in the information flows of the participants. Profiles were calculated (and even presented in conferences), but reliability analysis revealed significant non-reproducibility, which was tracked to disagreements between coders.

VI. DISCUSSION

A. Correlation with Other Studies
This research aims at objective identification of the educational benefits of the ALN. A complementary approach is to evaluate learners' subjective perceptions with regards to this point. An important, quantitative study that summarizes results from an extensive set of ALN courses was recently published by Hiltz [22]. This study is, in particular, relevant to our study, since it is based on the Virtual Classroom (VC) courses that were partially structured for efficient cooperation. Specifically, VC courses included a degree of positive interdependence (implemented by resource interdependence and role interdependence) and individual accountability. The report does not mention group reflection or acquisition of social skills. Promotive interaction was implemented by active resource interdependence (exchanging assignments between learners) that apparently was monitored by the class supervisor.

In Hiltz's study, learners compared (via post-course questionnaires) their experience in the VC environment to their experience in traditional face-to-face college courses. It was found that learners agree that the VC increased the quality of education (58% said yes, 20% said no), that they were and more involved in taking active part (49% said yes, 15% said no), "more motivated because others read their assignments" (55% said yes, 16% said no). Learners found resource interdependence useful (66% said yes, 9% said no), and, in general, believe that they worked harder in learning with ALN (67% said it was not easier, 13% say it was). These findings correlate with high levels of learning and student-centered responsiveness observed in our study as well as with the motivation increase predicted by the social theory of cooperative learning.

Two other findings are worthy of note. Hiltz reported that VC learners developed fewer friendships than face-to-face learners did. This could mean that the Social Process in the VC courses was relatively weak, which might be explained by the fact that social skills component was not structured into the VC courses. In addition, the modal answers to whether the VC increased the efficiency of educational delivery and to whether the VC increased the quality of education is that learners were not sure (in both cases 58% of answers were in the central three items in the seven-item Likert-type scale used to analyze these questions). Learners will feel that they get quality service or that their time is efficiently used if they will continuously participate in analysis of what and how they are doing, getting feedback, and regulate their learning procedures. This is the role of the group reflection component of the effective cooperation infrastructure, a component that (apparently) was not incorporated into the VC courses.

B. Shortcomings and Further Research
There are several shortcomings to our study worth mentioning. Overcoming these limitations can serve as starting points for further research.

• Sampling - The findings of this study are based on small sample. The word small does not refer to the size of the class (10)-this is appropriate for this kind of study (cooperative discussions). It refers to the number of groups, which was one. This is appropriate for a feasibility study such as this. To get a firm, consistent set of standards or alarm performance levels one needs to repeat this experiment with many groups, in a variety of structuring methods for effective cooperation, various levels of educational tasks, in different subject matters.

• Bias - This particular course and this particular group of learners were not picked to represent a general distance education course or a general student population. Learners were already strongly interested in asynchronous learning and the course materials recursively emphasized the same issues that were tested by content analysis (group dynamics). This was done in an effort to maximally regulate the educational processes towards their best profiles. Future studies on regular courses and students should compare performance levels with the performance observed here in order to get more insight on the parameters that determine performance.

• Scope - This study concentrated on reasoning as the primary learning process. There is no indication that other important processes cannot be maximized in an ALN environment. The small-scale group reflection process that is structured into ALN (for the purpose of directing the group, efficiently, on course) could be developed by the group dynamics to a full-fledged learning process in which learners are constantly using meta-cognitive skills and knowledge to perform learning reflection as an educational objective. Henri [13] suggested a set of levels for this process as well a set of coding rules to identify these levels in content analysis. Other educational processes can be added to the scheme presented here.

• Reliability - The reliability procedure used here was at the level of reproducibility. To go beyond that level one can correlate the results with results of an analysis performed by a different method, such as tests or questionnaires.

VII. SUMMARY

This research started by modeling ALN by a set of educational processes. The ALN group is modeled by an abstract entity that provides services to the learners via its group educational processes. The learners reciprocate by their corresponding educational processes. Following findings of the social interdependence theory of cooperative learning, we conjectured that the ALN is cooperative learning enhanced by extended think time. If ALN is structured for effective cooperation then the group dynamics will regulate the high level reasoning and the interpersonal relationships of the learners towards their highest levels.

To test the conjecture, we developed a methodology for the evaluation of the performance profiles of the ALN educational processes. Performance profiles were calculated via content analysis of the information flows exchanged between the participants, and the results were tested for reproducibility. We used this methodology to analyze three weeks of asynchronous discussions embedded in an ALN course at OUI. The results of this analysis indicate the plausibility of our conjecture. This is indirectly supported by other experiments.

The methodology presented here has several identified shortcomings that could serve as starting points for further researches. These will expand the sampling base, remove bias, extend the scope and increase the reliability level. If these researches corroborate the findings reported here then the maximization of reasoning and interpersonal relationships are identified as educational benefits of an ALN.

REFERENCES

• Bourne, J. What is ALN? The ALN Web, The Center for Asynchronous Learning Networks, http://www.aln.org/alnweb/aln.htm.
• Gasen, J., and Preece, J. Collaborative team projects: Key issues for effective learning. Journal of Educational Technology Systems, Vol. 24, No. 4, pp. 381-394, 1996.
• Bourne, J. R., Brodersen, J. A., Campbell, J. O., Dawant, M. S., and Shiavi, R. G. A model for on-line learning networks in engineering education. Journal of Asynchronous Learning Networks, Vol. 1, No. 1, pp. 73-96, 1997.
• Harris, D. A., and DiPablo, A. Institutional policy for ALN. Journal of Asynchronous Learning Networks, Vol. 3, No. 1, pp. 1-6, 1999.
• Carswell, L., Thomas, P., Petre, M., Price, B., and Richards, M. Understanding the electronic student: Analysis of functional requirements for distributed education. Journal of Asynchronous Learning Networks, Vol. 3, No. 1, pp. 7-18, 1999.
• Graham, M., Scarborough, H., and Goodwin, C. Implementing computer-mediated communication in an undergraduate course: A practical experience. Journal of Asynchronous Learning Networks, Vol. 3, No. 1, pp. 32-45, 1999.
• Hiltz S., R., and Turoff, M. The Network Nation: Human Communication via Computer, Cambridge: The MIT Press, 1993.
• Moonen, J. The efficiency of telelearning. Journal of Asynchronous Learning Networks, Vol. 1, No. 2, pp. 68-77, 1997.
• Mason, R., and Kay, A. Mindweave: Communication, Computers and Distance Education, Oxford: Pergamon Press, 1989.
• Salomon, G. No distribution without individual's cognition: A dynamic interactional view. Distributed Cognition: Psychological and Educational Considerations, pp. 111-138, Cambridge, England: Cambridge University Press.
• McReary, E. Eliciting more rigorous cognitive outcomes through analysis of computer-mediated discussion. Fifteenth International Conference on Improving University Teaching, Vancouver, 1989.
• Newman, B., Webb, B., and Cochrane, C. A content analysis method to measure critical thinking in face-to-face and computer supported group learning, http://www.qub.ac.uk/mgt/papers/methods/contpap.html.
• Henri, F. Computer conferencing and content analysis. In: Collaborative Learning Through Computer Conferencing, Kay, A., R., Ed., Berlin, Springer Verlag, 1991.
• Wegereif R. The social dimension of asynchronous learning networks. Journal of Asynchronous Learning Networks, Vol. 2, No. 1, pp. 68-77, 1997.
• Ellis, M., L., and Mcreary, E., K. The structure of message sequence in computer conference: Comparative study. Workshop on Computer Conferencing and Electronic Messaging, Guelph, University of Guelph, Canada, 1985.
• Harasim, L. On-line education as a new domain. In: Mindweave: Communication, Computers and Distance Education. Mason, R., D., and Kay, A., R., Eds. Oxford: Pergamon Press, 1989.
• Johnson, D., W., and Johnson, R., T. Learning Together and Alone, Cooperative, Competitive and Individualistic Learning, Allyn and Bacon (a Viacom Company): Needham Heights, MA, 1999.
• Johnson, D., W., and Johnson, R. Cooperation and Competition: Theory and Research, Edina, MN: Interaction Book Company, 1989.
• Gabbert, B., Johnson, D., W., and Johnson, R. Cooperative learning, group-to-individual transfer, process gain and the acquisition of cognitive reasoning strategies. Journal of Psychology, Vol. 120, No. 3, pp. 265-278, 1986.
• Johnson, D., W., and Johnson, R. Effects of cooperative and individualistic learning experiences on interethnic interaction., Journal of Educational Psychology, Vol. 73, No. 3, pp. 454-459, 1981.
• Skon, L., Johnson D., W., and Johnson, R. Cooperative peer interaction versus individual competition and individual efforts: Effects on the acquisition of cognitive reasoning strategies. Journal of Educational Psychology, Vol. 73, No. 1, pp. 83-92, 1981.
• Hiltz, S., R. Impacts of college-level courses via Asynchronous Learning Networks: Some preliminary results. Journal of Asynchronous Learning Networks, Vol. 1, No. 2, pp. 1-19, 1997.
• Winieki, D., J. Keeping the thread: Adapting conversational practice to help distance students and instructors manage discussions in an Asynchronous Learning, DEOSNEWS 9 (2), 1999. http://www.ed.psu/ACSDE/.
• Quellmaz, R., S. Needed: Better methods for testing higher-order thinking skills. Educational Leadership, Vol. 43, No. 2, pp. 29-35, 1985.
• Biggs, J. What the student does: teaching for enhanced learning. Higher Education Research & Development, Vol. 18, No. 1, 1999.
• Mason, R. Methodologies for evaluating applications of computer conferencing. In Collaborative Learning Through Computer Conferencing, Kay, A. R., Ed., Berlin: Springer Verlag, 1991.
• Bonanno, D., M. Evaluating electronic communication patterns over a semester: A qualitative content analysis, http://trochim.cornell.edu/Webeval/webcomm/webcomm.html.
• Krippendorf, K. Content analysis: An Introduction to Its Methodology, Beverly Hills: SAGE Publications, 1980.
• G. Alexander and C. Lincoln. The thought box: A computer-based communication system to support distance learning. In Mindweave: Communication, Computers and Distance Education, Oxford: Pergamon Press, 1989.
• S. Ehermann Improving a distributed learning environment with computers and telecommunications. In Mindweave: Communication, Computers and Distance Education, Oxford: Pergamon Press, 1989.

ACKNOWLEDGEMENTS

This research was supported in part by grant from the Multimedia Online Services & Technology Consortium, Inc., Israel. Special thanks are given to Tami Shapira and Orli Doron for the initial analysis of the data.

ABOUT THE AUTHOR

Dr. Reuven Aviv is a senior lecturer at the Department of Computer Science, Open University of Israel (OUI). He specializes in Computer Communication Networks, Distributed Computer Systems and in Asynchronous Learning Networks. He is the director of the Tele Learning Methodologies (TELEM) project and senior consultant to the Director of the Center for Information Technology in Distance Education, both at the OUI.

Contact: Department of Computer Science, Open University of Israel, 16 Klausner Street, Tel Aviv; Telephone: +64-4-463-5654; Fax: 64-4-463-5045; E-mail: Aviv@oumail.openu.ac.il.