PARTICIPATORY EXAMINATIONS IN ASYNCHRONOUS LEARNING NETWORKS: LONGITUDINAL EVALUATION RESULTS

Jia Shen
School of Management
New York Institute of Technology

Michael Bieber
Starr Roxanne Hiltz
Collaborative Hypermedia Research Lab, Information Systems Department
New Jersey Institute of Technology

ABSTRACT

This paper presents longitudinal evaluation results for an online participatory examination process in an information systems course over three semesters. The exam process includes students making up questions, answering other students' questions, grading answers to questions they author, and appealing the grades. The surveys following each exam elicited students' feedback, and an experiment compared the participatory exam with the traditional exam in the third semester. Survey results reveal that the majority of students have favorable attitudes towards the participatory exam, and would recommend the participatory examination for future courses. Students in the participatory exam enjoyed the process significantly more than students in the traditional exam, and have higher overall preference for the exam mode, although their perceived learning and perceived fairness in grading are lower than with the traditional exam. Discussion and future research on this topic are also presented.

KEYWORDS

Examination, Participatory Examination, Peer Assessment, ALN, Asynchronous Learning Networks, Collaborative Learning, Constructivism

I. Introduction

With the proliferation of the Asynchronous Learning Networks (ALN) technologies and pedagogy, "Virtual Classrooms™" [1] as predicted by researchers in the 1980s have become a reality. Eighty-one percent of all US institutions offered online courses by 2003 [2], and online learning is growing rapidly in the rest of the world [3]. In addition to providing “anytime, anywhere” accessibility, the online environment has shown advantages in supporting a variety of pedagogies and learning activities, with constructivist and collaborative learning found to be appropriate for this medium [4, 5].

When it comes to exams, however, the traditional instructor-centered examinations still dominate, where the instructor takes full control of the exam process by designing questions, proctoring exams, and grading the answers. Distance-learning students often have to travel to exam centers or take proctored exams. Can we take advantage of ALN to conduct exams so students can enjoy "anytime, anywhere" flexibility as well as maximize their learning through the exam? To address this question, three participatory online exams were conducted at NJIT in contiguous semesters (Fall 1999, Spring 2000 and Fall 2000), following a similar, but continuously refined process. The purpose of this paper is to describe the procedures used so that they could be replicated by others, and to present the longitudinal evaluation results of the participatory exam over the three semesters.

Traditional instructor-controlled exams reflect the objectivist learning model [6], where learning is regarded as the uncritical absorption of objective knowledge transferred from the instructor to students. New assessment approaches are proposed based on constructivist theory, in which knowledge is constructed actively by learners [7]. The constructivist model calls for learner-centered instruction [8]. In assessment, constructivism advocates learner-centered, non-criterion forms of assessment. The role of the instructor shifts from controlling the whole examination process, to supporting, guiding, and facilitating students' activities. For example, learner-centered assessment [9], and classroom assessment [10] propose to shift the attention from instructors and teaching to students and learning in assessment.

With the recognition of the constructivist nature of ALN, a small number of studies have been conducted to incorporate student active participation into the assessment process online. With the use of Group Support Systems (GSS), students' participation has been integrated into various phases of online assessment, such as development of a grading scheme [11], question composition [12], collaborative question answering [13], and peer and self-grading [14].

Based on constructivist learning theories, the participatory examination discussed in this paper features students' involvement in various phases of the exam process. Each student contributed and interacted with other students throughout the process, and the exam was conducted through cooperation with the whole class. While the term "Collaborative Examination" was used in prior publications describing the same exam process [15, 16, 17], in this paper "Participatory Examination" is used in accordance with the authors' subsequent studies [18] and others' publications [19].

This rest of the paper is organized as follows. Section II introduces the course where the exams were conducted, the exam procedures, and the ALN tools used in conducting the online exam. The research model, hypotheses, and the research design are presented in Section III. Research findings are organized into two sections: Section IV presents the survey results for the participatory exam, and Section V presents the field experiment results comparing the participatory exam with a traditional exam. Discussions, conclusions, and future research are included in Section VI.

II. COURSE BACKGROUND, EXAM PROCEDURES, ALN TOOLS

A. Course Background

The online participatory examinations in this study were conducted in CIS677, a graduate level course offered in the Information Systems Department in a U.S. university in Fall 1999, Spring 2000 and Fall 2000 semesters (hereafter referred to as semesters I, II and III). Masters and Ph.D. students majoring in computer science and information systems were enrolled in the course. The course studies how people and organizations should use information technologies effectively. It highlights both the current and the foundation research literature in the field of information systems. The course covers one topic for each lecture, with related textbook chapters and articles as instructional materials. There are two sections of the course: a traditional classroom or "Face-to-Face" (FTF) section where students meet once a week and a Distance-Learning (DL) section where students watch videotapes of the lectures. Students in both sections are required to participate in online discussions, which were on Virtual Classroom™ (VC) in semester I and WebBoard™ in semesters II and III.

B. Examination Procedures

Previously, one exam was scheduled in the middle of the semester for CIS 677. It was a threehour in-class proctored exam of 3–4 essay questions, with six pages of notes allowed. Based on constructivist learning theories and studies of constructive learning in ALN [7, 20], the participatory examination was designed as an entirely online process featuring students' active participation in all phases of the exam process. The basic procedures of the participatory and traditional examination are contrasted in Figure 1.

Figure 1. Comparison between the Participatory and Traditional Exam Procedures

Students began by composing essay questions. One question was required in semester I, and two questions in semesters II and III. While students reviewed all the questions posted online, the instructor reviewed and edited questions when necessary. Next each student selected (in semester I) or was assigned one question (in semester II) or two questions (in semester III) and answered them. Answers were restricted to 1250 words. Fourth, the student who created the question graded the answer and provided a justification of the grading. Each grade was broken into six sub-grades for the following criteria: framing of the answer, considering all sides of the issue, use of references, synthesis of these references, justification of points made, and overall answer quality. The professor gave detailed grading guidelines for each criterion. Students had access to these from the start of the exam process, so they knew what level of detail would be expected. As an optional step, Ph.D. students enrolled in the course did a second round of grading (not shown in Figure 1), and then the instructor provided a final grade. If the scores of the grader and second round reviewer were within a few points of each other, the instructor assigned the higher score. If the two disagreed, the instructor graded the question himself. Lastly students could contest the grades by evaluating their own answers and providing a detailed justification. The instructor resolved disputes by either keeping or adjusting the grades.

Two important features that are worth emphasizing are the disputes and the ability to read peers' comments throughout the exam process. Disputes are an especially important feature, as they help ensure the fairness of the participatory exam, especially when instructors do not review answers where the two student graders are in basic agreement. If a student believes the peer evaluations were incorrect a dispute will ensure that the instructor focuses adequate attention to this particular answer. Another important feature that the ALN environment provided is that students could read everything their peers posted—questions, answers, grades and disputes—and many did. We believe this is a vital aspect of constructivist learning as students think over their peer's understandings. McConnell, for example, found that knowing that their peers would read their assignments motivated student learning [21].

Based on students' feedback elicited from the first survey and instructor's experience, the following changes were made in the second semester:

• The process was made anonymous;
• Two essay questions were solicited instead of one;
• Students were assigned a time-slot to reserve two questions for answering;
• The entire process was streamlined to 3.5 weeks.

Unfortunately, there was a system crash during the question reservation procedure in semester II. To try to recover from lost time and lost postings, the instructor asked students to select and answer only one question each and graders had to be reassigned to make sure every student graded one question.

In semester III, the following modifications were made to the online exam process based on the experiences from the first two semesters:

• Students were asked to post two questions on two different topics in the course;
• To avoid system congestion, the instructor assigned two questions to each student. Students were asked to look at all the questions to avoid posting identical questions;
• The whole process was streamlined to three weeks.

In addition, a traditional in-class exam was given to half of the students (the control group) in semester III. The instructor, matching the number of questions that students in the participatory exam had, designed two open-ended questions. That exam was closed book, but students were allowed to bring six pages of notes for the traditional exam. Students had three hours to answer the two questions, and the maximum length of the answers was limited to five pages, which was roughly equivalent to the 1250 word limit of the participatory exam. Grading guidelines equivalent to those of the participatory exam were provided in advance to students. Thus the same criteria were used in grading both exams.

C. ALN Tools

Virtual Classroom™ (VC) and WebBoard™ were the two asynchronous conferencing tools used to facilitate the participatory exam processes discussed in this paper. VC was used at NJIT for over a decade and was where the first exam was conducted. WebBoard™ use began in semester II and served as the environment for the second and the third semesters. Both systems have "conferences", which are separate threaded discussion areas for a topic where students can post, read and reply to comments of others. In semester I three conferences were constructed on VC for the exam. One dealt with administrative information on the exam process; one served as the main conference where all the questions, answers, and grades were posted; and the last one was reserved for Ph.D. students and the instructor for managing the second round of grading. In semesters II and III, one main conference was created in WebBoard for the main exam activities; administration information and students' feedback were posted in the regular general administration and feedback conferences for that semester. In all three semesters, several important announcements, such as how to compose a question and grading procedures, were posted on the course web site.

Utilizing ALN tools in implementing the participatory exam provided many benefits. In addition to allowing students to read their peers entries as discussed earlier, online tools streamline the management of the exam process for both instructors and students (e.g., making postings immediately available for processing among a large number of people and tracking participant progress in the process). Online tools also facilitate anonymity. Intermediate graders used online tools to discuss grading challenges. Furthermore, ALN provides the means for distance and online students to utilize this alternative exam approach.

III. RESEARCH MODEL, HYPOTHESES, AND RESEARCH DESIGN

A. Research Model, Research Questions, and Hypotheses

As discussed in previous sections, the participatory online exam can be considered as a unique online learning process. Based on the Online Interaction Learning Model by Benbunan-Fich, Hiltz, and Harasim [22], an input-process-output collaborative examination research model is presented in Figure 2.

• Independent variable: exam mode (participatory exam vs. traditional exam)
• Intervening variables: exam process enjoyability and exam grade
• Dependent variables: perceived learning, perceived fairness in grading, and recommendation for future use or preference for the exam mode.

In addition, course characteristics and student characteristics are examined as attribute variables in the research model.

It is necessary to point out that students received their grades before they answered the survey, therefore grade is placed as an intervening variable that may affect perceived enjoyability in the exam process and exam outcomes (perceived learning, perceived fairness in grading, and recommendation).

Figure 2. Research Model

First, students' feedback on the participatory exam is investigated. No specific hypotheses are stated for questions 1–5 as not enough previous studies can be drawn upon:

Q1. Did students enjoy the participatory exam process?

Q2. How do students' grades distribute in the participatory exam?

Q3. Did students think they learned through the participatory exam process?

Q4. Did students think the participatory exam grading system was fair?

Q5. Would students recommend the participatory exam in future courses?

Questions 6 and 7 investigate the relationship among students' feedback on the participatory exam with specific hypotheses below:

Q6. Is there an association between grades and students' perceptions of the participatory exam process?

H1: Students' grades will positively correlate with the perceived enjoyability of the participatory exam process.

H2: Students' grades will positively correlate with their perceptions of the participatory exam outputs

(H2a: perceived learning; H2b: perceived fairness in grading; H2c: recommendation).

Q7. Is there an association between enjoyability of the exam process and students' perceptions of the participatory exam process?

H3: Students who perceive higher enjoyability will report higher perceptions of the participatory exam outputs

(H3a: perceived learning; H3b: perceived fairness in grading; H3c: recommendation).

H4: Students who report higher perceived learning will recommend the participatory exam process more.

H5: Students who report higher perceived fairness in grading will recommend the participatory exam process more.

Questions 8 and 9 examine student and course characteristics and their effects on students' perceptions of the participatory exam. No specific hypotheses are designed as not enough previous studies can be drawn upon:

Q8. Are there differences in students' perceptions of the online participatory exam between courses that are offered FTF vs. DL?

Q9. Are there differences in students' perceptions of the online participatory exam among students with different backgrounds (e.g., age, gender, etc.)?

Second, the participatory exam is compared with the traditional exam in a field experiment in the third semester. Question 10 compares the traditional exam with the participatory exam with the specific hypotheses listed. Based on previous studies on online learning and online assessment, it is hypothesized that the participatory exam will be superior to the traditional exam, except for perceived fairness in grading, which was a main concern in the first two surveys [16, 17].

Q10. How do students' perceptions differ between those taking the traditional exam and the participatory exam?

H6a: Students who participate in the participatory exam will have higher enjoyability of the process than students in the traditional exam.

H6b: Students who participate in the participatory exam will receive higher grades from the instructor for their answers than students in the traditional exam.

H6c: Students who participate in the participatory exam will report higher perceived learning than students in the traditional exam.

H6d: Students who participate in the participatory exam will report lower perceived fairness in grading than students in the traditional exam.

H6e: Students who participate in the participatory exam have higher overall preference for the exam than students in the traditional exam.

Hypotheses are shown in the research model in Figure 2 with corresponding numbers.

B. Evaluation Method: Surveys and Field Experiment

To evaluate the participatory online examination processes, surveys were conducted within a couple of days after the examination processes finished. The questionnaires were distributed in the classroom to FTF students and were put on the course web site for downloading for DL students. In semester III, two different questionnaires were designed for the online exam and the in-class traditional exam respectively. Table 1 shows the number of students in each semester and the response ratio in all three surveys.

Table 1. Number of Subjects in the Three Studies

As can be seen, the return rates in all three surveys are favorable. The instructor assigned three extra credit points towards the course grade for participation in the survey. In semester III, both questionnaires were made available to be directly filled out online; this convenience resulted in high response rates.

Questions in all three surveys for the online exams were kept the same wherever possible for comparison, and changes were made only to accommodate the minor changes in the exam process each semester. In addition, the questionnaire used in the in-class traditional exam in semester III has matching questions to the online collaboration exam questionnaire. In this way, comparison can be made in the data collected from all three semesters, as well as in-depth comparison between the two exam modes in the third semester. Some questions were adapted from the "Course Questionnaire — Virtual University Project" [1]. 5-point Likert-type scales were used in the surveys. Results in the following sections are scored on a basis of 1–5, where SA: Strongly Agree=5; A: Agree=4; N: Neutral=3; D: Disagree=2; SD: Strongly Disagree= 1.

IV. RESEARCH FINDINGS I — Survey Results of participatory exams in Semesters I, II and III

The raw data, gathered over three semesters, were analyzed using SPSS™. This section reports the survey results of the online participatory exams in the three semesters. Factor analysis is discussed in subsection A, whose results (factors) are used to organize the results of the participatory exam (Subsections B–F). Subsection G presents a correlation analysis among the factors. The final subsections analyze the effects of course characteristics and student characteristics.

A. Factor Analysis and Reliability of Scales

Factor analysis is used to uncover the common underlying factors in students' perceptions of the participatory exam. Principle component factor analysis with Oblimin rotation method was used as factors are not expected to be orthogonal, or unrelated, as the research model shows. Using the 190 subjects' responses in the participatory exam mode, three factors are found corresponding to the constructs in the research model: exam process enjoyability (4 items), perceived learning (9 items), and perceived fairness in grading (6 items). Reliability analysis of the factors show Chronbach's alpha above the .7 level, except for the enjoyability scale (at the .69 level). See the alpha value for each construct below tables in Subsection B.

The following subsections are organized based on the main constructs in the research model and factor analysis results, including: enjoyability of the exam process, grades, perceived learning, perceived fairness in grading, and recommendation. Because of the procedural adjustments in the three participatory exams, results are analyzed individually for each semester rather than combined in order to allow detailed examination. Further, to find out whether there are significant differences among the three semesters, a non-parametric test (Kruskal Wallis) is used and results are reported. A non-parametric test is used because normality tests reveal that the survey results in participatory exams are not normally distributed.

B. Enjoyability of the Exam Process

Table 2 explores Q1: did students enjoy the participatory exam process? Students reported that the exam provides a comfortable timeframe, flexibility in organizing their resources, is less stressful, and they enjoyed the exam process in all three semesters. However, the results from semester II are less positive than the other two semesters. The response to the question “the exam provided a comfortable timeframe” was significantly lower in semester II (mean=3.3 out of 5) than the other two semesters (mean=4.0, 3.9) at .05 level. A system crash in the middle of the exam process and the adjusted schedule are speculated to be related to the survey results in semester II. This result demonstrates that the online participatory exam is vulnerable to technological glitches. On the other hand, though results were less positive with system problems, they were still positive, which is important to know in a world in which malicious attacks on systems may bring them down at any time.

Table 2 . Enjoyability of the Exam Process

C. Grades

The unique grading phase of the exam features three levels of grading: student grading, intermediate grading, and instructor's grading. To answer Q2: how do students' grades distribute in the participatory exam, three grades are compared and normality tests are conducted. Figure 3 shows distributions of grades as students reported in the three questionnaires. Note the maximum grade for student grading was 100 points, 105 points for intermediate grading, and 115 points for instructor's grading. Extra credit points were awarded by intermediategraders and the instructor in accordance with the instructor's grading guidelines.

Figure 3. Distribution of the Three Grading Levels

As can be observed in Figure 3, most students gave each other high scores; however, intermediate grading and the instructor grading have increasingly enhanced the even distribution of grades compared with student grading. This is confirmed in the Skewness and Kurtosis scores in table 3 (Skewness= -2.21, -1.9, -0.89; Kurtosis=5.91, 4.55, 1.08). Normality test shows (Table 3) that all three grades are not normally distributed (Shapiro-Wilk test, Sig=.000).

Table 3. Grade Distribution Normality Tests

D. Perceived Learning

To investigate Q3: Did students think they learned through the participatory exam process, answers to the nine items related to perceived learning are analyzed. The majority of students in all three semesters agreed that all exam phases were part of the learning process (Table 4). Students learned from making up questions, reading all questions online designed by their peers, and grading. It is interesting to note the steady increase in perceived learning in the question composition stage (mean= 3.41, 3.52, 3.62). This may be attributed to the fact that students were only required to make up one question the first semester, while in the second and third semesters, the number increased to two. Also note that the perceived learning from looking at all the other questions is the highest in the second semester, and the lowest in the third semester. The difference is significant at the .05 level using the Kruskal Wallis test. The higher perceived learning in semester II may be attributed to the fact that two questions were solicited rather than one in semester I, and students were initially required to select two questions on their own for answering. The lowest perceived learning in semester III may be due to procedure change where students were assigned questions rather than picking questions themselves. In terms of the grading phase, it is perceived as the most valuable learning activity in semester I (mean=3.82) while in semester II it ranks the lowest (mean=3.41). This result may be a consequence of the system crash and the reassignment of half of the graders to questions which they did not author.

Table 4. Perceived Learning (Part I)

Students were also asked to compare this exam process with the traditional one on questions related to learning (Table 5). More than half of the students rated the online exam better than the traditional exam. Yet it appears that slightly fewer students thought so in the third survey than the previous two surveys. T he answer to the question "I mastered the course material" is most positive in semester II and least positive in semester III. The difference is significant at .05 level using Kruskal Wallis test. This may be attributed to the fact that when the online exam was launched in Fall 1999, it was a totally new experience to students. Since the first two surveys demonstrated favorable results, some other classes adopted this process. Therefore when the third survey was conducted in Fall 2000, this exam was not novel to some students anymore. This can be seen as at least 30% of students' reponses to all the questions are "no difference" between online and traditional exam in the third survey. It shows they did not perceive the process to be as stimulating in motivating learning and extra work as before.

Table 5. Perceived Learning (Part II)

E. Perceived Fairness in Grading

Table 6 reports students' responses to the participatory exam's quite unique grading process to investigate Q4: Did students think the participatory exam grading system was fair? The answer to the question “the grading process was fair” fluctuated around neutral for the three semesters. Part of the reason for the unfavorable attitude towards the grading process in the second semester may be due to the reassignment of graders. Students continue to question the grading capability of their peers (mean= 3.07, 2.73, 2.63), and express increasing favor towards having the instructors do all grading (mean=3.27, 2.35, 2.25), which is significant at the .05 level using the Kruskal Wallis test. It is noticeable that second round (Ph.D. student) graders' ability to improve grading fairness was agreed to by the majority in all three surveys (mean=3.25, 3.31, 3.12). Although this may suggest that having qualified second round graders was a good approach in this exam process, it is not generally practical to find Ph.D. students for every class. In terms of question design, students' assessment of their own abilities in designing good questions to measure the learning objectives of the course dropped from 3.22 in the first two semesters to 2.65 in the third semester. The difference is significant at the .05 level using Kruskal Wallis test. Yet considering students in the third exam were only required to read the two questions assigned to them, they may not have read enough questions to estimate question quality.

Table 6 . Perceived Fairness in Grading

F. Recommendation

Overall, students were asked whether they would recommend the participatory exam process to be used in other courses in the future (Q5). Results (Table 7) from the three surveys show an increasingly strong recommendation of the process to be used in the future, though the mean difference is not significant.

Table 7. Overall Recommendation of Using the Process for Other Courses

Thus, the study indicates that students report subjective favorable attitudes with the participatory exam in general.

G. Correlations Among Constructs

Correlation analysis was conducted to find the relationships among the main factors. First, analysis is conducted to investigate (Q6): is there an association between exam grades and students' perception of the participatory exam process. It is hypothesized that students' exam grade is positively correlated with students' perceived enjoyability of the exam process (H1). In addition, it is hypothesized that students who receive higher grades would report higher perceived learning through the exam process (H2a), higher perceived fairness in grading (H2b), and higher recommendation to use the exam in the future (H2c). In this analysis, the average of the student grader's grade, the intermediate grader's grade, and the instructor's final grade was used as the grade for the participatory exam students. Given that the average grade is not normally distributed, nonparametric correlation analysis is used. Results show exam grade is significantly correlated with perceived learning (Spearman's rho= .234, p<.01), perceived fairness in grading (Spearman's rho= .360, p<.01), and recommendation (Spearman's rho= .182, p<.05), although the correlations with perceived learning and recommendation are weak. The correlation with the enjoyability factor is not significant. The result supports hypotheses 2a, 2b, and 2c, but not hypothesis 1.

Second, analysis was conducted to investigate (Q7): Is there an association between enjoyability of the exam process and students' perception of the participatory exam process? It is hypothesized that students who perceive higher enjoyability report higher perceived learning (H3a), higher perceived fairness in grading (H3b), and higher recommendation to use the exam in the future (H3c). Results show the enjoyability factor is significantly correlated with perceived learning (Spearman's rho= .396, p<.01), perceived fairness in grading (Spearman's rho = .209, p<.05), and recommendation (Spearman's rho= .575, p<.01), although the correlation with perceived fairness is weak. The results support hypotheses 3a, 3b, and 3c.

Third, the correlations among the output variables are investigated. It is hypothesized that recommendation of the process is positively associated with perceived learning (H4) and perceived fairness in grading (H5). Results show that recommendation is significantly correlated with perceived learning (Spearman's rho= .545, p<.01) and perceived fairness in grading (Spearman's rho= .432, p<.01). The results support hypotheses 4 and 5.

H. The Effects of Course Characteristics

In order to find out whether the mode in which the sections were offered (DL vs. FTF) had any effects on students' perceptions of the participatory exam (Q8), Mann-Whitney tests were conducted on dependent variables. In the majority of the variables, there was no significant difference between the two exam modes. Table 8 shows the four items for which significant differences exist. In the two items related to enjoyability (flexibility in resources, enjoyed the process), DL students reported higher positive perceptions than the FTF students in the three semesters in most cases, and the differences in both items are significant in semester I. In semester II, DL students reported that they enjoyed the exam less than the FTF students. This may be attributed to the system crash and the procedural changes, which may have caused more stress on DL students than FTF students. In the two items related to perceived learning (value others' viewpoints, stimulated to read more), DL students reported less learning in valuing others' points of view (significant at .01 level in semester II) but being more stimulated to read more (significant at .05 level in semester III) through the exam. The effects are consistent in all three semesters. This result may suggest that the participatory exam encouraged DL students to read more on their own while it encouraged FTF students to learn more from other students.

Table 8. Comparison between Distance Learning and Face-to-Face Courses

I. The Effects of Students' Characteristics

In addition to course sections, students' background information was also collected and analyzed to examine whether prior differences among students affect students' perceptions of the participatory exam (Q9). Data collected include age, gender, English as the native language, prior distance learning experience, course expectations (e.g., expected course grade, expected course difficulty level), etc. While other background characteristics do not yield significant or consistent impacts on the results, English as native language does show a significant and repeated effect on students' perceptions of learning throughout the three studies. Given that the research was conducted in a graduate program in an urban university located in the metropolitan East coast region, non-native English speaking students outnumbered native English students each semester (57.1%, 65.3%, 51.9% in semesters I, II, III respectively). Mann-Whitney tests were conducted on all the dependent variables. While there was no significant differences found in grades, exam process enjoyability, perceived fairness in grading, or recommendation of the process to be used in the future, significant differences were found in seven out of the nine items on perceived learning. Table 9 shows the seven items that have significant differences between students with or without English as the native language. In all semesters, students who are not native English speakers reported higher perceived learning on all seven items than students who are native English speakers, except for one item (stimulated to read more) in semester III which was not significant. Noticeably in all semesters, non-native English speaking students reported significantly higher perceived learning in making up exam questions than English speaking students. Non-native English speaking students also reported higher perceived learning in valuing others' points of view than English speaking students through the exam process, and the results were significant in semesters II and III.

Table 9. Comparison of Students With or Without English as Native Language

V. RESEARCH FINDINGS II — experiment Results comparing the participatory exam and the traditional exam in Semester III

The analysis in Section VI draws from data of the 190 returned questionnaires in the participatory exam in the three semesters. This section reports the field experiment results for the third semester, where 52 questionnaires were collected from the participatory exam and 41 from the traditional exam. The question is: (Q10) How do students' perceptions differ between those taking the traditional exam and the participatory exam? It is hypothesized that compared with the traditional exam, students in the participatory online exam mode report higher enjoyability of the process (H6a), receive higher grades from the instructor (H6b), have higher perceived learning (H6c), lower perceived fairness in grading (H6d), and have higher overall preference for the exam mode (H6e).

The common items used in the two questionnaires for the two conditions are used for comparison. Similar factors as from the participatory exam surveys are extracted, such as enjoyability of the exam process (2 items: I felt under much pressure taking the exam this way; I enjoyed the exam process), perceived fairness in grading (2 items: I felt the grading process was fair; The grading criteria given by the professor were explicit enough), and perceived learning (6 items: as listed in column one of Table 5). Figure 4 compares the means of all the items organized by main factors in the two exam modes.

Figure 4. Comparison between Participatory and Traditional Exam (Item-level)

Table 10 shows the grand means for each of the main factors, as well as grades awarded by the instructor. Normal distribution is acceptable in this data set, and T-test is used to test the significance of differences. Results show compared with the traditional exam, students in the participatory online exam condition significantly enjoyed the process more (mean=3.11, 3.46, p<.05), and significantly prefer to take that exam mode (mean=2.78, 3.33, p<.05). This supports hypothesis H6a and H6e. Compared with the traditional exam, students also received significantly higher grades for their answers from the instructor (mean=70.9, 76.8, p<.05), which supports H6b. Although students in the participatory exam condition have lower perception of the fairness in grading as predicted in H6d, (mean=3.62, 3.40), this result is not significant. Our prediction of higher perceived learning (H6c) is not supported (mean= 3.81, 3.65), and the result is not significant.

Table 10. Comparison between Participatory and Traditional Exam (Factor-level)

Summarizing the results as discussed in Sections IV and V, Figure 5 presents the significant differences and correlations among the variables in the research model.

Figure 5. Research Model with Results

VI. DISCUSSION AND Conclusions

Students' answers to the two-opened questions in the surveys provide further insights into what they liked and disliked most about the participatory exam process. For example, the most appreciated features of the participatory exam include flexibility in time and resources, and less pressure (e.g., "The best part about the online exam process was the capability and the time to thoroughly review and research the articles that were covered during the semester," "It gives me less pressure, and I think it is more flexible to really learn the knowledge"). Related to the flexibility is higher perceived learning (e.g., "I like the idea that I can take my time thinking about the answers instead of jumping to a quick conclusion," "The great thing is that we are given time for critical thinking"). In terms of the open-book exam format, students commented that this better resembles real-world situation when "faced with a problem to solve", and "memorization skills (were) not a requirement for this type of exam which most traditional closed-book would have required." Using ALN systems, students also valued the exam experience as supporting learning from each other, from question design (e.g., "(the exam allowed me to) view a range of questions over different fields of information systems"), to answering (e.g., "I gathered others view on a particular subject"), to grading (e.g., "the views of the person answering questions also made me think and look into the course materials to evaluate the answer"). To summarize the advantages of the online exam using student's comments, the participatory exam "provided the true experience of distance learning."

In describing what they liked least about the process, student concerns center around the grading process. For example, students commented: "some of the grading is very subjective based on the answer," "Students should not be allowed to grade papers as they are not necessarily capable of providing the correct judgment on what constitutes a right answer for the questions they set." Despite the fact that students tend to give better scores than the instructor (Figure 3), students continue to question the grading capability of their peers, and express increasing favor towards having the instructors do all grading. This seems to suggest that the perceived fairness in grading relies more on the authority of the grader than the grades students receive. While the validity and reliability of peer grading has been established in previous meta-analysis studies [23, 24], students still seem to lack confidence in themselves in grading other's answers, nor do they trust their peers for grading their answers. Yet the ability to critically evaluate other's work is essential in people's professional life [25], and should be fostered in school education. Indeed, some students reported that the grading process provided "the opportunity to learn about how to grade an answer", and allowed students to "see both sides of the learning process: one side, the view of a student to who is trying to pass a test, and the other side is from the prospective of the professor."

One approach would be for the instructor to evaluate the grading. In preparation, he or she could prepare several high and low quality answers for training students. Training students to be better graders could ameliorate the feeling that grading performed by many people would result in uneven qualify. Training, of course, is used by major testing services to ensure uniform grading in standardized test essay questions.

Another, perhaps counter-intuitive approach would be to structure ways for students to dispute their grade if they feel it is too high. Doing so would be instructive for the disputing student, who would have to justify why his or her work was overrated, for the graders who potentially overrated, and the instructor. One could award bonus points greater than the grade if the dispute were successful. The graders could optionally lose points for grading poorly.

In addition to grading, students also pointed out a number of areas for future improvement. For example, the exam process was "too lengthy" and "time consuming." Some pointed out that the syllabus needs to be adjusted to better incorporate the exam with other course assignments and projects. A few comments were made calling for "a greater involvement on the part of the instructor" and "more interactions with the instructor." Although plagiarism was not found in the participatory exams in the three semesters, some students suggested that "the grading should be more stringent."

The results of comparing DL to FTF classes reveal that in general the participatory exam worked equally well in both DL and FTF classes. Some differences were found such as DL students reported enjoying the exam more, and were more likely to be stimulated to read more, while FTF students reported being more likely to learn to value others' viewpoints. The participatory exam also worked equally well for students with different backgrounds, such as age, gender, prior distance learning experience, and course expectations. Differences were found in perceived learning related to English as the native language; students whose English is not their native language reported higher perceived learning from the online exam. The result suggests that the unique features of the online participatory exam, such as a democratic atmosphere [26], a culture of reflection [27], active construction of knowledge [28], and less time pressure may be more beneficial to those non-native English speakers.

Removing the ALN environment would seriously impoverish the collaborative exam. Reading how peers crafted questions, answers and grade justifications is an integral component of how students construct knowledge. Removing the technology also would slow down the process, and complicate information passing and editing questions. It also would hamper participation by online students.

Further improvements to the ALN environment could better support student learning and instructor management. Both students and instructors could utilize repositories of exam instructions, questions, answers and grade justifications. Students could study these to learn from prior efforts. Repositories could prove useful training tools; instructors could point to both good and bad examples, and these could be annotated and discussed online. Instructors also may find others' efforts useful in designing and refining exam components for their own classes.

Instructors further could use a combination of templates, guidelines and training examples to guide students in question design, how to answer questions appropriately, how to justify grades, and how to craft a dispute. Templates, for instance, explicitly show categories that a proper question, answer or grading justification should incorporate.

Learning could be enhanced by having students work in teams to design, answer and grade questions. One area of research currently underway is based on collaborative learning theories [29] and online interaction studies in ALN research [4]. The aim is to investigate whether the participatory exam process as described in this paper can be further enhanced through small group activities.

In conclusion, this paper reported longitudinal evaluation results for the online participatory exam, which inherently embraces constructivism by actively involving students in the entire exam life cycle. The overall favorable results revealed in the evaluation suggest that the ALN environment is not only capable of supporting constructive learning activities, but in particular online exam processes to further enhance learning. This challenges researchers and instructors to reconceptualize other traditional teaching aspects as learning activities that fully utilize ALNs potential for online learning.

VII. ACKNOWLEDGEMENTS

Many thanks to Murray Turoff, Yooncheong Cho and Kung-E Cheng for their advice and help in earlier phases of this research. We gratefully acknowledge partial funding support for this research by the United Parcel Service Foundation, the New Jersey Center for Pervasive Information Technology, the New Jersey Commission on Science and Technology, and the National Science Foundation under grants IIS-0135531, DUE-0226075 and DUE-0434581, and the Institute for Museum and Library Services under grant LG-02-04-0002-04. Portions of this paper are adapted from preliminary results published in the Proceedings of the 2004 Americas Conference on Information Systems [15], Proceedings of the 34 th Hawaii International Conference on Systems Science, 2001 [16], and the Proceedings of the 2000 Americas Conference on Information Systems, 2000 [17].

VIII. REFERENCES

1. Hiltz, S. R. The Virtual Classroom: Learning without Limits Via Computer Networks. Norwood, NJ: Ablex, 1994.
2. Allen, I. E. and J. Seaman. Sizing the Opportunity: The Quality and Extent of Online Education in the United States 2002 and 2003. Needham, MA: Sloan-C, 2003. Online: www.sloan-c.org.
3. Castro, M., A. Lopez-Rey, C. M. Perez-Molina, et al. Examples of Distance Learning Projects in the European Community. IEEE Transactions on Education 44(4): 406–411, 2001.
4. Hiltz, S. R. and R. Goldman, eds. Learning Together Online: Research on Asynchronous Learning Networks. Mahwah, NJ: Lawrence Erlbaum Associates, 2005.
5. Curtis, D. D. and M. J. Lawson. Exploring Collaborative Online Learning. Journal of Asynchronous Learning Networks 5(1): 2001.
6. Jonassen, D. H. Thinking Technology. Educational Technology: 35–37, January 2003.
7. Vygotsky, L. Mind in Society: The Development of Higher Psychological Processes. Cambridge: Harvard University Press, 1978.
8. Leidner, D. E. and S. L. Jarvenpaa. The use of information technology to enhance management school education: A theoretical view. MIS Quarterly 19(3): 265–291, 1995.
9. Huba, M. E. and J. E. Freed. Learner-Centered Assessment on College Campuses: Shifting the Focus from Teaching to Learning , 1st ed. Pearson Allyn & Bacon, 1999.
10. Angelo, T. A. and K. P. Cross. Classroom Assessment Techniques: A Handbook for College Teachers, 2nd ed., 448. Jossey-Bass: 1993.
11. Kwok, R. C. W. and K. Ma. Use of a Group Support System for Collaborative Assessment. Computers & Education 32(2): 109–125, 1999.
12. Wilson, E. V. Examnet Asynchronous Learning Network: Augmenting Face-to-Face Course with Student-Developed Exam Questions. Computers & Education 42: 87–107, 2004.
13. Shindler, J. Examining the Soundness of Collaborative Essay Exams in Teacher Education Courses. National Forum of Teacher Education Journal 12(3): 2003.
14. Liu, E. Z.-F., S. S. J. Lin, C.-H. Chiu, et al. Web-Based Peer Review: The Learner as Both Adapter and Reviewer. IEEE Transactions on Education 44(3): 246–251, 2001.
15. Shen, J., K.-E. Cheng, M. Bieber, et al. Traditional in-Class Examination vs. Collaborative Online Examination in Asynchronous Learning Networks: Field Evaluation Results. Americas Conference on Information Systems 2004. New York City, NY: 2004.
16. Shen, J., S. R. Hiltz, K.–E. Cheng, et al. Collaborative Examinations for Asynchronous Learning Networks: Evaluation Results. Proceedings of the 34th Hawaii International Conference on Systems Sciences. Maui, Hawaii: IEEE Computer Society Press, 2001.
17. Shen, J., K.-E. Cheng, Y. Cho, et al. Evaluation of an On-Line Collaborative Examination Process. Proceedings of the 2000 Americas Conference on Information Systems. California, 2000.
18. Shen, J. Collaborative Examinations in Asynchronous Learning Networks: Field Experiments on Collaborative Learning Through Online Assessments, Ph.D. Dissertation, Department of Information Systems, NJIT, May 2005.
19. Wu, D., M. Bieber, S. R. Hiltz, et al. Constructivist Learning with Participatory Examinations. Proceedings of the HICSS 37th Hawaii International Conference on Systems Sciences. Big Island, Hawaii: IEEE Computer Society Press, CD-Rom, 2004.
20. Leidner, D., and M. Fuller. Improving student learning of conceptual information: GSS supported collaborative learning vs. individual constructive learning. Decision Support Systems 20: 149–163.
21. McConnell, D. Examining a Collaborative Assessment Processes in Networked Lifelong Learning. Journal of Computer Assisted Learning 15(3): 232–243.
22. Benbunan-Fich, R., S. R. Hiltz, and L. Harasim. The Online Interaction Learning Model: An Integrated Theoretical Framework for Learning Networks. In Hiltz, S. R. and R. Goldman (eds.), Learning Together Online: Research on Asynchronous Learning Networks. Mahwah, NJ: Lawrence Erlbaum Associates, 2004.
23. Topping, K. J. Peer Assessment between Students in College and University. Review of Educational Research 68(3): 249–276, 1998.
24. Falchikov, N. and J. Goldfinch. Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks. Review of Educational Research 70(3): 287–322, 2000.
25. Karoly, L. A. and C. W. A. Panis. The 21st Century at Work: Forces Shaping the Future Workforce and Workplace in the United States. RAND Labor and Population, 2003.
26. Harasim, L. On-Line Education: Perspectives on a New Environment. New York: Praeger, 1990.
27. Hiltz, S. R. The Virtual Classroom: Using Computer-Mediated Communication for University Teaching. Journal of Communication 36(2): 95–104. 1986.
28. Picciano, A. G. Beyond Student Perceptions: Issues of Interaction, Presence, and Performance in an Online Course. Journal of Asynchronous Learning Networks 6(1): 21–40, 2002.
29. Swan, K. and P. J. Shea. The Development of Virtual Learning Communities In Hiltz, S.R. and Goldman (eds.), Learning Together Online: Research on Asynchronous Learning Networks. Mahwah, NJ: Lawrence Erlbaum Associates, 2004.

IX. About the Authors

Jia Shen is an Assistant Professor in the School of Management at New York Institute of Technology starting in September 2005. She completed her Ph.D. in the Information Systems Department at New Jersey Institute of Technology in May 2005. Her current research interests include Computer Supported Cooperative Work (CSCW), Asynchronous Learning Networks (ALN), Human Computer Interaction, and Knowledge Management. She has presented papers at many conferences including AMCIS, HICSS, and CHI. She may be reached through http://iris.nyit.edu/~jshen/.

Michael Bieber is a Professor in the Information Systems Department of the College of Computing Sciences at the New Jersey Institute of Technology. He is conducting research in several related areas: participatory learning, hypermedia functionality, automatically generating links and services for Web applications and for digital libraries, relationship analysis (as part of the software engineering process) and virtual communities. He co-directs NJIT's Collaborative Hypermedia Research Laboratory. He holds a Ph.D. in Decision Sciences from the University of Pennsylvania. He may be reached through http://web.njit.edu/~bieber.

Starr Roxanne Hiltz is a Distinguished Professor in the Information Systems Department of the College of Computing Sciences at the New Jersey Institute of Technology. She conducts research on applications and social impacts of computer technology, publishing widely in journals including JMIS, MISQ, Communications of the ACM, and Management Science. Her research interests currently include Group Support Systems, Asynchronous Learning Networks, and Pervasive Computing. She co-directs NJIT's Collaborative Hypermedia Research Laboratory. She may be reached through http://web.njit.edu/~hiltz.