Inside Online Learning: Comparing Conceptual and Technique Learning Performance in Place-based and ALN Formats

Download PDF version:

Abstract
Online learning is coming of age. 'Traditional' universities are embracing online components to courses, online courses, and even complete online programs. With the advantage of distance and time insensitivity for the learning process, there appears to be a growing sense that this form of teaching and learning has strong pedagogical merit. Research has shown that students do comparatively well in this new format. There is, however, a lack of evidence illustrating particular strengths and weaknesses of online teaching and learning. This paper discusses experiences with a single course taught using two forms: (1) traditional place-based, and (2) a form of asynchronous learning network (ALN) defined as interactive virtual seminars. Differences in learning performance are tested using longitudinal observations. In a course comprised of both conceptual material and the application of techniques, the students performed overall equally well in either place-based or virtual format. Their degree of learning, however, differed significantly between conceptual and technique-based material. Implications are promising, showing that there are relative strengths to be exploited in both place-based and virtual formats.

Keywords
Comparison, E-Learning, Learning Theory, Longitudinal Experiment

I. Introduction

The Internet changed everything. Post-secondary education was traditionally offered as a place-based, classroom-oriented phenomenon, with 'distance education' available as a correspondence option. Since the revolution of email and the world wide web in 1994, universities have changed dramatically in pedagogical structure, embracing telecommunications-enabled teaching and learning opportunities. This radical change, begun by 'evangelists' who simply believed there could be a new way to teach and learn, has been followed by evaluative research, anecdotal evidence, and experimentation suggesting that alternative learning environments could provide comparable, even preferable, learning outcomes [3].

While evidence is encouraging for proponents of ALN, the results are preliminary and there remains much to be learned. This paper discusses the comparison of a course offered in two formats that alternated between a traditional place-based and an ALN format over a four-year period. Identical material and grading instruments were offered across the two formats providing an opportunity to examine differences between the two formats. The findings illustrate that students provide similar overall learning performance scores from either format. They do not, however, develop skills in the same way. These results suggest two key conclusions. First, different formats give significantly different results when comparing the learning of conceptual material and the ability to apply techniques. Second, purely ALN-based courses and place-based offerings could benefit from further research into the effectiveness of alternative learning approaches.

II. Background: Course History

The course outlined in the study is a 3^rd year Business Administration course which teaches elements of Systems Analysis and Design to (mostly) declared Business majors taking either an Accounting or a Management Information Systems specialization. The course was considered for adaptation to an ALN format in 1995 due to a perceived fit between demand and methodological possibilities. Many of the students who take this course are co-operative education students who expressed interest in taking one course during a co-operative work term, but didn't want to spend the time commuting weekly to a university campus located outside of the downtown core. Additionally, the material outlined appeared, heuristically, to be a good fit for an attempt at an online offering.

The course was first offered in a completely online format in the Spring term of 1996. The instructor offered two sections per year, one place-based the other as an ALN. The ALN was formed as a 'virtual seminar,' where students take the course completely online in the standard 13-week term. Topics are covered weekly with assignments due in the same time frame as the place-based format and a final examination during the regular exam period. The student grades in the online format were equivalent to those of earlier place-based formats, so the decision was made to continue to offer this course online once out of three annual offerings. Other than routine checks of final grade distribution, no distinct experimental measures were made on the course. The evaluative instruments, namely assignments, tutorial exercises, and examinations were identical in format for either treatment.

The course has two distinct objectives. First, the nature and importance of the role of a systems analyst needs to be understood. Included in this understanding is the nature of the process of the analysis and design of information systems, audit and quality control approaches, and procedural fit within an organizational unit. This material is abstract and conceptual in nature. The second objective is to provide students with experience in the use of technical tools used by a systems analyst, such as Data Flow Diagrams, Entity-Relationship Diagrams, and tools in the Unified Modeling Language (UML). This material is focused on developing students' ability to apply specific techniques.  The course design followed a cognitive flexibility approach, where students were provided with background material through advanced readings and then applied these concepts in situations meaningful to them [9].

The first offering of the course in an ALN format was in the Spring term of 1996. Technology itself offered a significant challenge, but the course progressed and met expectations of both faculty and students. Preliminary feedback was anecdotal, but indicated that different parts of the course were both stronger and weaker for one treatment over another. A first positive outcome was noted very early on, in that the ALN format facilitated significantly greater opportunities for student participation and communication in the weekly discussion. For example, if 50 students were expected to contribute a weekly comment that would require approximately 10 minutes of class discussion, there was an opportunity to require the equivalent of 500 minutes discussion per week when total face-to-face meeting time in the synchronous format was only four hours, or 240 minutes. These 500 minutes were also exclusive of the lecture material. This also enabled considerable socialization among the students, the opposite of the anticipated result. For further discussion of this finding, see [8].

A further result was a perception by the instructor that the techniques were considerably more difficult to teach. This perception was supported by student feedback, where they expressed less confidence in their abilities to deal with the techniques, yet greater confidence in their overall understanding of which technique should be used and why these techniques were appropriate.

General feedback from the students confirmed these differences. The ALN format naturally lent itself to rich discussions of the conceptual material. Teaching in this format required considerably more effort on the instructor's part to communicate the technique-oriented material. Further, the students anecdotally expressed a greater lack of confidence in their ability to master techniques than witnessed in the synchronous, face-to-face group.

The results observed from the ALN course were not inconsistent with results reported in other studies ([2]; [3]; [4]). Discussions of the course material kept leading, however, to a sense of differing strengths of place-based and ALN approaches. These strengths appeared to focus on the student's ability to handle conceptual material related to the course versus applying techniques. It was these discussions that led to the development of the theoretical framework described in the next section and research question driving the empirical analysis.

III. Theoretical Framework

Mayer's learning process model provides the basis for the theoretical framework used in the study [5]. While Mayer's model was designed to describe the learning process in particular situations, we believe it can be extended to the larger task of course design and delivery. This model is extended later in this paper by further refining the "presentation method" construct. Mayer's original model is provided below in Figure 1.

Figure 1: Components of Mayer's Learning Model

A brief description of the model begins with the material to be learned. In our study, the material to be learned would be the content of the course. In particular we separate the content into two areas: conceptual material and the application of technique. We will show that an important difference between place-based and virtual seminars, in our study, is the different levels of conceptual and technical learning that occurs within groups.

The material must be presented to someone for learning to take place. The learner characteristics include attributes of individual learners such as ability, skill, and motivation.  These characteristics will influence the learning process, and eventually, the learning performance as measured by performance tests. 

The material to be learned is then presented using some sort of method. The method is described in more detail below. 'Prior knowledge may be given at the outset through instructional or text-based overviews, but these serve primarily as advance organizers for the acquisition of richer layers of meaning to come' [9]. A result of the process is the learning outcome. The learning outcome is not directly measurable, however, so we rely on measuring the learning performance. While the performance is not observable, the test scores relating to that learning performance are measurable. 

The method(s) used to present material to the learner is called the presentation method. In Mayer's research ([5]; [6]), the presentation method is the treatment variable. One group is provided with text, the other group with text and an additional resource (diagram, animation, or narration). For particular tasks that can be well controlled, the presentation method can be seen as one construct. When considering the wider task of course design and delivery, however, the presentation method must be separated into several constructs.

We argue that the presentation method can be split into two constructs: Presentation Format and Learning Approach. The Presentation Format describes the dissemination of material to be learned. There are four dimensions included in the Presentation Format of a course. These dimensions describe by whom, where, when, and how the material is presented. For a place-based course, the format is described as a teacher presenting both visual and verbal information to students who are located in the same room at the same time as the teacher. Virtual seminars, on the other hand, are presented by the teacher using primarily visual information and do not necessarily require teacher and students to be available in the same time and same place.  These differences are summarized below in Table 1:

Table 1: Dimensions in the Presentation Format Construct

Note that Table 1 above describes two formats for a single learning task, namely the delivery of lecture material. It is important to note that this model should be used to describe or compare the delivery of only a single learning task in a course. A learning task such as a virtual 'lecture' should not be combined with other tasks such as assignments, feedback, virtual office hours, or exams. Instead, each of these tasks should be treated separately so that different formats and material to be learned can be considered.  

Since we are considering a single learning task, it is important to situate the task in the learning environment. The Learning Approach construct does this by enabling the consideration of the reasons behind the learning task. There are several examples of learning approach. Two of the most frequently used are cognitive flexibility [9] and problem integration [7]. 

The Learning Approach is important because courses with the same material to present, using the same Format (for example place-based) might yield significantly different learning performance outcomes if different approaches are used. Since the Learning Approach can impact learning performance independent of the material to be learned and format used, it is an important construct in the model. The revised model of the learning process is provided below in Figure 2. 

Figure 2: Revised Learning Process Model

One of the objectives in outlining this Learning Process Model has been to provide a clear and comprehensive indication of the choices that teaching professionals face when designing and delivering courses. Having developed the model we suggest that course development requires the serious consideration of four primary constructs: (1) What is the material to be learned, (2) What approach to learning will guide the design, (3) What is the best format to present the material, and (4) What are the special characteristics of learners that should be kept in mind.

Having outlined the model, it is important to note that the Learning Process Model described above does not describe the entire learning environment. The learning process described above has been abstracted from other influences. In the real world, the learning process is situated in a learning environment that is subject to a large number of influences. Hiltz et. al. [3] provide a comprehensive list of these factors including technology, instructor skill and effort, student motivation, class size, and equipment access. These factors are important considerations in the development of any course. Any empirical study of alternative formats or learning approaches should consider these factors carefully.

IV. Research Question

The Learning Process Model discussed above provides a framework for the analysis of learning performance associated with alternative course designs. Our previous discussion relating to course history suggested an opportunity for comparing two alternative designs. Consistent with previous findings ([2]; [4]), we hypothesize no significant difference in terms of overall score between the two formats. 

H_A: There is no difference in overall results of a comprehensive final examination between a course offered in an ALN format or Place-Based.

Anecdotal evidence, however, pointed toward a higher level of conceptual learning and a lower ability to apply techniques for students in the ALN offering.  Taken together these suggest a possible empirical test of the conceptual and technique learning associated with place-based and virtual seminar formats. As previously stated, the virtual seminars allowed for a richer communication environment. Students had the opportunity of considerably more input and dialog, and were expected to contribute to a discussion much richer than that available in a 240 minute weekly place-based meeting of 50 to 70 students. Since cognitive flexibility can be described as the layering of meaning over basic understanding of a situation [1], the level of dialog would imply a greater situational understanding. This leads us to the following hypothesis

H_B: Students taking a course in ALN format will have significantly higher scores on conceptual understanding than those taking the same place-based course.

The second factor under study, namely the application of technique, could also be explained by considering the learning environment. Mayer [7] describes various methods for teaching and learning. A particular technique, along with its approach for standardized application, could best be thought of as an analog. 'An analog is a problem that contains a similar structure but not necessarily the same story line as another problem, that is, an analog has structural similarity but not surface similarity with a target problem' [7]. The place-based version of the course included structured tutorials where problems were postulated and discussed with considerable hand waving, iterative technique development and application, and repeated problem solving. The learning process was spontaneous, visual and interactive, methods that present a somewhat daunting challenge for an online offering. Replacing place-based learning environments with an online step-by-step explanation for self study would, assumedly, lose some of the richness of the learning process itself. This leads to the final hypothesis:

H_C: Students taking a course in ALN format will have significantly lower scores on technical ability than those taking the same place-based course.

V. Method

Our empirical analysis consists of a longitudinal exploratory study of final exams grades associated with students taking an undergraduate system analysis and design class (BUS 362) at Simon Fraser University. Five semesters worth of data gathered from the fall of 1996 to the summer of 2000 was included in the study. In two of these semesters, the course was run as a placed-based course with 13 weekly 3-hour lectures and weekly one-hour tutorials in a computer lab. The placed-based courses were supported with a course web page containing lecture notes and supplementary information. In the remaining three semesters, the same course material was offered virtually.  The "FirstClass" conferencing system was used to provide areas for discussion and the same web site used for the place-based course was used to support the ALN class. Lecture notes were altered from place-based offering and written in a more narrative style in order to provide a wider perspective to students taking the ALN format.

The content covered in the lecture notes remained constant throughout the period. The same instructor taught all five sections. The material in the course was altered only slightly from semester to semester.

A. Participants
All participants were third year undergraduate students in the Faculty of Business Administration at Simon Fraser University. A total of 107 students received final exams grades for the two place-based offerings (1996, 1999). The three ALN offerings (1997, 1999, and 2000) included a total of 128 students. There were no dropouts from either the ALN or place-based courses. No significant difference in class size or average grade on final was found between groups as shown below in Table 2.

      Table 2: Summary of Variables Across Classes

B. Data: The Final Exam
The final exam format was exactly the same across the five semesters. The same amount of time to complete the exam was provided in all 5 sections. There were no substantial differences in the types of material the students were provided in the exam, and all students - place-based and ALN - wrote the exam at the same time and in the same room.

Students in both formats had similar, identically structured evaluative instruments. Included in their materials was a final examination designed to test exactly what the two key learning objectives of the course were. First, there were twenty-five short answer questions asking such conceptual issues as 'Describe the relationship between a systems analyst and a professional accountant during the various stages of the design and implementation of a new information system.' While these questions varied across the years, the level of difficulty, as judged by average grades across years, was not significantly different. The second part of the final examination was a short (two page) case describing a systems design situation. The students were asked to apply their understanding of the various techniques learned to the material described in the case. While the case changed over the years, the difficulty, as judged by average grades, was not significantly different. These examinations offered insight into the effectiveness of longer term learning in each of the two areas under study.

VI.  Analysis

There were two sections in the test: one for conceptual and the other for technique. The exam scores were compiled for each section and assigned marks were divided by the total for each section. This provided the percentage assigned to each section of the test. Multiplying this percentage by the weight attributed to each section of the exam created the contribution of each section to the final exam score.

The contributions of the conceptual, technique, and total scores were then compared using One-Way ANOVA with the method of presentation (ALN or place-based) as the treatment variable.  One-way ANOVA is a robust parametric technique that is appropriate in comparing means between one or more groups when the sample size is relatively large (greater than 30). Means and standard deviations for the variables are provide below in Table 3:

Table 3: Descriptives for Contribution of Conceptual and Technique to Final Exam

Results from the ANOVA are shown in Table 4. The analysis of combined totals reveals that the average combined exam score for ALN students was 75.1 while for place-based it was 75.5. The results suggest there is no significant difference between the ALN and place-based students in overall score, proving H_A.

The results also indicate that the group of students in the ALN course scored significantly higher in conceptual contribution than place-based students (a contribution of 38.1% for the ALN versus 34.5% for place-based), proving H_B. Finally, the results also indicate that students in the ALN course score significantly lower in the technique contribution to the final exam score when compared to place-based students (a technical contribution of 37% for the ALN as compared with 41% for the place-based group), proving H_C.  Both of these results are significant at the p=.001 level as shown below in Table 4.

Table 4: Results from One Way ANOVA: Place-based versus Virtual Seminar     

These results provide some validity for the proposition that while there may be no overall difference in performance, ALN courses may provide a better environment for conceptual learning whereas place-based courses may provide a superior environment for learning the application of techniques. 

VII. Discussion and Conclusions

This study has focused on the comparison of conceptual and technique learning in ALN and place-based environments. The study is based on results gathered from one course that was offered in two different formats: virtual seminar and place-based. The course was taught in both formats by the same instructor, using the same textbook, and focusing on the same material. The final examinations followed the same format and were roughly equivalent. This provided the researchers with an opportunity to compare several offerings of the same course in two different formats when a large amount of similarity existed between course offerings.

Three important findings emerge from the analysis. These are listed below:

1. No significant difference was found in final exam scores between place-based and virtual seminar students (H_A).

This result indicates that place-based and virtual seminars can both be effective methods for delivering course information. 

2. Students taking the virtual seminar scored significantly higher on the conceptual section of the final exam than place-based students (H_B).

We have hypothesized that the higher conceptual scores for students using virtual seminars are related to the increased interactivity associated with virtual seminars. As noted earlier, the online environment helps to increase the level of interactivity with students as they are not limited to same room same time participation which includes fixed time allocations for discussion.  

3. Students taking the virtual seminar scored significantly lower on the technique section of the final exam than place-based students (H_C).  

We have hypothesized that the lower technical scores for the students in the virtual seminar may be the lack of appropriate analogs upon which to base their knowledge. Place-based seminars provide an immediate feedback and can more thoroughly handle questions about a particular case. Since the students in a place-based environment are provided with a better opportunity to develop analogs early in their learning, place-based students were expected to have higher technique scores. The results indicate that place-based students did indeed show significantly better scores in technique than virtual seminars.

Having reported these differences, it is important to note that more research is needed to better confirm and identify the source of these differences. While a longitudinal field study reported in this study provided a good opportunity to view the difference between conceptual and technique learning across different formats, more controlled experiments should be able to more precisely indicate where the differences lie. For example, it cannot be stated conclusively that the students were randomly assigned to treatments. The format of each class was announced in advance and followed a pattern of online courses offered in the Spring term with place-based either in the Summer or Fall. Students could, to the extent their academic schedule allowed, self-select for a particular format.

It is also important to note that not all courses have both conceptual and technique learning. The results reported in this study suggest that courses that contain largely conceptual material may offer excellent opportunities for ALN, while courses focusing on technique may be best provided in a traditional place-based environment.

Even more interesting is the implication that it may be possible to capture the best of both worlds in course design and delivery. For example, place-based courses with rich conceptual learning objectives could be augmented with online discussion areas. This would provide increased participation that proved to be valuable for conceptual learning in the virtual seminar environment. Conversely, virtual seminars could be augmented many simple examples of technique that provided some level of interactivity such as animation, interactivity through object-oriented environments, or a combination of image and sound. This would provide a more favorable environment for developing analogs that could lead to improved performance in techniques.   

VII. References

1. Ausubel, D.P., Novack, J.D., and Hanesian, H., Educational Psychology: A Cognitive View, Holt Rinehart and Winston, New York, 1978.
2. Gersting, J.L., Chen, B., Muraoka, K.T., Web-Based Education in Hawai'i, WebNet 99 Proceedings, Association for the Advancement of Computers in Education, http://www.aace.org
3. Hiltz, S.R., Coppola, N., Rotter, N., Turoff, M., Measuring the Importance of Learning for the Effectiveness of ALN: A Multi-Measure, Multi-Method Approach, Journal of Asynchronous Learning Networks, Vol. 4, Issue 2, 2001.
4. LaMaster, K.L., Martin, D.B. and Vinge, S., Does Electronic Communication Enhance Guided Reflection among Graduate Students?, Society for Information Technology and Teacher Education (SITE) 1999 Conference Proceedings, Association for the Advancement of Computers in Education, pp 1380-1386.
5. Mayer, Richard E., "Models for Understanding," Review of Educational Research, Spring 1989, Vol. 59, No. 1, pp 43-64.
6. Mayer, Richard E. and J.K. Gallini, "When is an Illustration Worth Ten Thousand Words?" Journal of Educational Psychology, 1990, Vol. 82, no.4, pp. 715-726.
7. Mayer, Richard E., Thinking, Problem Solving, Cognition, Second Edition, W.H. Freeman and Company, New York, 1992.
8. Parker, D.C., Rossner-Merrill, V., 'Socialization of Distance Education: The Web as Enabler,' WebNet '98 Proceedings, Association for the Advancement of Computing in Education, Orlando, FLA, This paper is online at http://parker.bus.sfu.ca/webnet/webnet98.html
9. Rossner-Merrill, V., Parker, D., Mamchur, C., and Chu, S., Using Constructivist Instructional Design Featured in Two Online Courses: Notes from the Field, Journal of Educational Media International, Volume 35, No. 4, December 1998.

VIII. ABOUT THE AUTHORS

Drew Parker is Associate Professor of Information Technology in the Faculty of Business Administration at Simon Fraser University. He has been teaching online since the start of this study in 1996, and has helped develop and launch several courses and a completely online Graduate Diploma in Business Administration at Simon Fraser University. He holds a B.Comm. and MBA from the University of Calgary, and a Ph.D. from the Ivey School of the University of Western Ontario.

Andrew Gemino is Assistant Professor of Information Technology in the Faculty of Business Administration at Simon Fraser University.