MEETING THE NEEDS OF NONTRADITIONAL RESPONDERS THROUGH SUCCESSFUL ONLINE COURSES IN BIOTERRORISM RESPONSE

Karen Vignare
Director, MSU Global Ventures

John Sener
President, Sener Learning Services

ABSTRACT

The Monroe County (NY) Health Alert Network (MCHAN) and the Rochester Institute of Technology (RIT) recently completed their second year of a Centers for Disease Control grant-funded project that offers a series of instructor-led asynchronous online learning courses. The courses are designed to enhance the workforce competencies of local and state health department staff and other nontraditional responders in planning for and responding to bioterrorism and other health emergencies. The project meets an important need for critical skills and knowledge training in this field and demonstrates how partnerships of higher education, government, and industry can deliver such training online. Contrary to the results of many e-learning initiatives, the vast majority (84%) of students involved in the program successfully completed the courses they started. Student and instructor evaluation surveys show high degrees of satisfaction and success; they also provide useful information on how to improve the courses. A critical element in the success of the courses was the employment of a program coordinator to manage course creation, instructor training, marketing, and registration. If one considers project costs relative to completion rates, this project has been highly cost effective despite the added cost of paying instructors. Overall, project results indicate that the asynchronous instructor-led online course model can be implemented successfully in many corporate and government e-learning initiatives.

KEYWORDS

Asynchronous Instructor-Led Online Learning, e-Learning, Government e-Learning, Higher Education–Government Partnership

I. Historical Background

A. Monroe County Health Alert Network

Located on the southern shore of Lake Ontario in New York State, Monroe County surrounds Rochester, New York's third largest metropolitan area. Through a competitive application process, the Centers for Disease Control chose the Monroe County Health Department along with only two other sites in the country to serve as a Center for Public Health Preparedness. Each site chosen was required to develop a specific focus for public health preparedness; the Monroe County Health Alert Network (MCHAN) project focused on creating and implementing scalable models to support bioterrorism preparedness and emergency response. The county created a two-part approach: one designed to enhance on-the-ground capabilities and another designed to train nontraditional responders. The latter category is the focus of this paper. It was decided that the preparation and training would use asynchronous distance learning.

B. Distance Learning at Rochester Institute of Technology

With over 8,000 enrollments last year and thirteen years of experience offering online anytime/anywhere education, Rochester Institute of Technology is a leader and an early pioneer in the asynchronous learning field. Its commitment to real quality in distance learning is validated by the more than 3,000 students who have graduated from its various distance learning programs since 1994. At RIT more than 72% of all students enrolled in distance programs and taking more than 50% of their coursework online graduate within six years, and more than 95% of all students complete the distance courses they start [1]. In addition, annual student satisfaction surveys show that students rank their online experience at RIT very favorably.

Courses taken through the Online Learning Department at RIT provide the same qualitative experience as courses taken on campus. The same faculty that teach distance-learning courses teach on-campus courses using the same syllabus and objectives. RIT's online format is designed to allow adults with professional and personal obligations to get a full graduate or undergraduate degree without coming to campus. A founding motivation of RIT's distance learning program was to make sure that what was offered in distance would be accessible to students in terms of technology and, later, in terms of the standards of the Americans with Disabilities Act, and this aim has contributed greatly to the success of distance students at RIT.

RIT offers a full range of online services to distance students. Online learners are given access to all the services available to any student at RIT, including academic advising, library services, general and technical support, financial aid services, registration, and online student communities. RIT's Wallace Library ranks as one of the country's most progressive in offering students a plethora of resources electronically. In addition, the library has a distance and online learning librarian who specifically addresses the needs of distance learners.

C. Sener Learning Services

The mission of Sener Learning Services is to help universities, government agencies, and nonprofit organizations create high-quality, effective, accessible learning environments, systems, programs, and experiences. Sener Learning Services focuses primarily on supporting the development of various forms of technology-enabled learning (online learning, e-learning, distance learning) and the diffusion of innovative learning designs and strategies.

II. Project Overview

The Rochester Institute of Technology's Online Learning Department was contacted by the Emergency Medical Services Office of the Monroe County Public Health Department regarding collaboration on a grant-funded project, the Health Alert Network, in the spring of 2001. The project leaders had decided they wanted to offer online courses to meet the grant specifications. In the discussions that followed, RIT Online Learning recommended instructor-led asynchronous courses as opposed to self-paced study online or even game-type applications, given the target audience and budget for the project. Eventually, the collaborating organizations chose to develop four courses targeting nontraditional responders who may need to know how to help traditional responders in the case of an actual bioterrorism event. Development funds were available for content and course creation, hosting, technical support, instructor training, and evaluation.

Over the first year of the grant, instructional designers and subject matter experts designed the courses to work within a course management system environment. To simplify the process for future developers, a standard look and feel was adopted for all courses, which were built collaboratively and assembled by RIT. The lead coordinator for the county, who was trained to serve as the technical, instructor, and subject matter expert liaison, was also responsible for building awareness and making sure courses met the needs of the learners. Her role was critical to the project's success, and she supported the creation of courses that were highly interactive and required student participation. Students were graded on assignments that varied by course but generally included discussion, role playing, testing, and content mastery.

Four courses were developed: Bioterrorism: ICS (Incident Command System) (referred to as ICS in Evaluation Results section); Bioterrorism: What is Public Health? (WIPH); Bioterrorism: Management of Public Health Emergencies (MPHE); and Agents, Treatments, and Protections for Health Care Workers (Agents). The first course introduced the Incident Command System and how it works, and the second course provided a basic overview of public health. The third course was created to enhance the learner's understanding of how public health emergencies were managed. The final course provided students with information on the top six biological agents, how they act on the body, and what can be done to treat victims and protect against attack. Each course had its own syllabus and set of learning outcomes. For more information on one of the courses, see Appendix A.

As soon as the first course was developed, it was offered to a pilot group of students within Monroe County. RIT hired an evaluator, Sener Learning Services, to work on assessing the outcome of the course. The survey assessment and instructional design were reviewed, and improvements were suggested and implemented. The review process was designed to continuously enact positive changes to the courses; as data was collected, the courses and instructors were continuously improved. Monroe County collaborated with other health agencies and began offering all the courses to the larger New York State audience. The rest of the paper will discuss research on and scholarly outcomes of this project.

A. Summary of Literature Reviews

When the project was first started in the fall of 2001, an initial literature review was conducted for two purposes: (1) to find out whether there were reported results on comparable projects; and (2) to build the theoretical foundation for an approach centered on student success and learning effectiveness rather than the comparison of delivery modes, an approach still relatively common in the field despite its many drawbacks [2]. This literature review succeeded in finding supporting research for an effectiveness-centered approach, whose theoretical foundations are drawn from several sources in the research literature:

• Characteristics of adult learners as abundantly described in adult learning literature [3, 4];
• Description of key attributes of the ICS course based on how they might contribute to learning, drawing from media attribute theory as it pertains to online learning courses [5];
• Focus on student characteristics that facilitate or contribute to success [6];
• Focus on student perceptions of interaction, based on research that perceptions are more important than actual level of interaction [7]; and
• Importance of creating a sense of learning community and social interaction in online courses, based on learning community and social presence theories [8, 9, 10].

However, the literature review found little if any research that reported results on comparable projects, in part because of the pioneering nature and unique characteristics of this project but also for reasons more fully explained below.

In the fall of 2003, a follow-up literature review was conducted to find out whether the situation had changed during the past two years with regard to available relevant research. The findings were the same for the most part: research related to instructor-led online learning continues to be largely proprietary (i.e., conducted at private companies and restricted to internal use), undocumented (i.e., work has been done but not written up or published), or not directly relevant to the present case.

The latter point requires some clarification. As a joint initiative between a higher education institution and two social service agencies, this project occupies a different niche than either the corporate or higher education online learning worlds. Thus, research done in either of these two sectors may have limited applicability to the project. Indeed, corporate and higher education online learning are seen as “very different and separate marketplaces” [11], and their respective practitioner communities likewise have different notions about the meaning of instructor-led learning in an online environment. In the academic world, instructor leadership in an online course is generally a given and thus is not commonly studied as a separate variable. A number of recent studies indirectly indicate the effectiveness of instructor-led online learning via other variables and their interrelationships—for example, amount and quality of perceived instructor-student interaction [12], instructor guidance and feedback [13], and the correlations between perceived teaching presence, reported learning, and student satisfaction [14, 15]. However, there is a lack of research directly linking student interactions with learning outcomes [16].

In the corporate online learning world, by contrast, the term instructor-led online learning is at present usually applied exclusively to synchronous delivery modes; in fact, instructor-led online learning as commonly defined is impossible asynchronously. For instance, a commonly used definition of synchronous learning is “a real-time, instructor-led online learning event in which all participants are logged on at the same time and communicate directly with each other,” whereas asynchronous learning equals “interaction between instructors and students [that] occurs intermittently with a time delay” [17]. However, synchronous instructor-led online learning is becoming increasingly common, particularly via the use of commercial course management systems such as Centra, ILINC, and LearnLink, among others. In short, the corporate training world appears to be following a model that assumes that instructor leading can only happen in real time (although physical presence is not required).

Not surprisingly, asynchronous instructor-led online learning is not even a category in surveys that track the adoption of online delivery modes in the corporate training world [18]. Although online searches suggest that the term instructor-led online learning and related terms (e.g., instructor-led online training ) are now much more widely used, research on the topic is virtually impossible to find. To avoid confusion, this paper hereafter will use the term asynchronous instructor-led online learning (AILOL) to refer to the online learning approach employed by the project.

Despite the relative paucity of related results, the literature review indicated that an interesting variety of organizations are using AILOL. Some examples follow:

• The Online Instruction Center for staff development at the University of North Carolina at Wilmington uses an AILOL program that is an outsourced software from Thompson Learning, but no results are available (see http://www.ed2go.com/webu/index.html)
• The Energy Providers Coalition for Education (EPCE), a program sponsored by the Council for Adult and Experiential Education (CAEL), started an AILOL-based program in November 2002, but the program is too new to have generated results (see http://www.epceonline.org/)
• The American Bankers Association offers AILOL courses (see http://www.aba.com/Online+Courses/course.htm), but, again, no results are readily available.

In the few instances in which unpublished results are available, they are either too sketchy or too new to be given much weight, although in some cases they portend more substantive results to be available in the future. Some examples follow:

• Data supplied by a contact at a university in the northeast on two AILOL-based information technology courses consisted of anecdotal comments and results from only two completed post-course student surveys and thus was too sketchy to be useful.
• Data obtained from an evaluation of a pilot AILOL course offered by a professional association serving the financial services industry indicated very positive results (n=20) from the initial course offering. However, these results need to be confirmed by subsequent evaluations, and the data is at present proprietary and unavailable for public use.
• The most notable exception is a project conducted by the Illinois Public Health Preparedness Center (IPHPC) (see http://www.uic.edu/sph/prepare/). The IPHPC received another CDC grant to perform activities similar to those conducted for this project. Although the IPHPC project focus is somewhat different, it does include offering some AILOL courses, including two (one on bioterrorism preparedness and response and another entitled “What Is Public Health?”) that are similar to this project's offerings. Learner evaluations were conducted for these courses; although the results are unpublished to date, the data have been shared with the project evaluator and indicate generally favorable results.

In short, this project still appears at present to be the best source of data for evaluating the efficacy of AILOL, in this type of setting and for this target population in particular.

B. Course Design and Teaching Strategies

The course design flowed from the project requirements and the preferences of the county project coordinator and RIT's instructional design staff. As stated earlier, the preference was to offer AILOL courses. The targeted population of students was full-time working adults who were taking the course voluntarily. In some cases, managers of these full-time working adults recommended the course to the students. The course design documents reflected the fact that these students would have limited time to study. The county project coordinator decided that each course should require about four hours per week for four weeks, and it was assumed that students would do most of the work during the workday.

Learning objectives were developed for each course, and instructional activities were designed to support each learning objective. Simultaneously, the coordinator worked with subject matter experts to produce content presentations. Finding the right materials was difficult; in some cases, content had to be written from scratch because no content was publicly available. The presentations were created in specialized software so that static and animated graphics could be added and an active left-hand menu bar could be installed to navigate students through the presentations. It is clear that the intention was a valid one, but by today's standards (three years later), the cost of using this software was too high. In addition, the software did not allow students to easily download course materials.

Once the learning objectives and content were coordinated, a syllabus began to take shape. The county coordinator and RIT representatives met repeatedly to discuss instructional activities. A sample course syllabus is included in Appendix A. Although active learning is not a strategy discussed in the current literature on corporate e-learning, the courses were designed to be active and build interaction with and among the instructor, students, and content. It was decided that students would be graded on assignments and participation in the online course. The choice was an interesting one: the design was based on what works in higher education e-learning and not what was standard practice in corporate e-learning.

C. Evaluation Design

As stated above, the evaluation of the project courses focused on student success and learning effectiveness rather than a comparison of online to face-to-face delivery. This approach was utilized because of the difficulties and limitations of employing experimental or quasi-experimental approaches to evaluation in real-life learning environments. The evaluation team decided that it is more useful to have answers to questions about what works, what characteristics support favorable outcomes, and how to make the course the best it can be than to try to prove whether one delivery method is better than another.

The evaluation investigated three key questions: Is AILOL an effective way of offering these courses? What are the elements that make them effective? What improvements can be made to the courses to make them more effective? Because the courses were designed to enable adult voluntary students to learn job-related “just-in-case” knowledge about the topics, the evaluation focused on how the key elements of the courses—in particular, delivery format and related attributes, student characteristics, and desired learning outcomes—contributed effectively to student learning.

III. Evaluation results

A summary of key project results suggests that the answer to the first question (Is AILOL an effective way of offering these courses?) is a resounding yes:

• A total of twenty-nine courses were offered in Years One and Two.
• More than 300 students completed courses during the project's first two years.
• Aggregate completion rates for the courses were 85% for Years One and Two.
• High levels of student satisfaction and perceived learning effectiveness were maintained for both years of the project.
• High levels of satisfaction in each related area (instructor role, course interactivity, etc.) were also maintained for both years of the project.
• Project results indicate successful scaling of course offerings during the second year of the project.

A. Enrollment and Completion Data

As Table 1 indicates, course enrollments totaled 402 for all course offerings in Years One and Two. A total of twenty-nine course offerings were completed during the first two years of the project. Year Two course completion rates improved upon the respectable completion rates obtained in Year One, raising the cumulative completion rate from 75% to 85%. Both agencies offered these courses almost exclusively to students located in New York State; most of the out-of-state residents who took these courses did so in Year One.

Table 1. Composite Course Enrollment Data and Completion Rates

B. Overview of Composite Student Survey Results, Years One and Two

Cumulative survey response rates for Years One and Two decreased somewhat relative to Year One, for which there was a 69% response rate. Survey response rates by course type vary from 67% for the WIPH courses to 41% for the MPHE courses. A total of 160 students responded to the online course evaluation surveys for a 52% response rate for course completers, as indicated in Table 2. A course-by-course breakdown follows:

Table 2. Years One and Two Composite Online Survey Response Rates

The composite results from the student surveys indicate highly positive results overall in each area. As indicated in Table 3, the mean positive response rate for the survey questions was around 80%, whereas the mean negative response rate was just over 5%. Even for the lowest-ranked indicators, the positive response rate was around 70%. Similarly, the negative response rate was fairly low; all but five of the twenty questions had a negative response rate of 4% or less, and only two of the questions had a double-digit negative response rate, with the highest negative response rate at 23% (see Table 5) and the next highest at only 11% (see Table 6). The mean score (4.13) also indicates a high level of positive response to the questions overall.

Table 3: Years One and Two Composite Results for All Survey Questions

C. Elements of Effectiveness

For the question of identifying the elements that make these courses effective, project results indicate that each of the key elements of these courses—instructor role, course interactivity, delivery format, and related attributes—was effective in each of the courses. High levels of student satisfaction and perceived learning effectiveness were maintained for both years of the project, and high levels of satisfaction in each related area (instructor role, course interactivity, etc.) were also maintained throughout the project.

1. Instructor Role

Student survey results related to instructor role were overwhelmingly positive, as indicated in Table 4. Level of positive response was fairly uniform, ranging from 76% to 88%, whereas level of negative response was uniformly low (2% to 4%).

Table 4. Years One and Two Composite Results Related to Instructor Role

2. Course Interactivity and Learning Community

Survey results related to course interactivity (Table 5) were also positive, although not as strongly as those for instructor role. Questions related to levels of student-student interaction, efficacy of digital interaction, and learning community had the lowest positive (69%, 71%, and 72%) response rates out of all the questions asked. However, these percentages improved during Year Two, and the mean scores (3.81 to 3.89) indicate an overall positive response for each measure. The relatively large neutral response rates (21% to 24% for three of the questions) may indicate that these factors are not as important to a significant percentage of survey respondents.

As noted previously, the question on making professional contacts is an exception to this analysis. The high neutral response rate (58%) and the mean score (2.89, or almost neutral) strongly suggest that this issue may not be so important to students in these courses.

Table 5. Years One and Two Composite Results Related to Course Interactivity and Learning Community

3. Delivery Format and Media Attributes

As indicated in Table 6, response rates for questions related to delivery format and media attributes were almost uniformly and highly positive. Based on qualitative student feedback from in-course and post-course surveys, the somewhat higher negative response rates for the questions appear to be related to periodic course access and navigation issues such as server outages, connection and download speed, and printing difficulties. To some extent, the level of negative responses to this question set reflects issues external to the courses themselves, such as student access to high-speed internet service. Nonetheless, cumulative survey results represent a slight improvement over Year One results for this area, and the questions related to course materials, learning resources, and course discussions and case studies rates some of the most positive response levels.

Table 6. Years One and Two Composite Results Related to Delivery Format and Media Attributes

4. Learning Outcomes, Comfort Level, and Overall Satisfaction

As indicated in Table 7, the composite results related to learning outcomes, comfort level, and overall satisfaction were highly positive as well. The question related to overall satisfaction (taking another online course based on experience in this one) had the highest positive response rate and overall mean score out of all survey questions. In fact, only the ICS and MPHE courses had any dissatisfied survey respondents (four out of 160 respondents over the two-year period). Nearly 80% of survey respondents reported that taking the course helped them feel comfortable about distance learning. The high rate (32%) of neutral responses to the question of whether the course helped remove fears about distance learning likely indicates that a significant percentage of survey respondents did not have any initial fears about the delivery method, so the fact that 64% of respondents reported that taking the course removed their fears about distance learning also indicates a high rate of success in achieving comfort level. Respondents also achieved a high level of reported learning based on the content-related question, which probed whether taking the course enabled students to understand the content of the course. The questions pertaining to perceived learning and overall satisfaction received the highest positive response rates.

Table 7. Composite Results Related to Learning Outcomes, Comfort Level, and Overall Satisfaction

D. Course Improvement Process

Identifying improvements to be made to the courses was a continuous process throughout the project period. As is the nature of pilot offerings, the first offering of the ICS course yielded a number of improvements that were subsequently implemented during the first year of the project—for instance, increasing the variety and type of assessment methods and activities; revising test questions in some sections to reflect related discussion topics more closely, improve flow, or save student time; adding or revising a discussion topic in several of the course modules; and modifying the role-play session to improve participation. Many of these improvements were prompted by specific student suggestions. Some fine-tuning occurred for subsequent course offerings—for instance, facilitating the use of the course content as a post-course reference aid by creating a printer-friendly version of the material, offering more learning resources such as more real-life examples and scenarios, providing more opportunities for interaction through discussions and other means, and adding specific instructions on what a good discussion response is in the course syllabi.

The course improvement process in the second year of the project consisted of the following elements:

• Collection and analysis of student and instructor feedback from in-course and post-course surveys.
• Telephone conferences with project staff to review student and instructor feedback and establish priorities for course improvements based on feedback.
• Compilation of tips for communicating effectively online and a list of recommended strategies for creating additional activities to enhance existing opportunities for interaction among learning participants.
• Course revision process conducted with instructional design staff at RIT and project staff.

In addition to student and instructor post-course surveys, the course improvement process also relied on collection and analysis of in-course student feedback. Each session (section) of every course included two survey questions, one asking students for feedback on needed additions or improvements to the session and the other asking students to self-assess their learning from the session.

During May 2003, the project evaluator reviewed every comment related to the additions and improvements question as well as selected comments related to the learning self-assessment question for courses offered from the fall of 2002 through April 2003. Comments were categorized as either positive/OK or negative/suggestions for improvement. Although the in-course survey did not explicitly ask for positive feedback, at least half of the comments entered were positive/OK comments, often simple ones such as “OK,” “No changes needed,” “Great work!” and so forth. The survey questions also yielded numerous suggestions for improvement.

During Year Two, a total of seven instructors led MCHAN courses. In May 2003, an online instructor survey was conducted to determine the general level of instructor satisfaction, perceived effectiveness, and possible areas of instructor concern about the courses. The online instructor survey closely paralleled the online post-course student survey; some survey questions were identical, and others were slightly modified to find answers to related questions from the instructor point of view. The survey also included several open-ended questions addressing more specifically instructor-related issues. Six out of the seven participating instructors responded to the survey. The survey results were positive overall (mean score = 4.16) and in each related area, although they also indicated several areas that warrant closer attention.

Key results included the following:

• Instructors were positive about almost all aspects of the online teaching experience and its efficacy.
• Instructors were very comfortable and highly satisfied with teaching online. Most instructors found interaction with students to be rewarding and appreciated the flexibility of instructing online and being able to instruct students from different backgrounds in the same course.
• They were also very confident that their role made an important difference in student learning and satisfaction.
• Unlike students, instructors were slightly dissatisfied with the level of student-student interaction in their courses. The disparity in findings suggests a mismatch between student and instructor expectations on this issue. A possible explanation is that instructors are not fully aware of the level of student-student interaction taking place outside of the course.

Analysis of student and instructor feedback indicated several areas for possible course improvements. Among the major ones were:

• Navigational issues, such as navigating from one course page to another;
• Content download and printing issues;
• Inclusion of additional learning resources—for instance, more graphics depicting diseases and a more comprehensive medical dictionary; and
• Various minor issues with particular test questions in a few of the course sessions.

Many of the suggested improvements are not practical because implementing them would involve unacceptable trade-offs of one sort or another—for instance, creating a new (and probably more serious) navigational problem by solving an existing one. Other suggested improvements are the result of the built-in features and limitations of the existing course management system (Prometheus). Some of these problems would be mitigated if the courses were migrated to a new course management system, a move that has been under consideration. Other suggestions for improvement were quickly implemented for subsequent iterations of the course—for instance, the addition of word definition lists, more optional reading materials, and a non-credit–bearing discussion topic area.

Using on the feedback provided, the course development team made the following improvements to the courses:

•   Linked course objectives to the competencies published by the Centers for Disease Control for all public health care workers
•   Rewriting student discussion topics to facilitate interaction
•   Rewording test questions for clarity
•   Reducing the number of questions to reduce time spent on tests
•   Updating course material to reflect current events and new scientific knowledge
•   Updating the case studies and examples

E. Cost Analysis Data

No project review would be complete without a thorough analysis of costs and an assessment of whether those costs achieved desired results. The costs described below constitute either best estimates of expenses or actual expenses. Estimates are given in place of actual salaries and where Monroe County provided resources through the other parts of the grant. (The full grant required more deliverables than just the development and delivery of the courses.)

Table 8. Online Course Development and Delivery Costs

Using the above data and the enrollment and completion information covered earlier in the paper, we can examine cost effectiveness and return on investment measures. The total cost per completion was $1,000 per student, and the total cost per enrolled student was $766 dollars per student. The question of whether this is a reasonable cost and a reasonable return on investment is difficult to answer. If this project is compared to a self-paced computer-based training or interactive webpage, development costs would likely be significantly higher. Current industry development costs range dramatically: the time required to develop one hour of self-paced training can be anywhere from 125 hours to 1,000 hours. The cost of that one hour can range from $75 per hour to $1,000 (or more) depending upon the multimedia skills needed by the content developers. So if we look at one reasonable comparison, each AILOL course of the kind examined in this paper consisted of sixteen hours of self-paced training. If each hour of training takes 250 hours to develop and costs $125 per hour, then one course would cost $500,000. Once we had 500 students per course (500 x $1,000), the price of self-paced training (250 x $125 x 16) would be the same as that of an asynchronous instructor-led online course, and our break-even point would have been reached. Of course, costs for the instructor-led course decline year by year until the content needs to be revamped.

And, more important, it is difficult to speculate on whether the self-paced learning would have achieved as high a completion rate. The comparison above assumes that the completion rate was 100%, but the completion rate for e-learning is highly debatable; research shows that academic course completion rates fluctuate widely [18] and training completion data is not included in the typical measurement analysis. However, most case study data seem to indicate that non-mandatory training has completion rates of 50% or lower. So if one assumes that only 50% of students complete, at least 1,000 students need to take the course for it to match the costs of self-paced training. However, it should be noted that self-paced learning may be used more frequently as a just-in-time learning method. Just-in-time or on-demand learning is far more important in corporate settings than it is in higher education. Measuring course completion rates varies in importance depending on the purpose of the learning.

IV. Conclusions

The following list is a set of recommendations the authors believe could be applied to other online learning projects in corporations and organizations.

1. Instructor-led training works in fields besides higher education.
2. Instructors motivate students regardless of whether the content of the course is required for work.
3. Good instructional design that keeps students motivated and interactive leads to completion.
4. Assessment and evaluation should be an iterative process designed to improve learning experiences and not just result in data.
5. A project coordinator is essential and cost effective.

V. References

1. Vignare, K. Longitudinal success measures for online learning students at the Rochester Institute of Technology. In Bourne, J and J. C Moore (eds.), Elements of Quality Online Education: Practice and Direction, 261–278. Needham, MA: Sloan Consortium, 2002.
2. Sener, J. Escaping the comparison trap: Evaluating online learning on its own terms. Innovate: Journal of Online Education 1(2): 2004/5.
3. Knox, A. Helping Adults Learn. San Francisco: Jossey-Bass, 1986.
4. Knowles, M. Self-directed Learning. New York: Association Press, 1975.
5. Smith, P. L. and C. L. Dillon. Comparing distance learning and classroom learning: conceptual considerations. The American Journal of Distance Education 13(2): 1999.
6. Diaz, D. Carving a new path for distance education research. The Technlogy Source: March/April 2000. Online: http://technologysource.org/article/carving_a_new_path_for_distance_education_research/.
7. Fulford, C. P. and S. Zhang. Perceptions of interaction: The critical predictor in distance education. The American Journal of Distance Education 7(3): 8–21, 1993.
8. Gunawardena, C. Social presence theory and implications for interaction and collaborative learning in computer conferencing. International Journal of Educational Telecommunications 1(2–3): 147–166, 1995.
9. Haythornthwaite, C., M. M. Kazmer, and J. Robins. Community development among distance learners: Temporal and technological dimensions. Journal of Computer-Mediated Communication 6(1): September 2000. Online: http://jcmc.indiana.edu/vol6/issue1/haythornthwaite.html.
10. Swan, K. Content analysis of course transcripts: Social presence, 2002.
11. Masie, E. Across the moat: Higher education e-Learning. Learning and Training Innovations: April 1, 2003. Online: http://www.ltimagazine.com/ltimagazine/article/articleDetail.jsp?id=56420.
12. Picciano, A. Beyond student perceptions: Issues of interaction, presence, and performance in an online course. Journal of Asynchronous Learning Networks 6(1): 2002. Online: http://www.sloan-c.org/publications/jaln/v6n1/v6n1_picciano.asp.
13. Richardson, J. and K. Swan. Examining social presence in online courses in relation to students' perceived learning and satisfaction. Journal of Asynchronous Learning Networks 7(1): 2003. Online: http://www.sloan-c.org/publications/jaln/v7n1/v7n1_richardson.asp.
14. Shea, P., E. Fredericksen, A. Pickett, and W. Pelz. A preliminary investigation of “teaching presence” in the SUNY learning network. In J. Bourne and J. C. Moore (eds.), Elements of Quality Online Education: Practice and Direction. Needham, MA: Sloan Consortium, 2002.
15. Shea, P., A. Pickett, and W. Pelz. A follow-up investigation of “teaching presence” in the SUNY learning network. Journal of Asynchronous Learning Networks 7(2): 2003. Online: http://www.sloan-c.org/publications/jaln/v7n2/v7n2_shea.asp.
16. Swan, K. Learning effectiveness: What the research tells us. In J. Bourne and J. C Moore (eds.), Elements of Quality Online Education: Practice and Direction. Needham, MA: Sloan Consortium, 2002.
17. American Society for Training and Development. Learning Circuits Glossary. Online: http://www.learningcircuits.org/glossary.html.
18. Howell, S., P. Williams, and N. Lindsay. Thirty-two trends affecting distance education: An informed foundation for strategic planning. The Online Journal of Distance Learning Administration 6(3): 2003. Online: http://www.westga.edu/~distance/ojdla/fall63/howell63.html.

VI. About the Authors

Karen Vignare currently serves as the Director of MSU Global Ventures at Michigan State University. In that role, Karen is responsible for creating online entrepreneurial approaches for extending both non-credit and credit programs at MSU. Before that, she was the Sr. Research Analyst for the Online Learning Department at the Rochester Institute of Technology. She also served in other roles at RIT in the Online Learning department.  Before coming to RIT, Karen was a full-time faculty member at SUNY-Alfred State in the marketing, retail, and computer technology fields. She also served as a vice president and political economist for a Wall Street financial firm. She publishes regularly on various topics in online learning. She has an MBA from the University of Rochester 's William Simon School of Business and a BS from Frostburg State University in political science and economics. She is currently attending doctoral classes at Nova Southeastern University.

John Sener is the founder of Sener Learning Services, a consulting practice that focuses on supporting the evolution of online and other technology-enabled learning environments in higher education institutions and consortia, nonprofit organizations, and government agencies. Sener's consulting activities span strategic planning, evaluation, program development, instructional design support and development, workshop delivery, and grant writing.

Sener's current project evaluation activities include MarylandOnline's FIPSE-funded Quality Matters project and the Maryland Students Online Consortium (K–12). Other recent activities include providing strategic planning services for a governmental organization's implementation of a new online learning program, and designing instruments to evaluate the prototype and pilot of an online course certification program for a major international membership organization.

As director of special initiatives for the Sloan Consortium (Sloan-C), Sener is involved in a wide variety of the consortium's activities. He serves as the access-related effective practices editor and Speakers and Consultants Bureau coordinator; for recent Sloan-C conferences, he also served as a track chair or steering committee member. Sener has been on the editorial board of the Journal of Asynchronous Learning Networks since its inception and is a contributing editor for the monthly newsletter Educational Pathways.

Sener's career in education and training over the past twenty-five years includes directing a number of foundation and federally funded projects. He has also been a trainer, teacher, administrator, instructional designer, and a tutor in the areas of adult literacy, basic skills, information systems, and English as a second language. Sener holds an MS in education from Johns Hopkins University and a BA in psychology from Oberlin College.

VII. Appendix A

Sample Syllabus for the Course Traditional Incident Command System for the Nontraditional Responder

  General Course Information: Traditional Incident Command System for the Nontraditional Responder (insert appropriate course number information here)

Instructor Information:

(insert appropriate information here)

Instructor Name

Instructor Title

Office Address

Instructor/agency address

Instructor/agency phone number

Instructor e-mail address

Course Description:

The Bioterrorism: Traditional Incident Command System for the Nontraditional Responder course will give an overview of the subject of bioterrorism and demonstrate how the traditional Incident Command System can help all individuals responding to a bioterrorism event to deal effectively with its consequences. Traditional emergency response personnel are already familiar with the Incident Command System. Nontraditional emergency responders who may be involved in the response to a bioterrorsim event may not be as familiar with the Incident Command System and how it works.

This program is intended for hospital emergency personnel, public health department personnel and other nontraditional emergency responders. This course will cover the basic concepts of the Incident Command System. It is designed to give you an understanding of how traditional emergency responders work during an event so that you will have a better understanding of what happens if you are asked to assist during an event.

Prerequisites: None.

Course Objectives:

Upon completion, you will be able to:

Describe the traditional Incident Command System (ICS).

Explain the roles of traditional participant agencies in the ICS.

Participate in the implementation of the ICS in a practice scenario.

Emergency Preparedness Competencies Addressed :

For All Public Health Workers:

DESCRIBE the agency chain of command in emergency response.

IDENTIFY and LOCATE the agency emergency response plan (or in large agencies, the permanent portion of the plan).

Describe his/her functional role(s) and responsibilities in emergency response and DEMONSTRATE his or her role(s) in regular drills.

IDENTIFY limits to his/her knowledge, skills, and authority and IDENTIFY key systems resources for referring matters that exceed these limits.

RECOGNIZE deviations from the norm that might indicate an emergency and DESCRIBE appropriate action (e.g., communicate clearly within the chain of command).

APPLY creative problem solving and flexible thinking to unusual challenges within his or her functional responsibilities and EVALUATE effectiveness of all actions taken.

For Public Health Leaders/Administrators:

DESCRIBE the chain of command and management system (Incident Command System or similar protocol) for emergency response in the jurisdiction.

For Public Health Professionals:

PARTICIPATE in continuing education to maintain up-to-date knowledge in areas relevant to emergency response (for example, emerging infectious diseases, hazardous materials, and diagnostic tests).

Method of Instruction:

Learning Environment

The Prometheus System will be used as the online learning environment. Prometheus is accessible through the web, so the courses can be accessed from anywhere. Your instructor will facilitate the delivery of the course. You will come to class within a set period of time to view the content for each of the sessions and respond to questions, exercises, discussions, and scenarios posted by the instructor.

Role of the Instructor

You instructor will direct you to the course content you will be covering at the appropriate time. The instructor will be able to answer routine questions about the learning environment and about key concepts of the program. The instructor will also encourage interaction between you and your classmates during discussions and exercises.

Expectations of the Learners

You are responsible for participating in the course on a regular basis; downloading and viewing content; asking questions; and participating in discussions, exercises, quizzes, and scenarios and simulations.

Methods of Evaluation

Students will be evaluated on their level of participation in class discussions and simulations and on their scores on quizzes and exercises. Participation will be weighted as 40% of the grade. Participation and discussion expectations are listed below. Quizzes will count for 60% of the grade. Some questions will ask for single-word answers or require that you select from multiple choices. For questions that ask you to explain a concept, a complete quiz answer is a full thought that is phrased in sentence or paragraph form. I consider good participation to be responding to discussion questions and tests in a prompt and appropriate manner.

Participation and Discussion Expectations

Participation involves reading, analyzing, and responding to questions, comments, and issues that your fellow students and I post. You are expected to contribute to the class discussion in a meaningful way at least once, and preferably more often, on each discussion topic. I recommend logging on and viewing all of the current and past discussions at least four times per week. Sometimes there will be nothing new to add. However, it is important that you view and understand the comments that your fellow students and I post. My expectations are that posted messages contain significant information related to the week and are at least 75 words in length. I may provide some additional material related to the topics, and I will make sure the discussion stays on course. I urge you to find additional material related to each topic to share with the class.

Course Schedule:

You are expected to complete two course sessions per week. The session schedule is listed below. The date the session is posted will be the first day it is available for viewing. The corresponding assignments, tests or quizzes will not be available until that date.

Each session should take approximately two hours to complete (or four hours per week). This should include viewing sessions, participating in discussions, and completing quizzes.

What To Do Next: Once you have reviewed the syllabus (course description, objectives, etc.), select OUTLINE and then Introduction, or click on this link (insert appropriate information here) to access directly.

Note: Other course syllabi are available upon request; please contact primary author.