QUALITY MATTERS: INTER-INSTITUTIONAL QUALITY IMPROVEMENT FOR ONLINE COURSES

John Sener
Sener Learning Services

ABSTRACT

The Quality Matters (QM) project funded by FIPSE and administered by MarylandOnline (MOL) is creating a replicable inter-institutional continuous improvement model to assess, assure, and improve the quality of online courses. Designed to address statewide and national needs for credible quality assurance in online learning, the inter-institutional collaboration is an integral and essential feature of QM project organization, implementation, and impact. The project uses inter-institutional, intersegmental peer review teams as an integral part of the quality improvement process; the expansion of the peer reviewer pool to involve faculty from two- and four-year institutions beyond MOL members in peer reviews of courses is also particularly noteworthy. The QM project's success to date indicates the viability of creating an inter-institutional collaborative process for quality improvement in online courses. Future efforts will focus on determining whether the project can build and maintain a sustainable model for the long term.

KEYWORDS

Online Learning, Inter-institutional Collaboration, Quality Assurance, Quality Improvement

I. Introduction

The Quality Matters (QM) project is creating a replicable inter-institutional continuous improvement model to assess, assure, and improve the quality of online courses [1]. In Fall 2003, Maryland Online (MOL) (http://www.marylandonline.org/) received a three-year, $509K grant from the U.S. Department Fund for the Improvement of Postsecondary Education (FIPSE) to implement the project. MOL is a statewide consortium of 19 Maryland community colleges and senior institutions with a very small central office staffing (currently < 2.0 FTE). As such, MOL relies heavily on the ability of its member institutions to collaborate effectively, as evidenced by its prior success with implementing the Faculty Online Technology Training Consortium (FOTTC) and Project Synergy (http://www.mdfaconline.org) grant-funded initiatives. FOTTC trained 40 Faculty Fellows (two each from 20 Maryland institutions) to train others in the use of technologies in teaching; over 3,800 faculty were trained through the various FOTTC components. Project Synergy was a collaborative effort that created a repository of web-accessible learning objects in key discipline areas, a network of over 500 higher education faculty trained to enhance and utilize learning objects, and a model for applying processes developed through this project to training faculty in other disciplines. MOL's experience in successfully implementing these and other inter-institutional collaborative projects served as a solid foundation for the QM project.

The genesis of the QM project was an access issue: MOL has an innovative “seat bank” program in which participating institutions make seats available to each other in selected online courses depending on need and demand. Since Fall 2000, participating schools have shared 456 courses and provided 1,853 otherwise unavailable seats to students. Despite this relative success, quality assurance for these shared courses has remained an issue. As with most higher education institutions, MOL members are more comfortable with assessing the quality of their own courses than other institutions' offerings. How to assure that another institution's online courses meet one's own institutional standards?

MOL's solution was to explore the feasibility of establishing a common set of standards and a means for applying those standards to online courses. In Fall 2002, an MOL subcommittee on Quality Assurance, consisting of faculty from three Maryland community colleges (Frederick, Carroll, and Chesapeake) conducted a “proof of concept” pilot peer course review project. The subcommittee developed a set of quality assurance instruments and conducted a pilot review of an online Frederick Community College course. These instruments were designed to assist the instructor in improving the design of the course. Even more importantly, the course review process demonstrated the feasibility of using an inter-institutional peer review team to assess course quality.

Based on the success of this pilot project, a team of representatives from eight MOL members ( Baltimore City, Carroll, Chesapeake, Frederick, Howard, Montgomery, and Prince George 's CCs, and University of Maryland University College) prepared a grant proposal which MOL submitted to FIPSE.

II. The Need for Quality Assurance in Online Courses

As stated in the original grant proposal, the QM project's original focus was partly on an MOL-specific need: how to facilitate course sharing among its member institutions, and thereby better serve their students. Noting that "the absence of a credible quality assurance certification process for online courses limits student access to superior online learning opportunities" in Maryland, the QM project proposal was intended to enable participating institutions to provide that access while using course sharing to avoid needless duplication of effort.

The QM project was also designed to address a growing nationwide need for credible quality assurance in online learning. With nearly two million students studying online in American higher education institutions in the fall of 2003 [2] and projections indicating that the number of students taking online courses continues to grow at a rapid rate [2, 3], online learning has entered the mainstream of higher education. Various components of online learning, in particular the Internet, are now well integrated into the postsecondary learning experience. However, despite rapid growth and growing acceptance, concerns remain about the quality of online learning for a variety of reasons [e.g., 4, 5, 6, 7].

More importantly, practical needs such as increasing demand, sustaining cost-recovery, attaining profitability, and meeting regional accrediting agency requirements necessitate quality assurance for online learning. Many individual institutions and consortia have adopted internal quality assurance programs, but these and other quality assurance efforts tend to focus on providing solutions to institutions individually, collectively, or consortially but not collaboratively. Some examples:

• Michigan Community College Virtual Learning Collaborative's Online Course Development Guidelines and Rubric (http://www.mccvlc.org/~staff/Course-Guidelines-Rubric-v1.2.html) rates online courses on a variety of criteria and a four-point scale (beginning, developing, accomplished, exemplary) [8]. MCCVLC and QM have had a mutually beneficial relationship. MCCVLC's rubric was consulted during the QM rubric development process, and MCCVLC's experience with working as a collaborative helped inform the QM project in developing its processes. In turn, MCCVLC is a member of the QM advisory board (see QM Project Organization section below), and MCCVLC has been using the QM process as the core of its attempts to replicate a similar process in Michigan [9].
• California State University-Chico's Committee for Evaluation of Exemplary Online Courses provides a similar three-tiered (baseline, effective, exemplary) framework to evaluate online courses. CSU-Chico's Rubric for Online Instruction (ROI) is a nationally-recognized tool for evaluating the effectiveness of online instruction. In addition to its uses as a course design and self-evaluation tool, the ROI is also used as part of a process for means for gaining public recognition for exemplary online instruction within the university [10]. However, the rubric is implemented by faculty within their institutions rather than as part of an inter-institutional collaboration.
• The University System of Maryland's Web Initiative in Teaching (WIT) was another precursor (1997–2000) project which informed the QM project and shared some of its features [11]. The Peer Course Review for Online Learning Rubric developed by WIT uses a four-point rating scale (poor, average, above average, superior) and accommodates both fully online and hybrid courses. It also describes considerations and implications for each measure, linked directly to relevant research supporting the criteria on which the measure is based. WIT also used a peer review process to review web-based courses, but peer review teams were composed internally, i.e., institutions used only their own faculty to review their own courses [12].
• The Monterey Institute for Technology and Education Online Course Evaluation Project (OCEP) evaluates existing online courses in higher education. Each evaluation addresses a list of over 50 categories related to course features, including aspects of User Interface, Course Features and Media Values, and Assessments and Support Material among others [13]. Course developers can respond to evaluations and provide input on unique course aspects and support services [14]. The OCEP evaluation process employs a three-member team for each course, including subject matter and technology experts. Team members are experienced in both the development and teaching of online courses. However, evaluation team members are paid consultants, so the course review process is not an example of inter-institutional collaboration [15].
• The SUNY Learning Network has a well-established quality assurance component as part of its course design process. However, this process is driven by a central administration rather than being an inter-institutional collaboration[16].
• The eArmyU initiative requires the prime contractor (IBM Business Solutions Consulting) to submit highly detailed Quality Control Plans and sets performance requirements on a variety of program parameters [17]. Although they are called “Education Partners” in this initiative [18], in practice academic institutions are in a contractor/vendor relationship with detailed contractual performance expectations rather than participating in an inter-institutional collaboration.

As these examples indicate, while many existing QA processes appear to be fairly adaptable, few if any of them are explicitly designed to support inter-institutional or intersegmental collaboration. By contrast, the QM project is designed to enable sharing among institutions, to facilitate a faculty-centered process, or to be easily replicable or scaleable. As with previous MOL projects, inter-institutional collaboration including both two- and four-year institutions is an integral and essential feature of QM project organization, implementation, and impact.

III. QM project organization

As Chart 1 indicates, many key components of the QM project promote and depend on inter-institutional collaboration. Composed of individuals from MOL and four member institutions (Chesapeake College, Frederick Community College, Prince George's Community College, University of Maryland University College), the QM Project Management Team (PMT) is itself an inter-institutional collaboration, ensuring that multiple perspectives from different types of institutions are incorporated into project management decisions. Most participating MOL institutions are also represented on one or more QM Working Committees, whose collaborative work has been essential for project development and implementation (see QM Project Implementation section below for more detail).

Chart 1. QM Project Organization Highlighting Inter-Institutional Collaboration

Within MOL, Administrative Representatives (ARs) are another key mechanism for promoting inter-institutional collaboration. Each participating MOL institution has appointed an AR who is responsible for collecting and reporting information, communicating project information to their institutions, and actively participating in data collection. Involving ARs effectively is an explicit "process goal" of the project. During the project's second year, ARs met three times via an interactive video network and once at an in-person retreat; ARs were also:

• Invited to become course peer reviewers and shadow a peer course review; 14 trained Peer Reviewers and one AR shadowed a course review.
• Provided with a monthly plan of suggested AR activities and outreach.
• Given a standard slide show and handout package to facilitate dissemination activities at their institution.
• Involved in the process for selecting courses and peer reviewers.
• Invited to a faculty forum that allowed them and their community to hear directly from faculty involved in the QM project.

Course peer review teams composed of faculty from three different institutions also constitute a form of inter-institutional collaboration; because of their collaborative structure, both the review process and peer reviewer training sessions are inter-institutional faculty development opportunities. Although most reviewers used to review MOL courses to date have been from MOL institutions, the use of reviewers from other institutions is increasing.

One unanticipated example of inter-institutional collaboration has been the development of an Instructional Designers Affinity Group (IDAG). Originally conceived to meet the need for instructional designers to assist with course revision, t his group now also provides much needed community and professional development activities for instructional designers and technologists, who work independently at many institutions. The IDAG is now an independent self-sustaining organization ( http://www.pt.umaryland.edu/idag ) and an official affiliate of the Maryland Distance Learning Association.

Beyond MOL, the QM project's External Partners consist of consortia, organizations, and individual institutions who agreed to participate in the project during the grant proposal phase. The Advisory Board consists of the original External Partners plus additional institutions and their representatives who have become involved with the project since its inception. Communicating via periodic audio conference calls and supported by the QM web site, both the External Partners and Advisory Board have provided advice and guidance, reviewed project documents and implementation protocols, participated in project dissemination, and provide additional visibility to the project. As Table 1 indicates, the QM project has utilized an inclusive approach to find partners to collaborate on informing project implementation.

Table 1. QM External Partners and Advisory Board (as of August 2005)

IV. QM Project implementation

The QM project is intended to assure the quality of courses, not of instructors, and focuses on reviewing course design rather than course delivery. A QM Working Committee played a major role in developing and implementing each of the key components in the QM model: tool set, process, training, and peer reviews.

A. Tool Set

The QM tool set is a refinement of the documents used in the pilot project and includes an Instructor Worksheet, Matrix of Review Standards, Exit Interview Form, and other documents which support various elements of the course review process. The QM Tool Set Committee (Table 2) focused most of its attention on the QM rubric, which is the most important tool and centerpiece of the QM faculty peer review process. The rubric was primarily designed to enable faculty peer review teams to apply practitioner expertise to evaluating the quality of online courses. The Tool Set Committee also developed protocols to explain how to apply the specific instruments within a peer course review.

Table 2. QM Tool Set Committee Members

The QM Tool Set Committee refined the pilot project rubric into the QM rubric, which consists of 40 Specific Review Standards grouped into eight General Review Standard areas corresponding to critical aspects of online course design and supported by the research literature and recognized quality standards.

The QM rubric also undergoes a continuous improvement process and has gone through several iterations since the project's inception. Refinements are made by the Tool Set Committee and by the PMT based on feedback from peer reviewers, project partners, and even prospective users. For example, the current version of the QM rubric includes annotations and examples to help peer reviewers apply or interpret each of the rubric's specific review standards. There is also an online demo version of the rubric which enables prospective users to try out the demo rubric to see how it works, and an online production version which enables peer reviewers to record their evaluations online and automatically tabulate the results.

B. Process

Although the QM rubric has had an extensive impact to date, the key to the success of the QM project is that it provides a process for faculty to actually implement quality practices in online courses in an inter-institutional, peer-centered context. As noted previously, there are many similar rubrics currently in use, mostly at individual institutions. However, the implementation of these rubrics often falters, and few if any are used for inter-institutional collaboration, because there is no process in place to do so.

Table 3. QM Process Committee Members

The QM Process Committee (Table 3) creates the procedures necessary to carry out various project activities, in particular peer course review. Its major focus is on course and peer reviewer selection and certification processes.

Chart 2: Quality Matters Course Review Process

As Chart 2 indicates, the purpose of the QM process is to enable faculty to improve their courses by providing a coherent, measurable process as well as sufficient support for making needed improvements. The QM Process Committee has also made refinements to the course review process throughout the course of the project, including developing procedures for contingencies such as substituting review team members in case of illness or other personal reasons.

C.Training

The QM Training Committee (Table 4) developed and implemented the training curriculum for peer course reviewers and faculty developers. In addition to the pilot peer review training in Spring 2004, other training products include subsequent in-person peer review training, an online version of peer review training, and most recently a ‘train-the-trainer' program to meet the demand for more trained QM Peer Reviewers. This committee also serves as a resource for providing instructional design services to faculty and convened the statewide Instructional Design Affinity Group which subsequently became an independent, self-supporting entity.

Table 4. QM Training Committee Members

D. Peer Reviews

Perhaps the most groundbreaking innovation of the QM project is its use of inter-institutional peer review teams, which consist of three members with one member designated as review team chair. Selection is based on several factors, including:

• Institutional distribution = representing as many different MOL institutions as possible
• Subject area = at least one review team member teaches the related subject or as closely related to the subject as possible
• Mix of sectors, including senior and community colleges, public and private, metropolitan and rural

For example, during the pilot review phase a Harford Community College online course was reviewed by faculty from Harford CC (team chair), UMUC (subject area expert), and Howard CC. (During the pilot phase, a member of the PMT also participated in each of the peer reviews as a non-voting member.) The three-member peer review team each reviews the course individually using the QM rubric, making a ‘yes/no' decision on each of the rubric's 40 specific review standards and also offering specific positive or corrective recommendations as desired for each standard. The team also discusses their individual decisions as a group, in particular focusing on those standards for which there is a ‘split decision' (i.e., one of the reviewers makes a different judgment from the others) rather than a consensus. Team members then make their final individual recommendations and post them on the online rubric review form. The form automatically compiles individual entries and tabulates the results, indicating whether the course meets expectations or needs improvement. To meet expectations, a course must receive a score of at least 68 points on an 80-point scale, and must also receive a ‘meets expectations' for each of 14 “essential” specific review standards. The results of the course review are shared with the faculty course developer, including specific positive and corrective recommendations.

In effect, each of these course reviews represents an inter-institutional collaboration among faculty. In fact, participating faculty report that the peer review process is a rewarding professional development opportunity which usually creates a collegial and collaborative rather than adversarial relationship between faculty course reviewers and developers. This is important because the QM peer review process ultimately depends on the individual and collaborative judgment of faculty. The option of creating a rubric with another level of specific criteria was considered and rejected due to concerns of making the process less faculty-driven and open to excessively restrictive application, administrative manipulation, or other potential problems. The result is a more seemingly “subjective” process with the potential of becoming a rubber-stamp mechanism. In practice, however, approximately 50% of courses do not meet expectations during the first review, a larger number than anticipated.

As of August 2005, the QM project has reviewed 46 courses using this process, and an additional 25 courses have been selected for review in Fall 2005. The project has also trained 258 peer reviewers from 65 different institutions. Initially, peer review participation was limited to faculty at MOL member institutions; faculty from external and other non-MOL institutions were trained to be peer reviewers, but the initial purpose was to disseminate QM beyond MOL. QM then took the additional groundbreaking step of involving faculty from non-MOL institutions in peer reviews of MOL courses. After being piloted to a limited extent during the project's second year, the use of trained faculty peer reviewers from non-MOL institutions has expanded to the point where in Fall 2005 there will be course reviewers from many more non-MOL institutions (23) than MOL member institutions (13), as Table 5 illustrates. Although the number of reviewers from MOL institutions (49) exceeds the number from non-MOL institutions (27), the ratio indicates the degree to which inter-state, inter-institutional collaboration by individual faculty representatives has become infused into the QM peer review process. Also, since MOL at present is comprised largely of two-year institutions, expanding the peer reviewer pool beyond MOL has enabled the project to increase significantly the number of participating peer reviewers from four-year institutions.

Table 5. Planned Course Reviews and Reviewer Pool, Fall 2005

In addition to project peer reviews (i.e., peer reviews designed to meet grant requirements), the peer review process has also expanded to include other types of peer reviews:

• Reviews of non-MOL courses – Several non-MOL institutions ( Kaplan University, Peirce College, Penn State University World Campus, and the University of North Carolina ) will have courses reviewed via the QM process in Fall 2005 as a pilot to expand this form of inter-institutional collaboration beyond MOL members.
• Paid reviews – UMUC paid MOL to facilitate the review of three courses in Spring 2005 using the QM process, and plans to pay for the review of six additional courses in Fall 2005 (Table 5).
• Independent reviews – in Spring 2005, Frederick and Prince George's Community Colleges reviewed a total of eight courses independently using the QM process 'internally' at their institutions; more independent reviews are planned for the 2005-06 academic year. Although these course reviews are being conducted internally, they are also inter-institutional collaborations since both institutions plan to use external reviewers on their peer review teams (two per team at Frederick CC, one per team at Prince George 's CC).

E. Distributed Activities as Inter-Institutional Collaboration

As the QM project evolved, it became apparent to the PMT that the project of project implementation was unfolding in an unanticipated way. The PMT itself seemed to be functioning very effectively as a collaborative group and has been largely able to depend on the project's working committees to complete needed tasks in a timely and substantive manner. Even more interestingly, project participants were willingly volunteering to take on several key project tasks, including web site development, IDAG formation, and peer reviewer training delivery. Project funds were available to reimburse participants for their activities in some cases, but it was also clear that compensation was not driving the activity. The QM co-directors adopted the term “distributed leadership” to describe this phenomenon. For a variety of reasons, the project structure has empowered various project participants to take the lead on performing project tasks effectively both individually and collaboratively.

Creating a distributed research effort is another example of inter-institutional collaboration through distributed activities. The QM project is designed primarily to address improved student learning indirectly by promoting improved online course design. However, since improving student learning is a primary stated aim of the FIPSE program, the QM PMT issued a call for research proposals to all institutions who had actively participated in the project, providing broad guidelines but distributing the decision making about specific content to applicants. Eight research projects were funded and will commence in Fall 2005, studying the impact of the QM rubric and process on student learning, adaptation of the rubric for use by continuing education, and the transfer the QM process to another statewide consortium. (See www.qualitymatters.org/research.htm for a list of projects.) Although these projects will be implemented individually, collectively their scope and breadth is far superior to what the PMT would have come up with on its own; thus the research projects are the product of a collaborative effort in their own right.

V. FUTURE directions

In its first two years, the QM project has focused primarily on developing the QM model for promoting quality improvement and has seen unanticipated benefits from the model as a vehicle for faculty professional development. The QM project is pursuing several significant initiatives during the third year of the grant which will involve additional inter-institutional collaborative efforts:

1. Making the QM process self-sustainable is the largest issue facing the QM project. Operational control of the project will transfer to MOL at the end of the grant period. Although MOL is taking the lead on this issue by developing a business plan for post-grant sustainability, the QM project will also create an inter-institutional Course and Peer Reviewer Committee to help with the transition course and peer reviewer nominations, and the selection and matching of courses and peer reviewers, based on criteria established in the grant.
2. Application of the QM rubric and process to hybrid and classroom courses will be a primary focus of the Tool Set Committee, which has begun soliciting input and preparing drafts of rubrics modified to accommodate these delivery modes as needed. In addition, the process of identifying hybrid courses for pilot review has been initiated.
3. Other inter-institutional collaborative activities under consideration include updating the research literature and suggesting research projects related to verifying various rubric review standards and coordinating the design and maintenance of separate but compatible rubrics and processes.

The QM project's success to date indicates the viability of creating an inter-institutional collaborative process for quality improvement in online courses. Future efforts will focus on determining whether the project can build and maintain a sustainable model for the long term.

VI. References

1. Quality Matters website. Online: http://www.qualitymatters.org/about_qm.htm.
2. Allen, I. E. & Seaman, J. Entering the Mainstream: The Quality and Extent of Online Education in the United States, 2003 and 2004. The Sloan Consortium, Needham, MA, 2003. Online: http://www.sloan-c.org/resources/survey.asp.
3. Symonds, W. C. Giving it the old online try. Business Week Online, December 3, 2001. Online: http://www.businessweek.com/magazine/content/01_49/b3760072.htm.
4. Yang, Y. and Cornelius, L. Preparing instructors for quality online education. Online Journal of Distance Learning Administration 8(1): 2005. Online: http://www.westga.edu/~distance/ojdla/spring81/yang81.htm.
5. Buck, J. Assuring quality in distance education. Higher Education in Europe 26 (4): 599–602, 2005.
6. Berge, Z. Concerns of Online Teachers in Higher Education. 2001.Online: http://www.emoderators.com/zberge/iste98.html.
7. Bower, Beverly L. Distance education: Facing the faculty challenge. Online Journal of Distance Learning Administration 4(2): 2001. Online: http://www.westga.edu/~distance/ojdla/summer42/bower42.html.
8. Michigan Community College Virtual Collaborative. Online Course Development Guidelines and Rubric. Online: http://www.mccvlc.org/~staff/Course-Guidelines-Rubric-v1.2.html.
9. Edwards, R. Personal communications, October 27–28, 2004.
10. Center for Excellence in Teaching and Learning, California State University, Chico. Rubric for Online Instruction. Online: http://www.csuchico.edu/celt/roi/.
11. Web Initiative in Teaching. WIT Home Page. Online: http://www.umuc.edu/ide/wit/wit.html.
12. Web Initiative in Teaching. WIT Team Projects. Online: http://www.umuc.edu/ide/wit/projects/projects.html..
13. Monterey Institute for Technology and Education. OECP Home Page. Online: http://www.montereyinstitute.org/ocep.html.
14. Monterey Institute for Technology and Education. OECP Overview. Online: http://www.montereyinstitute.org/pdf/OCEP%20Evaluation%20Categories_Distribution%20Version%204_28.pdf.
15. Threlkeld, R. Online Course Evaluation Project: Bringing Quality to Online Learning. Presentation at Distance Learning Conference 2005, Madison, WI, August 4, 2005.
16. Pickett, A. SUNY Course Design Process. From Student Satisfaction and Perceived Learning with On-line Courses - Principles and Examples from the SUNY Learning Network. 1999. Online: http://sln.suny.edu/sln/public/original.nsf/0/67a0a1809ce6fb3585256eac0061cb56?OpenDocument.
17. EArmyU RFP. Online: http://www.timecook.com/consulting/eArmyRFP.doc.
18. EArmyU web site. Online: http://www.earmyu.com/public/public_about-auao_partners.asp.

VII. About the Author

John Sener is Founder and Chief Learner for Sener Learning Services, a consulting practice focused on supporting the evolution of online and other technology-enabled learning environments. He serves as the project evaluator for the Quality Matters project (http://www.qualitymatters.org) and is a member of the QM project management team (PMT). Other recent projects include evaluating the activities of a statewide K–12 online learning consortium, advising a major governmental organization on starting an instructor-led online learning program, and various Sloan Consortium activities. Sener is a contributing editor for the monthly newsletter Educational Pathways and has served on the Journal of Asynchronous Learning Networks editorial board since its inception. Sener's 25+ year career in education and training encompasses a unique mélange of learning experiences. He holds degrees from Johns Hopkins University and Oberlin College.