COMPARISON OF OUTCOMES ON LIKE EXAMS ADMINISTERED TO IN-RESIDENCE AND ASYNCHRONOUS DISTANCE-BASED PHARM.D. STUDENTS

James W. Kleoppel, R.Ph., M.S.
Clinical Assistant Professor of Pharmacy Practice
The University of Kansas School of Pharmacy
Department of Pharmacy Practice
Email: jkleoppe@kumc.edu

ABSTRACT

The purpose of this retrospective study was to compare the academic outcomes of distance-based students in the asynchronous non-traditional pathway with in-residence students in the traditional pathway on examples of like examinations in the Pharm.D. curriculum at the Kansas University School of Pharmacy. This study evaluated the commitment of assuring equivalent outcomes for students in non-traditional and traditional Pharm.D. pathways, as stipulated by the School of Pharmacy accrediting body, the American Council on Pharmaceutical Education (ACPE). Similar examinations were administered to both groups of students. Scores were evaluated using the Student t-Test and ANOVA. This study demonstrates that average test scores of distance-based students were equal to or higher than their counterpart in-residence students when tested on physical assessment content.

KEYWORDS

distance education, asynchronous, non-traditional, Pharm.D., learning outcomes

I. Introduction

In 1997, the American Council on Pharmaceutical Education (ACPE), which is responsible for accrediting schools of pharmacy in the US, revised accreditation standards and guidelines for professional programs in pharmacy leading to a Doctor of Pharmacy degree [1, 2]. These standards made earning a clinical baccalaureate degree in pharmacy no longer an option and mandated that all schools wishing to be ACPE-accredited and able to produce license-eligible graduates offer the Pharm.D. as their only clinical pharmacy degree.

The change in ACPE standards and guidelines created a need for additional training for practicing pharmacists who wished to upgrade their skills and degree to a Pharm.D. A possible crisis could occur if there was a worsening of the already severe national shortage of practicing pharmacists in the US. Therefore, it is important to practicing pharmacists and their patients that practitioners are able to upgrade their skills and degree without having to quit their jobs and/or relocate to a university campus. This educational need resulted in a unique opportunity for schools of pharmacy to provide specialized advanced clinical training specifically for the purpose of upgrading the existing skills of practitioners.

Schools offering so called “add-on” Pharm.D. programs (non-traditional Pharm.D. programs) began to address this unique need and have used a variety of pedagogical methods. An add-on Pharm.D. program, for our purposes here, is defined as one that leads to a Pharm.D. earned by completing traditional courses (in-residence, face-to-face lecture style) either on a full-time or part-time basis. A non-traditional Pharm.D. program is defined as a program that leads to a Pharm.D. degree by completing non-traditional courses. Non-traditional programs can range from programs offering traditional courses at non-traditional times (night classes, weekend “boot camps”, and so on) to those offering distance-based courses utilizing a number of pedagogical styles (mailed print medium, pre-recorded video tapes, synchronized, streamed online multimedia courses, and so on).

Presumably the preference for non-traditional programs is due to the convenience of distance programs allowing practitioners to continue working while completing their degrees, thus creating a far less dramatic impact on students’ personal finances and the pharmacy work force.

Web-based, degree-oriented, professional pharmacy education was not immediately accepted by schools of pharmacy but has slowly become an integral part of many curricula. One question that continues to be asked by faculty and administrators is whether these programs provide academic outcomes equivalent to traditional, in-residence, first-degree pharmacy training programs. The importance of assessing the equivalency of the new distance programs is underscored by the accreditation standards [1] that identify the need for non-traditional programs to also insure the achievement of the same professional competencies as traditional programs.

Many non-traditional pharmacy programs have been developed in the past five to ten years, and several have already ceased operation [3]. Granted, the window of need for a program to upgrade baccalaureate trained pharmacists to the Pharm.D. level is finite, but j udging from the interest in the Kansas program [4], there is still ample demand to justify continuing time and resource allocation. Even drawing from a limited applicant pool, non-traditional programs continue to report notable enrollments [5], but there remains a paucity of data regarding outcome comparisons of traditional and non-traditional pathways in pharmacy programs.

One obvious measure of student content mastery, and one that has been employed in schools of pharmacy for years, is passage of a national licensure examination (North American Pharmacist Licensure Examination or NAPLEX). This measure is not germane for the comparison of outcomes of Kansas University traditional vs. non-traditional Pharm.D. students, since the non-traditional group is already licensed to practice pharmacy and thus would have already passed the NAPLEX exam (except California pharmacists). It is imperative that we find alternate measures to assess the equivalence of educational outcomes produced in the two pathways to better determine the effectiveness of teaching using non-traditional techniques.

This retrospective outcome analysis provides a window into the educational outcomes of same-content professional credit-courses taught using different pedagogical methods. This study was conducted to determine if the School of Pharmacy is meeting the commitment made to the accrediting body, ACPE, that students in the non-traditional program achieve comparable academic outcomes when judged against their counterparts in the traditional first-degree program. This commitment exists independent of a student’s incoming knowledge level, or rate of knowledge acquisition, and is a measure of his or her content mastery at the completion of each respective program.

The hypothesis being tested here is that students completing a Physical Assessment course in the KU School of Pharmacy via the asynchronous, non-traditional pathway have at least equivalent comprehension of course content when compared to students completing a course via traditional, on-campus course delivery methods in the first pharmacy degree program.

II. Methods

The Advisory Committee on Human Experimentation at the University of Kansas, following an expedited IRB review, approved this study. All reasonable safeguards were utilized throughout the study to maintain confidentiality.

The populations studied consist of traditional, first-degree students in the on-campus Pharm.D. program, near the end of the didactic curriculum, along with distance-based, baccalaureate-trained practicing pharmacists enrolled in the asynchronous, non-traditional Pharm.D. program. No controls were implemented to equalize the populations for age, experience, or entry-level skill. This type of manipulation would not be practical for this study, since the comparison was of two groups that would be expected to be of different ages: adult students already possessing a clinical degree and first-clinical degree students. It should be noted that, based on review of transcript records, neither group of students had taken a physical assessment course prior to their entrance into the Pharm.D. program.

The physical assessment courses studied were taught in the Pharmacy Practice Department at the University of Kansas, used the same lecture topics and reading material, and were taught by the same faculty member. The textbook and readings were identical for both groups. The traditional students (first professional degree students) were instructed live (face-to-face), whereas the non-traditional students viewed streamed multimedia lectures on demand (asynchronous) via the Web. Part of the requirements of each course was that students must demonstrate familiarity with equipment used in physical assessment (i.e. Sphygmomanometer, Stethoscope, Ophthalmoscope, Otoscope, Percussion hammer, Tongue blades, Peak Flow meter and more). This demonstration was done under the supervision of the faculty member for traditional students, or under the supervision of a faculty member, physician or nurse for non-traditional students. Each group’s results were evaluated as a pass/fail participation grade for this section of the course. Physical assessment content in each course was limited in availability to a one semester time period. Two multiple-choice paper examinations were administered as open book, with a one-hour time limit for traditional students in a face-to-face format. Non-traditional examinations were administered as open book via the Web by password protected online examination software (Blackboard) with a one-hour time limit. Both sets of examinations had questions identical in content and number but appeared in random order. Results from like exams taken in traditional, in-residence courses (three courses taught from 1996–2000) were compared to results from non-traditional, distance-based courses (2000 and 2001). The mean scores of like exams in the two general groups (all traditional students vs. all non-traditional students) were compared using a Students T-Test, two-sample assuming equal variances in Microsoft Excel ®. An ANOVA with Bonferroni post hoc analysis (since variances were equal between groups) was conducted using SPSS to compare the individual classes (years) on each exam. The examinations quizzed students on the knowledge necessary to understand physical assessments. They were not intended as competency assessments to assure the acquisition of clinical skills necessary to collect physical assessment data.

III. Results

Demographic data for both groups of students is shown in Appendix, Table 1. The average age of the non-traditional students was higher than the traditional, first-degree group. The non-traditional group was made up of a higher percent of males and had more students with other baccalaureate degrees. In addition the traditional on-campus students were represented by a higher percentage of Kansas residents than the distance students.

Figure 1 in the Appendix shows the cumulative scores (average) of all students taking examinations by traditional methods (three classes from 1996–2000) compared to students taking like examinations via non-traditional methods (2000–2001). In all three samples (Exam 1, Exam 2, and Cumulative Total) the non-traditional students scored significantly higher than their counterparts who had taken their exams in a traditional setting (P = 0.01, P < 0.01, and P < 0.01 respectively per student’s T-test).

Average results, expressed in percent for individual years, of exam one, exam two, and the cumulative scores in the physical assessment course were also evaluated to look for differences or trends within the study population. An analysis of variance (ANOVA) of exam one (Table 2, Appendix) results determined that the variance in the groups were comparable. The overall F value was 42.6 and the P value was < 0.01. The Bonferroni post hoc analysis determined that the 2000 scores were significantly below all four other classes examined (P < 0.01 on each). In addition scores from the 2001 class were significantly lower than the 1998 class (P < 0.01).

When an analysis of variance was conducted on the second examination (Table 3, Appendix) it was found that the variance in the groups was again comparable. The overall F value was 31.2 and the P value was < 0.01. The Bonferroni post hoc analysis of the results from this exam demonstrated that the scores of the 2000 class were significantly lower than all other classes (P < 0.01 for each). There were no other significant differences between the examination scores on this exam.

When an analysis of variance was conducted on the cumulative scores (Table 4) of each class an identical pattern to the second exam was seen. The overall F value was 51.7 and the P value was < 0.01. The results of the 2000 class were significantly lower than all other classes but no other differences between classes were significant.

A. Study Limitations

Limitations of this retrospective comparison include that there was no gauge for incoming skill and knowledge to identify potential differences in the two groups at the start of the course. While this is important, the current study addressed the ACPE-mandated equivalence of outcomes instead of absolute changes in knowledge. Assessment of final content mastery as an index is consistent with the ACPE Policies on Postgraduate Education and Training section of the Cumulative Policies of the AACP House of Delegates from 1980–2003 [6]. The change in student quality from year to year (poor 2000 results) was not factored into the overall conclusions but could have a bearing on the study results. While there appeared to be a decrease in student performance in 2000, it is important to note that the class posted a 100% first try pass rate on the NAPLEX exam.

IV. Discussion and Conclusions

The difference in the average age of students in the two groups is not surprising when you consider the groups being compared. By definition non-traditional students already have a BS degree in pharmacy and are licensed to practice in the US, thus they would be expected to be older. Likewise the higher percent of previous non-pharmacy BS degrees would be expected to be higher in the older, non-traditional group. The percent of students listing Kansas as their home state would also be expected to be higher in the distance program, since the Non-Traditional Pharm.D. program is international in scope and has drawn from well over half of the states in the nation since its inception. To the authors, the most unexpected finding in the demographic data was the percent of minorities. It was somewhat surprising that there were more minorities in the in-residence traditional program, since Kansas tends to have minorities minimally represented throughout the state.

When we compared the like exam scores of all similar classes combined it was reasonable to conclude that students in the asynchronous, distance-based program had equivalent or superior academic outcomes. This conclusion is based on the significantly higher average scores on both of the examinations and the cumulative scores. When we evaluated the data using the results of the ANOVA it was apparent that the in-residence 2000 class scores were significantly lower than all other classes (of both pathways) and lowered the overall in-residence average. It is tempting to exclude this class for evaluations, but the authors don’t feel this would be appropriate and would introduce potential bias, since the class in question, on another universally employed outcome comparison, posted a 100% NAPLEX pass rate on their first try. These data, when viewed in their entirety, suggest that the mastery of content (grades) was at least comparable in both groups of students and suggests that the non-traditional class outcomes may have even been superior.

Regardless of whether we look at the cumulative average of all classes (including 2000), or compare individual classes, these data suggest that physical assessment content taught to practicing pharmacists using non-traditional methods yielded students with at least comparable knowledge, relative to students in the first pharmacy degree program (with the exception of 1998 trad. vs. 2000 non-trad.). This is important since our commitment to students, faculty, and ACPE is that students earning their Pharm.D. degree via coursework offered with distance techniques will have, as a minimum, course comprehension equivalent to traditional, in-residence graduates. It is also tempting on the surface to minimize these findings and assume that the improved performance on the physical assessment exams is due to practicing pharmacists having acquired this knowledge in their BS programs. The flaw with this assumption is that most practicing pharmacists were not exposed to physical assessment courses in BS curricula (as determined by a review of NTPD student transcripts), and thus most were not taught formal assessment skills previously. These findings support the hypothesis that Non-Traditional Pharm.D. students have equivalent comprehension of physical assessment course content compared to traditional, first professional degree students.

There are other factors in this comparison that may have an impact on these results that should be considered. Theoretically the trend towards better performance by non-traditional students may be impacted by fewer total credit hours per semester taken by the non-traditional students. Counting enrollment hours only, however, would not factor in the number of hours worked by either group, which may also play a role in academic performance. While most students work in addition to their studies, few would be expected to work as many hours as the non-traditional students, who generally work at or near full-time status.

Better time-management skills by the adult students have also been considered as a possible source of the difference in the outcomes of these two groups. This may be the case, but it is important to remember that the traditional students are in a professional program, nearing the end of their didactic training. At this point in a student’s career, those unable to manage their time wisely would be expected to have already left the program.

It is also plausible that access to other healthcare professionals and acquisition of terminology through practice by the non-traditional students may have played a role in the outcome of the study. While this would seem to favor the non-traditional student, who is working in a clinic setting, it does not consider the numerous prerequisite courses that teach many of these terms and concepts. Completion of these courses for the non-traditional student would be much farther removed than in the traditional student group. While these future study topics may impact the learning process, the final outcome serves as the initial comparison point to determine if, at the completion of the course, each group of students has comparable content mastery in physical assessment coursework.

Future studies are planned to compare acquired skills in like courses and to examine broad programmatic outcomes. When we look at acquired skills (a change in skill level) many unknowns that prevent us from making much broader conclusions regarding the comparison of these two teaching methods should be eliminated.

V. References

1. American Association of Colleges of Pharmacy. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree, 1997a. Online. Available: http://www.acpe-accredit.org/pdf/Standards2000.pdf.
2. American Association of Colleges of Pharmacy. Implementation Procedures for Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree, 1997b. Online. Available: http://www.acpe-accredit.org/pdf/pharmD.pdf.
3. Ukens, C. Schools pull the plug on nontraditional Pharm.D. Drug Topics Health-System Edition 6: 24, 2002.
4. Ragan, R.E. The Non-Traditional Pharm.D. Program at Kansas University: An Update. Journal of Kansas Pharmacy 76(9): 10–12, 2002.
5. American Association of Colleges of Pharmacy. Pharmacy School Admission Requirements 2004–2005. Online Available: http://www.aacp.org/site/page.asp?TRACKID=&VID=1&CID=693&DID=4747.
6. American Association of Colleges of Pharmacy. Cumulative Policies of the American Association of Colleges of Pharmacy House of Delegates 1980–2003. Online. Available: http://aacp.org/site/tertiary.asp?TRACKID=&VID=2&CID=53&DID=4963.

VI. About the Authors

Ronald E. Ragan is the director of the Non-Traditional Pharm.D. program in the School of Pharmacy at Kansas University in Lawrence, KS. He directed the design and development of the Non-Traditional program at Kansas. In addition to director responsibilities he is also a faculty member in the Pharmacy Practice department and is responsible for the Alzheimer’s disease section of the online program. He earned a B.S in Pharmacy, M.S. and Ph.D. in Pharmacology from Kansas University.

James W. Kleoppel is a Clinical Assistant Professor of Pharmacy Practice in the School of Pharmacy at Kansas University in Kansas City, KS. He earned a B.S. in Pharmacy and M.S. in Pharmacology from the University of Kansas.

VII. Appendix

Table 1: Demographic Data

Table 1 Legend: Summary of demographic data comparing traditional (in-residence) students and non-traditional (distance-based) students.

Figure 1 Legend: Results, reported as percent +/- standard error, of all students taking each test in-residence compared with students taking like tests via asynchronous, distance-based delivery. The P value from a Student T test is indicated at the top of each pair of bars compared.

Table 2: Bonferroni Post Hoc Aanalysis of Student Scores on Exam One

Table 2 Legend: A post hoc ANOVA analysis using SPSS was done to compare the scores (reported as percent correct) on the first like examination in Physical Assessment. The comparisons that resulted in a significant difference (P < 0.05) between the two groups are identified in the table. Sample sizes (n) are indicated in the horizontal header row.

Table 3 Legend: A post hoc ANOVA analysis using SPSS was done to compare the scores (reported as percent correct) on the second like examination in Physical Assessment. The comparisons that resulted in a significant difference (P < 0.05) between the two groups are identified in the table. Sample sizes (n) are indicated in the horizontal header row.

Table 4: Bonferroni Post Hoc Analysis Using ANOVA in SPSS — Cumulative Score

Table 4 Legend: A post hoc ANOVA analysis using SPSS was done to compare the scores (reported as percent correct) on the total score of both examinations combined in Physical Assessment. The comparisons that resulted in a significant difference (P < 0.05) between the two groups are identified in the table. Sample sizes (n) are indicated in the horizontal header row.