An independent evaluator (IOTA Solutions) administers reliable and valid evaluative measures via the Internet to track the effectiveness of the NACTEL Program, address and eliminate obstacles to student learning, and anticipate the changing needs of students. This online feedback system automatically tabulates results, highlights potential problems, and provides continuous feedback to key stakeholders (program administrators, faculty) anytime and anywhere, as the key stakeholders need this information to make ongoing improvements.
Evaluative measures associated with the student satisfaction and access pillars are administered to new students three times each term and to returning students twice each term (in addition to the individual course surveys given by instructors). Table 1 illustrates Pace University's success in effectively addressing these pillars. Three of the evaluative measures are described below:
- Application process / Web Site Measure
Delivered to new applicants to the NACTEL Program. Explores satisfaction with the application process and the effectiveness of the Pace web site (http://support.csis.pace.edu/nactel) in delivering information about the Program.
- Registration process / Course Satisfaction Measure
- New student version: Delivered to all new students who have registered for courses. Explores attitudes towards the registration process, various aspects of the website, and expectations toward online learning prior to the start of classes.
- Returning student version: Delivered to all returning students who have registered for courses. Explores attitudes towards the registration process, satisfaction with course experience, and attitudes towards online education. Survey results can be compared across new and returning Student versions to explore the relationship between expectations and experiences. Self-report performance items explore how attitudes and demographics relate to various levels of performance.
- Online Testing Measures
- Proctors: Given to all individuals who participate as proctors. Explores satisfaction with the process, suggestions for improvement, and interest in proctoring for other students.
- Students: Given to all students who have completed an online proctored examination. Explores understanding of the process, satisfaction with the process, difficulties encountered, and suggestions for improvement.
Table 1:
| Items and responses from the Online Support Services and Online Testing surveys. |
| Online Support Services Items (1 & 2 Above) |
Response Trend |
Effectiveness of answers to your questions regarding the application:
|
91% "Good" or "Very Good" |
Quality of the student advisement received:
|
85% "Good" or "Very Good" |
Overall satisfaction with online application process:
|
88% "Satisfied" or "Very Satisfied" |
Overall satisfaction with online registration process:
|
92% "Satisfied" or "Very Satisfied" |
Access to student support services is good:
|
68% "Agree" or "Strongly Agree"
(26% did not use) |
Received adequate support from my instructors:
|
90% "Agree" or "Strongly Agree" |
High quality learning possible without F-t-F interaction:
|
89% "Agree" or "Strongly Agree" |
Overall satisfaction with program:
|
96% "Satisfied" or "Very Satisfied" |
| Online Testing Items |
Response Trend |
Proctored exam information clear & understandable:
|
83% "Agree" or "Strongly Agree" |
Responses to questions concerning the proctoring program were satisfactory:
|
93% "Agree" or "Strongly Agree" |
Overall, proctored examination [online test] went well:
|
94% "Agree" or "Strongly Agree" |
[Note: Responses to Online Support Services items include those that were collected after the implementation of the current NACTEL Program website.]
Conclusions: Comparison of results between the New and Returning Student measures indicate that: (1) new students expect more interaction than returning students have experienced; and (2) new students expect to be bothered more by a lack of interaction than returning students have been bothered. These results illustrate student changing perceptions and indicate students are able to adjust to the limited interaction.
Student comments also provided encouraging results. Students indicated they would have been unable to pass their courses without the math tutoring services provided and that the convenience of online learning is an important advantage over traditional forms of learning.