INFLUENCE OF NON-MODERATED AND MODERATED DISCUSSION SITES ON STUDENT SUCCESS

Deborah A. Kashy

Download PDF version:

Department of Psychology
Michigan State University
East Lansing, MI 48824
E-mail: kashyd@msu.edu

Guy Albertelli II
College of Natural Science
Michigan State University
East Lansing, MI 48824
E-mail: albertel@msu.edu

W. Bauer, E. Kashy, M. Thoennessen
Department of Physics and Astronomy
Michigan State University
East Lansing, MI 48824
E-mail: bauer@nscl.msu.edu
kashy@nscl.msu.edu
thoennessen@nscl.msu.edu

ABSTRACT
One of the most successful applications of networked technology in education has been the development of individualized homework assignments. Such assignments provide immediate feedback to students and discourage copying. However, networked technology also allows students to develop sophisticated schemes designed to beat the system. This paper, supported in part by Andrew W. Mellon and Alfred P. Sloan Foundations, reports the educational outcomes for introductory physics students who used two web sites, one designed by the course instructor to provide assistance to students and the other a site designed by a third party to provide students with the solutions to problems. Results indicated that students who used the third-party site more tended to perform more poorly on all measures of success in the course, and that these relationships held even after accounting for students' academic ability. On the other hand, students who more often used the site designed by the instructor tended to perform better in the course.

KEY WORDS
Learning Effectiveness, Education, Personalized Homework

I. INTRODUCTION

For at least thirty years there has been the hope and belief that computers may be able to aid and eventually possibly replace humans in the process of teaching other humans [1]. Before the invention of the computer, similar claims were made for other technologies such as the printing press, radio, television, and video recorder/player. The computer may be qualitatively different from these information transfer technologies in its ability to produce customized feedback. Particularly since the invention of the World Wide Web a little more than a decade ago, the process of networking computers has amplified this possibility. It has, unfortunately, also amplified the possibility of sophisticated methods of copying assignments (or solutions) by web-savvy students.

Over the last few years the extensive use of interactive Java applets has been tried in the classroom [2,3], and computers have been used as tutors [4]. Most successful may be the use of the computer as a homework system, because the computer can provide individualized and personalized problem sets and enable students to input solution attempts with immediate feedback and customized help [5-9]. At their best, these systems can entice students to interact with each other and spend more time on task, create more active learners, and make learning problem-solving skills more enjoyable. Perhaps most importantly, computer-based individualized systems prevent students from mindlessly copying their friends' homework assignments.

Since the creation of sophisticated computerized assignments can be extremely time-consuming, substantial effort has been put into the creation of databases of problems and other multimedia resources on the Internet. Such materials should encourage and facilitate the use of technology by enabling faculty at universities, community colleges, and K-12 schools to take advantage of the work of their colleagues. Currently, our students access these resources via locally developed software (CAPA [5] and LOL [9]), with the LON-CAPA project, Learning Online Network with a Computer-Assisted Personalized Approach currently being developed to include the best features of these current systems [5,9,10].

However, the same computers that can be used by faculty to create richer and more individualized learning environments can also be used by students in ever more sophisticated schemes to beat the system. In other words, students can try to get credit for the assignment without doing the work. While the Internet is now a global research library of previously unknown power, it also enables enterprising would-be cheaters to cut and paste material for easy and relatively thought-free composition of essay assignments. This has forced faculty to resort to the creation of sophisticated search engines that check for hints of plagiarism [11].

We have now encountered a new level of sophistication in this “arms race,” commercial web sites constructed by a former student to defeat personalized computer homework assignments. These include problems with essentially the same text but with different numbers to different students to prevent the direct copying of the answers. The algorithm that determines if the given answer is correct is implemented in a server-side cgi-script and can thus not be discovered by a simple “View Page Source” command on the user's web browser. However, all student answers are computed from their input numbers via the same formula, derived from the relevant physics principles, in a given problem. Posting this formula constitutes reverse engineering of sorts and enables the kind of mindless plug-and-chug procedure that our homework systems were created to avoid.

II. STUDENT DATA

During the winter semester 1999, one of us (E. Kashy) taught an introductory calculus-based course in engineering physics using the CAPA personalized homework system [5]. The combined enrollment in all sections of this course was 472. An interesting component of this course for the purposes of this paper is that we created our own web-based threaded discussion forum that was monitored by teaching assistants. The assistants helped and guided the students who had difficulties but did not simply give solutions away. Of the 472 students in the class, 129 posted questions and contributions at least once. Based on this information, we created a variable labeled “Post-sanctioned” that indicates whether or not the student ever posted a question or contribution on this course sanctioned site. We also recorded how often students visited our site without posting, just “looking,” during the semester. This distribution was highly skewed, with a mean of 33.01 and a standard deviation of 29.95, ranging from 0 to a maximum of 172. We refer to this square rooted variable as “Look-sanctioned.”

After the semester was completed (and all grades were assigned to the students), we researched how widespread the use of the third party commercial web site for the homework assignments had been. We wanted to know if use of the site had been beneficial or detrimental to student learning. Should we take steps to limit access to that site? To this end, W. Bauer, in his role as associate chair and undergraduate program director, sent an email to all students in the course. This email asked for the percentage of homework problems for which each student had used the site, with the promise that no repercussions would result from the answers. We explicitly stated that we would like the students to respond even if they had never used the site.

Approximately one third of the students in the class responded to the email and of these 133 gave a quantitative estimate of their use of the site. More specifically, students were asked to provide a number between 0 and 100 indicating the (approximate) percentage of their homework problems for which they used the third-party site. This percentage estimate is referred to as “Third-party percent.” Of the responses containing numeric estimates, we were able to identify 122 by name and student number so that their estimates could be linked with classroom performance. Self-reported use of the third-party site ranged from 0 to 100 percent of the problems in the course. The average use was 53%.

Because only a subset of the students in the class responded to our email, we first tested whether there were any systematic differences between responders and non-responders in terms of general academic ability and class performance. Although there were small differences between responders and non-responders on the ACT composite, the ACT math, overall GPA, and score on the final exam for the course, such that responders tended to have slightly higher scores, no statistically significant differences emerged. Given these results, it seems reasonable to assume that the students who responded to our email comprise a representative sample of the overall student population in this course.

Table 1: Correlation coefficients (and p-values) among the scores on the exams, homework, and improvement on the Force Concept Inventory test and student use of different web sites. The column named 3rd Party Percent refers to percent use of a web site that simply gave away the solution. The other two columns refer to the use of our own course sanctioned web site with threaded discussion forum, which did not give away the solution, but attempted to guide the students to find their own solution.

Thus we have data concerning use of the sanctioned discussion forum, data concerning use of the third party site, and data concerning success in the course. Success in the course is measured by the exam scores of the students, their total homework score, and their improvement in the Force Concept Inventory [12] (FCC) score, administered at the beginning and end of the semester.

Table 1 contains the correlation coefficients between measures of success in the course, percent use of the third party site, and posting/looking at the course sanctioned site. Consider first the results for the third-party site. Students who used this site more according to their own self report, tended to perform more poorly on exams, quizzes, and the final. In other words: the more students relied on the easy plug-and-chug answers on that web site, the worse they ended up performing on the exams. Several students echoed exactly this sentiment in their email comments to us, too. For example, one student commented “I used it (the third party site) 100% and I have to take the course again.” It is also noteworthy that relative to those who used the site less, those who used it more did not perform better on the homework.

It is possible that students who may be destined to perform relatively poorly tend to seek more help wherever it can be found. Thus, there may be a built-in negative relationship between help-seeking behavior and success. However, the second and third columns in Table 1 contradict this line of reasoning. While students using the third-party site perform more poorly, students using the course sanctioned site perform better. There we see a small positive correlation between use of our web site and the exam scores. This shows that helping the students find their own solutions to problems seems to have a positive effect on the learning outcome, whereas simply giving away the solutions appears to have a negative effect.

Because we had access to a measure of the students’ general academic performance level (ACT composite), we were able to assess whether students who were poorer academically also tended to use the help sites more. The correlation between general academic ability and use of the third party site was r = -0.284, N = 111, p = .002. The corresponding correlation with Posting on the sanctioned site was r = .142, N = 383, p = .005, and the correlation with Looking was r = .130, N = 383, p = .011. Therefore there is evidence that students who used the third party site more frequently tended to be poorer students and students who used the course site tended (albeit weakly) to be better students. Such a pattern suggests that the results in Table 1 may simply reflect varying levels of academic ability rather than a problem with the web sites.

Table 2: Partial Correlation coefficients (and p-values) between the scores on the exams, homework, and improvement on the Force Concept Inventory test and student use of different web sites, partialling out ACT composite scores.

To examine this issue, partial correlation coefficients between site use and success were computed, partialling out the composite ACT score. The partial correlations are presented in Table 2 and are highly similar to those in Table 1. So even after accounting for varying levels of academic ability (and for the fact that people who were stronger academically tended to use the third-party site less), students using this site performed poorer. In addition there is still a tendency for students who used the course sanctioned site to do better even after accounting for academic ability, but the relationships are quite weak.

III. CONCLUSIONS

The results suggest that high-tech copying is associated with lower levels of performance across all levels of academic ability. However, even if students are provided with this information (i.e., they are told that use of high-tech copying has been shown to lower overall course performance), it is not clear that the majority will be convinced to do their own work. Thus the “arms race” in which educators try to develop more and more sophisticated problems and formats may be necessary in order to protect students from their own weaknesses and to help them to resist using the “cheap” solutions. Our results also offer a measure of “peace-of-mind” for instructors who may be concerned that students are artificially inflating their grades by high-tech copying, since that clearly is not the case.

Sophisticated design of questions for personalized assignments can make high-tech copying more time consuming than doing the work properly. We are making progress in designing such sophisticated questions, albeit at a significant cost in time and effort [13]. Still, with inventive students, new methods that allow a large fraction of the students to circumvent doing the assigned work will continue to be developed. We believe that competing with these inventive students not only provides instructors with an intellectual challenge, but that it is also necessary in order to insure that a majority of the students actually do the assigned work.

IV. REFERENCES

1. P.T. White, National Geographic, November 1970, p. 593.
2. M.V. Goldman and D. Rea, Siggraph Proceedings, p. 38 (1998); http://www.colorado.edu/physics/2000/
3. G.M. Novak, E.T. Patterson, D. Gavrin, and W. Christian, “Just in Time Teaching” (Prentice Hall, Upper Saddle River, NJ, 1999).
4. F. Reif and L.A. Scott, Am. J. Phys. 67 (9), 819 (1999).
5. E. Kashy, B.M. Sherrill, Y. Tsai, D. Thaler, D. Weinshank, M. Engelmann, and D.J. Morrissey, Am. J. Phys. 61 (12), 1124 (1993). Also http://capa.msu.edu
6. E. Kashy, S.J. Gaff, N.H. Pawley, W.L. Stretch, S.L. Wolfe, D.J. Morrissey and Y. Tsai, Am. J. Phys. 63 (11), 1000 (1995).
7. M. Thoennessen, and M. Harrison, “Computer-Assisted Assignments in a Large Physics Class”, Computers and Education, Vol. 27, No.2, pp. 141-147 (1996)
8. E. Kashy, M. Thoennessen, Y. Tsai, N. E. Davis, and S. L. Wolfe, “Using Networked Tools to Promote Student Success in Large Classes”, Journal of Engineering Education, ASEE, Vol. 87, No. 4, pp. 385-390 (1998)
9. G. Kortemeyer and W. Bauer, J. of Eng. Educ., 88 (4), 421 (1999); http://fie.engrng.pitt.edu/fie98/papers/1199.pdf
10. http://lecture.lite.msu.edu/~korte/lon/
11. Chronicle of Higher Education, December 14 (1988); http://plagiarism.com/; http://www.plagiarism.org/; http://www.turnitin.com
    Washington Post, September 26, C07 (1996)
12. D. Hestenes, M. Wells, and G. Swackhammer, Phys. Teach. 30, 141 (1992).
13. M. Thoennessen, E. Kashy, G. Albertelli II, and Y. Tsai, “Response to Student Attacks on On-Campus ALNs”, Proceedings of the Fifth International Conference on Asynchronous Learning Networks, http://www.sloan-c.org/conference/proceedings/1999/index.asp